Open pcm32 opened 1 year ago
pcm32
commented
1 year ago Reproducing the same tool call with the same inputs, I see again a smaller peak (to 55% of memory) for postgres, and the same errors on the logs regarding the database being in recovery mode.
pcm32
commented
1 year ago Changing the postgres Statefulset to use 1.5 Gi of RAM instead of the original 500Mi makes this work. I find mysterious that
Apologies, happy to move this issue to Galaxy Helm as it is a particularity of that setup.
pcm32
commented
1 year ago I think that Galaxy is putting an awful load of data on the database for some complex tool executions. Since what I posted above, I have re-run the same tool (which takes 3 different collections, 1 of them in two separate inputs) and the size usage of the database has grown considerably (and I'm not running anything else):
root@galaxy-galaxy-dev2-postgres-0:/home/postgres/pgdata# df -h .
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 20G 1.1G 19G 6% /home/postgres/pgdataso it has doubled with only 6 or 7 more executions of the same tool (due to resubmissions).
bgruening
commented
1 year ago Can you share this tool? Can you maybe narrow down the query that is causing this behavior? Maybe by turning on the slow query logging in galaxy.yml?
pcm32
commented
1 year ago I think that the table receiving the increase is history_dataset_association_history. After some 10 or so executions of the same workflow, db disk space looks like this:
Filesystem Size Used Avail Use% Mounted on
/dev/vdb 20G 2.0G 18G 10% /home/postgres/pgdatatool generating this is https://humancellatlas.usegalaxy.eu/root?tool_id=toolshed.g2.bx.psu.edu/repos/ebi-gxa/anndata_ops/anndata_ops/1.8.1+3+galaxy0 in a usage where you activate entrypoints for various collections of AnnData files:
where in this case inputs 2 to 6 are collections (usually of around 10 to 20 datasets). I will try the slow query logging as suggested.
Describe the bug
One of the last steps of our workflow for some reason triggers some very disruptive issues with the database.
On the workflow handler I see:
at the end it says again it is scheduling, but it doesn't or workflow comes back as failed. In the meantime, the job handler shows:
Last week I though that this was due to the database disk being nearly full, however, this is happening with a nearly empty disk now:
Based on the time, it could be that the maintenance scripts where running at that time as well (around 2:05 AM), although this seems more like a coincidence as last week it also failed at the same place at a completely different time of the day.
The database container shows the following on its log around that time:
At 2:08 it shows like there was a bump in the connection. There doesn't seem to be any abnormality due to the maintenance job at 02:05 or so. There is no restart of the pod at that time for postgres, but you can see a peak in memory usage and CPU:
however it says it only reaches 70% of memory utilisation (and k8s shows no signs of OOMs for that container). Perhaps a failsafe within the pod for too much memory being acquired?
Galaxy Version and/or server at which you observed the bug Galaxy Version: 22.05 Commit: 3b068eecb9f9573a90b51f3eeb1bc06d1d6eb88d
To Reproduce Steps to reproduce the behavior:
Expected behavior
The workflow should finish correctly or fail, but probably not due to the above database error.
Screenshots
This is what can be seen from the UI for the workflow invocation:
Additional context
This is running using the Galaxy Helm chart, which provisions postgres through an operator (perhaps this only happens there? as I haven't seen this issue in other instances).