Closed Fiwx closed 1 month ago
The solution here would be to run it on less scores at once, or allocate it additional memory.
Thanks! @smlmbrt, is the memory-intensive part likely the "Calculating overlap between target genome and scoring file" step here? Or do you think that is right before the memory-intensive step?
Perhaps, but it could also be the step write after (pivoting the df and writing it to disk) that doesn't complete. Either way, there's no getting around the fact that the more variants you have in scoring files the more memory it will take.
Description of the bug
pgsc_calc is repeatedly failing due to out-of-memory errors during the pgscatalog-matchmerge step. The process is being killed when attempting to filter best_match variants and calculate overlap between the target genome and scoring file. This causes the pipeline to crash and requires manual intervention to resume, which sometimes can work, but typically not. With 32 GB RAM, pgscatalog-matchmerge process using up to 68GB of virtual memory and about 30GB of physical memory. It goes into swap memory, and crashes. It also does not run fully on 62 GB of RAM. I am using 63 (local) scorefiles in the run.
Apologies for the multiple opened issues; they appear to be different issues on different steps.
Here is the issue:
/home/user/org/runner/test/test1_file8270_yofsample_uk_s_uk.23andme/work/b6/78b553f872176c7c50419d0f1bcda6/.command.sh: line 9: 1357 Killed pgscatalog-matchmerge --dataset test1file8270yofsampleuksuk --scorefile scorefiles.txt.gz --matches *.ipc.zst --min_overlap 0.0 --filter_IDs filter_ids.txt.gz --outdir $PWD --combined -v
After each kill, the system attempted to reclaim memory, as indicated by "oom_reaper" messages. The pgscatalog-matchmerge command was consistently failing at the stage of "Filtering to best_match variants" and "Calculating overlap between target genome and scoring file".
Command used and terminal output
Relevant files
No response
System information
pgscatalog/pgsc_calc: v2.0.0-beta.3 profile: Singularity Both on 64 and 32 GB of RAM Nextflow version: 24.04.4