search

sdparekh / zUMIs

zUMIs: A fast and flexible pipeline to process RNA sequencing data with UMIs
GNU General Public License v3.0
271 stars 67 forks source link

BUG - zUMIs does not respect num_threads #252

Closed stetner closed 3 years ago

stetner commented 3 years ago

Describe the bug When running a test for a user with

num_threads: 16

In the yaml file I observed at least 53 pigz threads.

To Reproduce yaml with num_threads: 16

top showing: 683+buff/cache KiB Swap: 11568537+total, 11087499+free, 4810384 used. 11379280+avail Mem

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 6506 uqdstetn 20 0 59100 2948 1572 R 0.0 0.0 0:47.20 top 30214 uqdstetn 20 0 1208484 10720 700 R 94.7 0.0 0:32.53 pigz 30119 uqdstetn 20 0 1208484 10720 700 R 93.7 0.0 0:32.50 pigz 30172 uqdstetn 20 0 1208484 10720 700 R 92.7 0.0 0:32.49 pigz 30197 uqdstetn 20 0 1208484 10720 700 R 94.0 0.0 0:32.45 pigz 30202 uqdstetn 20 0 1208484 10720 700 R 95.0 0.0 0:32.44 pigz 30112 uqdstetn 20 0 1208484 10720 700 R 95.7 0.0 0:32.36 pigz 30130 uqdstetn 20 0 1208484 10720 700 R 95.0 0.0 0:32.35 pigz 30162 uqdstetn 20 0 1208484 10720 700 R 93.7 0.0 0:32.35 pigz 30099 uqdstetn 20 0 1208484 10720 700 R 96.0 0.0 0:32.32 pigz 30066 uqdstetn 20 0 1208484 10720 700 R 94.0 0.0 0:32.28 pigz 30207 uqdstetn 20 0 1208484 10720 700 R 95.0 0.0 0:32.22 pigz 30142 uqdstetn 20 0 1208484 10720 700 R 94.7 0.0 0:32.16 pigz 30184 uqdstetn 20 0 1208484 10720 700 R 94.4 0.0 0:32.14 pigz 30225 uqdstetn 20 0 1208484 10720 700 R 95.0 0.0 0:32.12 pigz 30080 uqdstetn 20 0 1208484 10720 700 R 95.3 0.0 0:32.10 pigz 30179 uqdstetn 20 0 1208484 10720 700 R 94.7 0.0 0:32.10 pigz 30120 uqdstetn 20 0 1208484 10540 704 R 26.2 0.0 0:10.83 pigz 30107 uqdstetn 20 0 1208484 10540 704 R 29.9 0.0 0:10.73 pigz 30087 uqdstetn 20 0 1208484 10540 704 R 24.6 0.0 0:10.69 pigz 30234 uqdstetn 20 0 1208484 10540 704 R 27.2 0.0 0:10.68 pigz 30129 uqdstetn 20 0 1208484 10540 704 R 29.2 0.0 0:10.66 pigz 30147 uqdstetn 20 0 1208484 10540 704 S 23.6 0.0 0:10.58 pigz 30097 uqdstetn 20 0 1208484 10540 704 S 22.9 0.0 0:10.52 pigz 30187 uqdstetn 20 0 1208484 10540 704 S 27.9 0.0 0:10.50 pigz 30194 uqdstetn 20 0 1208484 10540 704 S 24.3 0.0 0:10.49 pigz 30137 uqdstetn 20 0 1208484 10540 704 R 23.3 0.0 0:10.48 pigz 30157 uqdstetn 20 0 1208484 10540 704 S 25.6 0.0 0:10.48 pigz 30215 uqdstetn 20 0 1208484 10540 704 S 25.2 0.0 0:10.43 pigz 30167 uqdstetn 20 0 1208484 10540 704 S 25.6 0.0 0:10.41 pigz 30152 uqdstetn 20 0 1208484 10540 704 R 23.9 0.0 0:10.40 pigz 30224 uqdstetn 20 0 1208484 10540 704 S 26.2 0.0 0:10.28 pigz 30074 uqdstetn 20 0 1208484 10540 704 R 20.3 0.0 0:10.15 pigz 30023 uqdstetn 20 0 34524 944 704 R 30.2 0.0 0:09.67 pigz 30024 uqdstetn 20 0 4416 668 496 R 15.6 0.0 0:05.01 split 30027 uqdstetn 20 0 34524 956 716 S 12.6 0.0 0:04.42 pigz 30025 uqdstetn 20 0 34524 948 708 S 9.3 0.0 0:03.85 pigz 28485 uqdstetn 20 0 165424 2520 1120 S 0.0 0.0 0:02.96 sshd 14887 uqdstetn 20 0 1208312 9768 704 S 59.8 0.0 0:01.98 pigz 14895 uqdstetn 20 0 1208312 9768 704 S 60.1 0.0 0:01.97 pigz 14888 uqdstetn 20 0 1208312 9768 704 S 59.8 0.0 0:01.95 pigz 14879 uqdstetn 20 0 1208312 9768 704 S 57.5 0.0 0:01.92 pigz 14898 uqdstetn 20 0 1208312 9768 704 S 58.5 0.0 0:01.91 pigz 14903 uqdstetn 20 0 1208312 9768 704 S 57.8 0.0 0:01.91 pigz 14878 uqdstetn 20 0 1208312 9768 704 S 57.8 0.0 0:01.89 pigz 14906 uqdstetn 20 0 1208312 9768 704 S 56.1 0.0 0:01.89 pigz 14899 uqdstetn 20 0 1208312 9768 704 S 57.5 0.0 0:01.88 pigz 14893 uqdstetn 20 0 1208312 9768 704 S 56.8 0.0 0:01.85 pigz 14919 uqdstetn 20 0 1208312 9768 704 S 55.5 0.0 0:01.85 pigz 14911 uqdstetn 20 0 1208312 9768 704 S 56.1 0.0 0:01.84 pigz 14891 uqdstetn 20 0 1208312 9768 704 S 55.8 0.0 0:01.82 pigz 14916 uqdstetn 20 0 1208312 9768 704 S 56.5 0.0 0:01.82 pigz 14884 uqdstetn 20 0 1208312 9768 704 S 54.2 0.0 0:01.79 pigz 4609 uqdstetn 20 0 34524 944 704 R 20.6 0.0 0:01.79 pigz 14902 uqdstetn 20 0 1208312 9768 704 R 54.2 0.0 0:01.76 pigz 30026 uqdstetn 20 0 4416 628 460 S 2.7 0.0 0:00.98 split 30028 uqdstetn 20 0 4416 628 460 R 2.3 0.0 0:00.77 split 30061 uqdstetn 20 0 1208484 10720 700 S 2.0 0.0 0:00.77 pigz

Desktop (please complete the following information):

$ ../ver1/zUMIs/zUMIs.sh -c -y jon.yaml Using miniconda environment for zUMIs! note: internal executables will be used instead of those specified in the YAML file!

You provided these parameters: YAML file: jon.yaml zUMIs directory: /scratch/90days/uqdstetn/build/zumis/ver1/zUMIs STAR executable STAR samtools executable samtools pigz executable pigz Rscript executable Rscript RAM limit: 50 zUMIs version 2.9.5

As an HPC administrator this isa problem.

Thanks

stetner commented 3 years ago

Later in the process:

uqdstetn 2104 $ ps -eLf | grep dstet| egrep -v ssh|bash|grep| wc -l 197

ps_-eLf_output.txt

cziegenhain commented 3 years ago

Hi,

Thanks for the feedback. The num_threads parameter is intended as a rough guide for the maximum CPU load zUMIs should use. In many stages of the pipeline there will be additional actual threads open, for example for input file decompression/compression (which also may take a bit of extra CPU use). There is potentially also some dependency on the size of the input data for certain steps. In any case, for a setting of num_threads 16, all the open threads should never be at 100% CPU! Since @sdparekh & I maintain this pipeline in our free time in parallel to our current positions, I don't expect significant changes for this behavior in the foreseeable future.

Thanks for your understanding, Christoph

stetner commented 3 years ago

OK, that makes it's use on HPC limited as either the user will need to allocate an entire machine, wasting resources as well as lengthening queue time as they will e waiting for an empty machine which is hard to get. As well, if if cgroups are implemented it will kill the users job when they exceed the requested cpus. I do see that pigz has a -p option.

Is there a way to hack zUMIs to simply change a call to 'pigz' to 'pigz -p 1' in order to not do that?

Thanks, Doug

   pigz [ -cdfhikKlLnNqrRtTz0..9,11 ] [ -b blocksize ] [ -p threads ] [ -S suffix ] [ name ...  ]
cziegenhain commented 3 years ago

I understand your concerns from the HPC management perspective. Surely one could limit the pigz parallelism by hard coding -p1 everywhere, however I'd think there will be other downstream steps where a number of threads are spawned, so I'm not sure about how successful this will be.

In the first processing step of zUMIs you'd be looking to change this line: https://github.com/sdparekh/zUMIs/blob/main/splitfq.sh#L39

Best, Christoph