How can I request cellbender to limit/use more threads/cores?

[angelasanzo]

Hi there,

I have tried to find a way to limit or increase the number of threads that is using, but have not found a result. Any argument that can be provided?

Thank you very much.

Angela

[sjfleming]

Hi @angelasanzo , unfortunately, I don't know of a good way to make this happen. CellBender does not have an input argument to enable this currently.

(If anybody else knows how to do this, please post here!) It looks like there might be some ways to get pytorch to run using multiple threads on CPU. Unfortunately I have never tried this, and don't know the benefits / limitations.

I'm guessing you are running on a CPU? Is that right?

[sjfleming]

Note to self: it is possible this is as simple as

torch.set_num_threads(n)

Test this.

[sjfleming]

Potentially useful example by Bo Li and collaborators here https://github.com/lilab-bcb/harmony-pytorch/blob/436e53cb3db1e51f3e23eb63fe2f3863e9e31f2d/harmony/harmony.py#L115

[angelasanzo]

Thanks a lot for your work! Yes, we are using CPU as CUDA cannot be performed under our graphics card.

[sjfleming]

Unfortunately, I tested

import psutil
n_jobs = psutil.cpu_count(logical=False)  # get physical cores
if n_jobs is None:
    n_jobs = psutil.cpu_count(logical=True)  # if undetermined, use logical cores instead
torch.set_num_threads(n_jobs)

and it doesn't make any difference in terms of runtime. So it's not that simple...

Will keep thinking...

[sjfleming]

Eh, okay, it's possible that using the number of logical cores is better. But I'm only seeing a speedup of like 4% :(

Still, I might include it as an input argument in the future. Default will use the number of logical cores.

[sjfleming]

Closed by #238 The input argument is --cpu-threads. But limited testing shows me it unfortunately does not seem to make a big difference.

[BradBalderson]

Hey @sjfleming,

Maybe off topic but it seems that the --cpu-threads is causing a stall when writing out the output? I ran cellbender like this:

cellbender remove-background --cuda --estimator-multiple-cpu --cpu-threads 21 --epochs 40 --checkpoint-mins 60 --input {sample_input_counts} --output {sample_out_dir}

It gives me this output: cellbender:remove-background: Command: cellbender remove-background --cuda --estimator-multiple-cpu --cpu-threads 21 --epochs 40 --checkpoint-mins 60 --input /home/jovyan/data4/UCSD_Oxy-multiome/Oxy-multiome//FTL_702_M957/outs/raw_feature_bc_matrix.h5 --output /home/jovyan/data4/bbalderson_runAI/rat_multiomics/data/cellbender_out//FTL_702_M957/ cellbender:remove-background: CellBender 0.3.0 cellbender:remove-background: (Workflow hash 8a1259eac2) cellbender:remove-background: 2023-12-04 17:46:12 cellbender:remove-background: Running remove-background cellbender:remove-background: Loading data from /home/jovyan/data4/UCSD_Oxy-multiome/Oxy-multiome//FTL_702_M957/outs/raw_feature_bc_matrix.h5 cellbender:remove-background: CellRanger v3 format cellbender:remove-background: Features in dataset: 33294 Gene Expression, 96665 Peaks cellbender:remove-background: Trimming features for inference. cellbender:remove-background: 122701 features have nonzero counts. cellbender:remove-background: Prior on counts for cells is 4609 cellbender:remove-background: Prior on counts for empty droplets is 72 cellbender:remove-background: Excluding 11105 features that are estimated to have <= 0.1 background counts in cells. cellbender:remove-background: Including 111596 features in the analysis. cellbender:remove-background: Trimming barcodes for inference. cellbender:remove-background: Excluding barcodes with counts below 36 cellbender:remove-background: Using 10437 probable cell barcodes, plus an additional 24850 barcodes, and 49210 empty droplets. cellbender:remove-background: Largest surely-empty droplet has 126 UMI counts. cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmpjiir3c4e cellbender:remove-background: Successfully unpacked tarball to /tmp/tmpjiir3c4e /tmp/tmpjiir3c4e/8a1259eac2_params.pyro /tmp/tmpjiir3c4e/8a1259eac2_random.cuda /tmp/tmpjiir3c4e/8a1259eac2_optim.torch /tmp/tmpjiir3c4e/posterior.h5 /tmp/tmpjiir3c4e/8a1259eac2_args.npy /tmp/tmpjiir3c4e/8a1259eac2_optim.pyro /tmp/tmpjiir3c4e/8a1259eac2_model.torch /tmp/tmpjiir3c4e/8a1259eac2_train.loaderstate /tmp/tmpjiir3c4e/8a1259eac2_test.loaderstate /tmp/tmpjiir3c4e/8a1259eac2_random.pyro cellbender:remove-background: Loaded partially-trained checkpoint from ckpt.tar.gz cellbender:remove-background: Checkpoint loaded successfully. cellbender:remove-background: Running inference... cellbender:remove-background: 2023-12-04 17:47:38 cellbender:remove-background: Inference procedure complete. cellbender:remove-background: Attempting to unpack tarball "ckpt.tar.gz" to /tmp/tmpheuaf78f cellbender:remove-background: Successfully unpacked tarball to /tmp/tmpheuaf78f /tmp/tmpheuaf78f/8a1259eac2_params.pyro /tmp/tmpheuaf78f/8a1259eac2_random.cuda /tmp/tmpheuaf78f/8a1259eac2_optim.torch /tmp/tmpheuaf78f/posterior.h5 /tmp/tmpheuaf78f/8a1259eac2_args.npy /tmp/tmpheuaf78f/8a1259eac2_optim.pyro /tmp/tmpheuaf78f/8a1259eac2_model.torch /tmp/tmpheuaf78f/8a1259eac2_train.loaderstate /tmp/tmpheuaf78f/8a1259eac2_test.loaderstate /tmp/tmpheuaf78f/8a1259eac2_random.pyro cellbender:remove-background: Loaded pre-computed posterior from posterior.h5 cellbender:remove-background: 2023-12-04 17:48:53

cellbender:remove-background: Saved summary plots as /home/jovyan/data4/bbalderson_runAI/rat_multiomics/data/cellbender_out//FTL_702_M957/.pdf cellbender:remove-background: Saved cell barcodes in /home/jovyan/data4/bbalderson_runAI/rat_multiomics/data/cellbender_out//FTL_702_M957/_cell_barcodes.csv cellbender:remove-background: Computing target noise counts per gene for MCKP estimator cellbender:remove-background: Using MCKP noise targets computed for FPR 0.01 cellbender:remove-background: Computing denoised counts using mckp estimator cellbender:remove-background: Dividing dataset into chunks of genes

It seems to stall here, 21 threads are started, but they are using 0% CPU, and it does not write any output beyond the .pdf and the _cell_barcodes.tsv. I am about to try not setting the number of threads to see if it helps.