multiple parallel run - checkpoint file

[muntajihad]

Hi there, I am using a bash script to run remove-background on multiple samples "in parallel". The problem is that only one ckpt.tar.gz file is generated and it is overwrited again and agian by all samples, as a result I am getting errors and no final output is generated.

the code I use is simple :

id=${1}/outs/raw_feature_bc_matrix.h5
cellbender remove-background \
            --cuda \
            --input $id \
            --output ${id/.h5/_cbender.h5}

Thanks

[sjfleming]

Hi @muntajihad , I completely agree with you, and there is a bit more information about this here: https://github.com/broadinstitute/CellBender/issues/241#issuecomment-1675016878

CellBender is currently saving the ckpt.tar.gz file in the directory from which the command was run. So, until I make a change, one way to work around this is to do something like the following:

id=${1}/outs/raw_feature_bc_matrix.h5
cbdir=${1}/outs/cellbender
cbout=${cbdir}/raw_feature_bc_matrix_cbender.h5
mkdir -p ${cbdir}
cd ${cbdir}
cellbender remove-background \
            --cuda \
            --input ${id} \
            --output ${cbout}

(Haven't tested the above code snippet to ensure it works, but the idea is that you could run each command from its own "output directory", so that there will be separate places for the checkpoint files to live.)

[muntajihad]

Oh I see. Thanks a lot for your quick answer @sjfleming. solution idea : It would be easy and practical if we can give a name to the ckpt.tar.gz generated. (e.g. mysample.tar.gz)

All the best,

[sjfleming]

Yes, I also think that is a very reasonable idea.

[shahrozeabbas]

Hi, was wondering if this edit would be included in the next update? If so, is there a time until the next update? @sjfleming

I'm just curious as it would be very helpful, currently using the workaround you've suggested, thanks!

[nfancy]

https://github.com/broadinstitute/CellBender/issues/290#issuecomment-1966572216