search

nf-core / sarek

Analysis pipeline to detect germline or somatic variants (pre-processing, variant calling and annotation) from WGS / targeted sequencing
https://nf-co.re/sarek
MIT License
351 stars 386 forks source link

How to remove temp dir(work dir) in one command #443

Closed parkjaeming closed 2 years ago

parkjaeming commented 2 years ago

Hello, I am studying bioinformatics with your sarek pipeline and it has been helpful for me so far! While using your sarek, I have faced a problem that needs your attention. I want to properly deal with the disc space by removing temp dir(work dir) in one command line like below! nextflow run nf-core/sarek \ -r 2.7.1 \ -profile test, docker \ **--clean true** So, I tried using hidden_params parameter, but there were no parameters related with my concern about disc. Can you recommend any other options? I would really appreciate your precious piece of advice.

priesgo commented 2 years ago

Probably scratch=true in your profile will help with the cleanup, see https://www.nextflow.io/docs/latest/process.html#scratch

I think the current recommendation is that cleaning up is handled by how you use/configure nextflow, but not in the particular workflows. I may be wrong, there have been some discussions about this issue in other channels before.

FriederikeHanssen commented 2 years ago

Hi @parkjaeming ! This is something that needs to be adressed at a nextflow level, if I understand your problem directly. You want to automatically delete the work dir, right? This you could do after each run with nextflow clean. Here is some docu: https://www.nextflow.io/docs/latest/cli.html#commands . There is a longer discussion on how to deal with large intermediate file going on here : https://github.com/nextflow-io/nextflow/issues/452

FriederikeHanssen commented 2 years ago

And maybe this could be relevant too: https://github.com/nextflow-io/nextflow/issues/649 cleanup=true should help as well, this won't allow resuming the workflow anymore though

priesgo commented 2 years ago

Is that cleanup=true still working? I read somewhere they may remove it...

FriederikeHanssen commented 2 years ago

possible, I couldn't find anything about either scenario. I guess easiest would be to try it out

parkjaeming commented 2 years ago

Thank you for all your attention! I also heard that cleanup=true is not working well... It seems that the best way is to clean up the temp file manually after the all pipelines...

maxulysse commented 2 years ago

I'd recommend following the issue linked by @FriederikeHanssen. But at the moment, I agree that manual cleanup is the best way