TB-profiler issue v2

[MarcNiebel]

          Hi Tom, unfortunately this is still not working. Any ideas?

error_log_lodestone_180624.txt

Originally posted by @MarcNiebel in https://github.com/Pathogen-Genomics-Cymru/lodestone/issues/58#issuecomment-2176038050

[MarcNiebel]

This is the tb-profiler error log if it is of any use. tbprofiler.errlog.txt

[WhalleyT]

Hi again Marc, thanks for the continued efforts giving it a go!

That's a new one to me, Just to double check, are the temporary files being written successfully? e.g. is something being written to ./bcftools.0EDiR4 like it says in the log?

[MarcNiebel]

Hi Tom, Using find . -name "bcftools*" in that specific working directory i.e. 8c/ yields no results. This is in contrast with the parent directory work/ which does. Is there another location I should be looking?

[WhalleyT]

It might be a permissions error if the file isn't showing up. Are you running it in Singularity and also is the input originally from a fastq or bam? I'll try and re-create on my end but so far I haven't seen a sample fail like that so I'll let you know.

edit: yes seems all okay on Docker. I've been having issues with tmp locations in Singularity with other parts of the pipeline so suspect it's that

[MarcNiebel]

Ok thanks for looking into this. This is the command I am using nextflow run main.nf -profile singularity --input_dir MTB_WHO_sequences --pattern '*_R{1,2}_001.fastq.gz'

[MarcNiebel]

Hi Tom, So that is actually what is happening when using singularity. The incorrect files are being grouped together so that is probably explaining the error.

[WhalleyT]

Ah right okay! As a quick fix if you export TMPDIR="<new_tmp>" or export SINGULARITY_TMPDIR="<new_tmp>" to somewhere writable that should work, at least that was enough on my end :). The new version of tb-profiler allows you to pass a temporary directory as an argument (as do some of the other functions in the module) so I can make a parameter for it in the workflow so the user doesn't have to mess around with that

[MarcNiebel]

No worries. I will use Docker in the meantime whilst you sort the temporary directory as an argument .

[WhalleyT]

Hi again! I've hardcoded a tmp directory for gatk and tb-profiler which will be created in the working directory and should be automounted by Singularity. I'll add them into the other places if this works, but it's probably not a priority as those containers don't have permission issues (they're all installed with pip et al. if that affects it?).

Please let me know how it goes, and if it works I'll add it to the rest of the pipeline.

If this doesn't work I will try rebuilding the container and installing everything from scratch. I've tried to avoid this as tb-profiler's authors don't recommend it, but it might be more stable.

[MarcNiebel]

Hi Tom, thanks for that. Will try it out in the coming days and keep you informed of my progress

[WhalleyT]

Great, fingers crossed. Apologies as well last night I forgot to actually make the commit so it's on now.

[MarcNiebel]

Have tested it out on two samples and worked successfully. Checked the location of all files and they are grouped correctly 👍 Thank you for sorting this out.

[WhalleyT]

Great stuff! Glad it worked. I'll close the issue for now but let me know if anything else crops up :smile: