Closed HenrikBengtsson closed 3 years ago
@hgputnam, I've just deployed these updates on C4, e.g.
$ module load CBI sratoolkit
$ fasterq-dump --version
"fasterq-dump" version 2.11.0
$ type fasterq-dump
fasterq-dump is a function
fasterq-dump ()
{
command fasterq-dump --temp "$(mktemp -d)" "$@"
}
Thanks. Do you think I should make listserv/slack announcements for stuff like this?
Nah. I've done some basic testing and it seems to work. Let's cross the fingers that there aren't any corner cases where this causes problem. If that happens, we'll hear about it. Not having this patch in place is probably worse; Wynton dev nodes have gone done over the last year because of this bug.
Issue
It turns out that
fasterq-dump
of the SRA Toolkit software can bring down a machine to the point wherew
andps
is unresponsive, although things such asls
works. I've troubleshooted this a bit over at UCSF Wynton.Troubleshooting
Turns out that others have reported similar problems, e.g.
The 2nd is interesting because there's a comment https://github.com/ncbi/sra-tools/issues/161#issuecomment-808294889 pointing toward BeeGDS:
which might also explain why
w
andps
doesn't work (or?)Now, the above comment also mentions:
So, the fact that
fasterq-dump
uses$PWD
for temp files could certainly explain why the BeeGFS gets hit too hard causing the above hang. I think the reason why they're using$PWD
is that they're worried about small, possibly RAM mounted/tmp
folders been filled up with their big files.BTW, I've posted some follow-up comments, which mostly are feature requests making it possible to change defaults via environment variables.
Patch
I've updated
module load CBI sratoolkit
to inject--temp "$(mktemp -d)"
so thatfasterq-dump
will default to$TMPDIR
instead of$PWD
. I do this by defining Bash function:I've verified (on my local Ubuntu machine) that
fasterq-dump SRR000001
works and evenfasterq-dump --version
(despite that--temp ...
option injected).