search

broadinstitute / picard

A set of command line tools (in Java) for manipulating high-throughput sequencing (HTS) data and formats such as SAM/BAM/CRAM and VCF.
https://broadinstitute.github.io/picard/
MIT License
963 stars 367 forks source link

Picard RevertSam java.nio.file.NoSuchFileException #1000

Closed sooheelee closed 6 years ago

sooheelee commented 6 years ago

This is the third user who observes a NoSuchFileException. Anyone else observe such an error AND/OR know what is going on?

One user was able to observe a successful run without the error that was occurring on their cluster by repeatedly trying the same command. They do not know what solved the issue.

I have asked followup questions in the forum thread. I am putting this question out there now. Please check the forum thread for more information. Thanks.

Hi, I have the same problem when I use the FastqToSam to convert my fastqfiles to sam files, I use the latest picard version.

I use the command line below: java -jar picard.jar FastqToSam F1=94210_CGTACTAG_S2_L001_R1_001.fastq F2=94210_CGTACTAG_S2_L001_R2_001.fastq O=fastq_to_bam.bam USE_SEQUENTIAL_FASTQS=true QUALITY_FORMAT=Illumina ALLOW_AND_IGNORE_EMPTY_LINES=true USE_JDK_DEFLATER=true USE_JDK_INFLATER=true SM=for_tool_testing

And it has the bug like these at htsjdk.samtools.util.IOUtil$DeletePathThread.run(IOUtil.java:374) Caused by: java.nio.file.NoSuchFileException: /tmp/yingma/sortingcollection.2893939607853080593.tmp at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:244) at sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:103) at java.nio.file.Files.delete(Files.java:1126) at htsjdk.samtools.util.IOUtil$DeletePathThread.run(IOUtil.java:372) Exception in thread "Thread-128" htsjdk.samtools.util.RuntimeIOException: java.nio.file.NoSuchFileException: /tmp/yingma/sortingcollection.2692803932741015812.tmp

I don't know what are the problems. Could you please help me?

This Issue was generated from your forums

sooheelee commented 6 years ago

Will tag @yfarjoun to delegate.

sooheelee commented 6 years ago

One user confirms exception happens only with Picard v2.15.0 and NOT with v2.14.1.

magicDGS commented 6 years ago

This was an error that I introduced without realizing in htsjdk, but tje latest version has a bugfix already. Sorry for the inconvenience...

sooheelee commented 6 years ago

Thanks for letting us know @magicDGS. I will convey to users.

sooheelee commented 6 years ago

Hey @magicDGS, I have a user reporting the same bug for Picard 2.16. It occurs sporadically. Any insight?

lbergelson commented 6 years ago

@sooheelee @magicDGS this should be fixed by #1007

magicDGS commented 6 years ago

Requires https://github.com/broadinstitute/picard/pull/1007 to be included. It just got accepted recently, so a new release will solve the issue.

lbergelson commented 6 years ago

@dianekaplan A new release would be a good idea.. It's a pretty major, annoying, bug.

sooheelee commented 6 years ago

Thanks for this update.

sooheelee commented 6 years ago

This issue has popped up again, this time for Picard v2.17.3.

https://gatkforums.broadinstitute.org/gatk/discussion/comment/46512#Comment_46512

magicDGS commented 6 years ago

Are you sure that it is exactly the same exception? Anyway, there is already in HTSJDK a more robust fix (https://github.com/samtools/htsjdk/pull/1037) - maybe if it is incorporated into picard will solve the issue. Otherwise, it will be helpful to know the exception class to catch it on the shutdown hook...

sooheelee commented 6 years ago

@magicDGS, actually, user has come back with the stack trace and it appears a different error:

...
INFO 2018-02-28 12:37:02 MergeSamFiles Processed 10,000,000 records. Elapsed time: 00:01:10s. Time for last 1,000,000: 6s. Last read position: chr1:235,728,122
/var/spool/slurmd/job5614122/slurm_script: line 27: 42247 Killed java -jar /cvmfs/soft.mugqic/CentOS6/software/picard/picard-2.17.3/picard.jar MergeSamFiles $file O=/home/bemmo/projects/rrg-jdrouin/bemmo/IAD/results/variants/AB-Hu-Ex-A107/AB-Hu-Ex-A107.bam USE_JDK_DEFLATER=true USE_JDK_INFLATER=true
12:37:10.830 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/cvmfs/soft.mugqic/CentOS6/software/picard/picard-2.17.3/picard.jar!/com/intel/gkl/native/libgkl_compression.so

Looks like they may be using a SLURM cluster.

sooheelee commented 6 years ago

I've moved this user's issue to https://github.com/broadinstitute/picard/issues/1129.