Closed drernie closed 3 months ago
Jan-31 09:19:24.417 [Actor Thread 5] DEBUG nextflow.quilt.jep.QuiltPackage - done: installed into /var/folders/tz/8q322ht10qzf9pswh01zv6880000gp/T/QuiltPackage17819673345642654279/QuiltPackage.quilt_example_examples_hurdat) Jan-31 09:19:24.429 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - getFileName[quilt-example#package=examples%2fhurdat]: paths=[] Jan-31 09:19:24.429 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat Jan-31 09:19:24.435 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat&path=data Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat&path=notebooks Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltFileSystem - QuiltFileAttributes QuiltFileSystem.readAttributes(quilt-example#package=examples%2fhurdat&path=data) Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat&path=data%2fatlantic-storms.csv Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltFileSystem - QuiltFileAttributes QuiltFileSystem.readAttributes(quilt-example#package=examples%2fhurdat&path=data%2fatlantic-storms.csv) Jan-31 09:19:24.437 [Actor Thread 5] WARN nextflow.file.FilePorter - Path integrity check failed because the following file has been deleted: /Users/ernest/GitHub/nf-quilt/work/stage-03462935-6c0c-4001-b27d-7e825ba4fe5f/ca/94a4c3caa5355bce05ed5aaa9104e1/data/atlantic-storms.csv -- make sure to not run more than one nextflow instance using the same work directory
The sequence seems to be re-creating a QuiltPackage for each new Path (and invocation).
Not quite. The first time, the QuiltPathe creates and installs the QuiltPackage.
Then somebody calls getFileName
Then we get stuck?
Ah. The install fix is what's breaking things. Commenting that out solves the problem. But why? Only happens once.
Hmm. Each error is a different work directory. Is the problem that we are reusing Packages across different local paths? Even if we disable auto-delete?
Feb-05 19:35:44.960 [Actor Thread 5] WARN nextflow.file.FilePorter - Path integrity check failed because the following file has been deleted: /Users/ernest/GitHub/nf-quilt/work/stage-5b7854a0-ed50-42b6-b019-65d2e7c2be48/02/7adf167fa0a7964aee2088fd01ffb4/data/atlantic-storms.csv -- make sure to not run more than one nextflow instance using the same work directory
Allowing duplicate installs did not fix the problem.
Okay, this is totally my bad. Using a NON-path input is the source of the problem:
params.input = 'quilt+s3://quilt-example#package=examples/hurdat/'
Next question: can I reproduce this issue in a unit test?
Sort of. It appears due to disabling:
// if (isJustPackage()) { return this } // IF DIRECTORY
if (isJustPackage()) { return this } // IF DIRECTORY
I disabled that because when it is enabled, I fail three tests:
QuiltNioTest > should read from a path FAILED
java.lang.NullPointerException at QuiltNioTest.groovy:77
QuiltNioTest > should iterate over package folders/files FAILED
org.spockframework.runtime.ConditionNotSatisfiedError at QuiltNioTest.groovy:250
QuiltPathTest > should validate getFileName > should validate getFileName [path: bucket, fileName: , #0] FAILED
org.spockframework.runtime.SpockComparisonFailure at QuiltPathTest.groovy:120
Naively, it seemed that disabling it made everything magically work. But the make pkg-test
failure implies otherwise, and it is time to take that seriously.
Solved with v0.7.12
With a fresh branch, re-enabling that line only gives the last error.
Which had a trivial one-word fix.
The other two were just artifacts of reading an unpinned package...