pkg-test stuck in loop

[drernie]

WARN: Path integrity check failed because the following file has been deleted: /Users/ernest/GitHub/nf-quilt/work/stage-8c398970-deda-4987-b96d-bc94f7ad5feb/d5/ffd6d402ef055564d806910a3bfaca/data/atlantic-storms.csv -- make sure to not run more than one nextflow instance using the same work directory

[drernie]

Possibly a recursive debug message?

Jan-31 09:19:24.417 [Actor Thread 5] DEBUG nextflow.quilt.jep.QuiltPackage - done: installed into /var/folders/tz/8q322ht10qzf9pswh01zv6880000gp/T/QuiltPackage17819673345642654279/QuiltPackage.quilt_example_examples_hurdat) Jan-31 09:19:24.429 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - getFileName[quilt-example#package=examples%2fhurdat]: paths=[] Jan-31 09:19:24.429 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat Jan-31 09:19:24.435 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat&path=data Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat&path=notebooks Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltFileSystem - QuiltFileAttributes QuiltFileSystem.readAttributes(quilt-example#package=examples%2fhurdat&path=data) Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltPath - Creating QuiltPath: quilt-example#package=examples%2fhurdat&path=data%2fatlantic-storms.csv Jan-31 09:19:24.436 [Actor Thread 5] DEBUG nextflow.quilt.nio.QuiltFileSystem - QuiltFileAttributes QuiltFileSystem.readAttributes(quilt-example#package=examples%2fhurdat&path=data%2fatlantic-storms.csv) Jan-31 09:19:24.437 [Actor Thread 5] WARN nextflow.file.FilePorter - Path integrity check failed because the following file has been deleted: /Users/ernest/GitHub/nf-quilt/work/stage-03462935-6c0c-4001-b27d-7e825ba4fe5f/ca/94a4c3caa5355bce05ed5aaa9104e1/data/atlantic-storms.csv -- make sure to not run more than one nextflow instance using the same work directory

[drernie]

The sequence seems to be re-creating a QuiltPackage for each new Path (and invocation).

Not quite. The first time, the QuiltPathe creates and installs the QuiltPackage.

Then somebody calls getFileName

Then we get stuck?

[drernie]

Ah. The install fix is what's breaking things. Commenting that out solves the problem. But why? Only happens once.

[drernie]

Hmm. Each error is a different work directory. Is the problem that we are reusing Packages across different local paths? Even if we disable auto-delete?

Feb-05 19:35:44.960 [Actor Thread 5] WARN  nextflow.file.FilePorter - Path integrity check failed because the following file has been deleted: /Users/ernest/GitHub/nf-quilt/work/stage-5b7854a0-ed50-42b6-b019-65d2e7c2be48/02/7adf167fa0a7964aee2088fd01ffb4/data/atlantic-storms.csv -- make sure to not run more than one nextflow instance using the same work directory

[drernie]

Allowing duplicate installs did not fix the problem.

[drernie]

Okay, this is totally my bad. Using a NON-path input is the source of the problem:

params.input = 'quilt+s3://quilt-example#package=examples/hurdat/'

Next question: can I reproduce this issue in a unit test?

[drernie]

Sort of. It appears due to disabling:

        // if (isJustPackage()) { return this } // IF DIRECTORY
        if (isJustPackage()) { return this } // IF DIRECTORY

I disabled that because when it is enabled, I fail three tests:

• QuiltNioTest > should read from a path FAILED java.lang.NullPointerException at QuiltNioTest.groovy:77

• QuiltNioTest > should iterate over package folders/files FAILED org.spockframework.runtime.ConditionNotSatisfiedError at QuiltNioTest.groovy:250

• QuiltPathTest > should validate getFileName > should validate getFileName [path: bucket, fileName: , #0] FAILED org.spockframework.runtime.SpockComparisonFailure at QuiltPathTest.groovy:120

Naively, it seemed that disabling it made everything magically work. But the make pkg-test failure implies otherwise, and it is time to take that seriously.

[drernie]

Solved with v0.7.12

[drernie]

With a fresh branch, re-enabling that line only gives the last error.

Which had a trivial one-word fix.

The other two were just artifacts of reading an unpinned package...