Properly handle CopyMoveHelper#copyToForeignTarget

[drernie]

quilt+s3://quilt-ernest-staging?Name=quilt&Owner=Kevin+Moore&Date=2023-03-07&Type=CRISPR&Notebook+URL=http%3A%2F%2Fexample.com#package=test/hurdat&path=. ./launch.sh run ./main.nf -profile standard -plugins nf-quilt --outdir "quilt+s3://quilt-ernest-staging?Name=quilt&Owner=Kevin+Moore&Date=2023-03-07&Type=CRISPR&Notebook+URL=http%3A%2F%2Fexample.com#package=test/hurdat&path=." N E X T F L O W ~ version 23.06.0-edge Launching ./main.nf [chaotic_boltzmann] DSL2 - revision: 08f61c1c67 [- ] process > transfer - done

nextflow.exception.ProcessStageException: Can't stage file quilt+s3://quilt-example#package=examples%2fhurdat -- file does not exist at nextflow.file.FilePorter$FileTransfer.stageForeignFile(FilePorter.groovy:332) at nextflow.file.FilePorter$FileTransfer.run(FilePorter.groovy:301) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: java.nio.file.NoSuchFileException: /Users/quilt/Documents/GitHub/nf-quilt/work/stage-845ccbaa-3d21-43db-a1ec-c2dba631302d/14/617f6197b1ce64b02d272966333aaa/quilt-example#package=examples%2fhurdat/data/atlantic-storms.csv at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:218) at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:484) at java.base/java.nio.file.Files.newOutputStream(Files.java:228) at java.base/java.nio.file.Files.copy(Files.java:3160) at nextflow.file.CopyMoveHelper.copyFile(CopyMoveHelper.java:90) at nextflow.file.CopyMoveHelper$1.visitFile(CopyMoveHelper.java:134) at nextflow.file.CopyMoveHelper$1.visitFile(CopyMoveHelper.java:108) at java.base/java.nio.file.Files.walkFileTree(Files.java:2811) at nextflow.file.CopyMoveHelper.copyDirectory(CopyMoveHelper.java:140) at nextflow.file.CopyMoveHelper.copyToForeignTarget(CopyMoveHelper.java:169) at nextflow.file.FileHelper.copyPath(FileHelper.groovy:961) at nextflow.file.FilePorter$FileTransfer.stageForeignFile0(FilePorter.groovy:354) [- ] process > transfer - done ERROR ~ Error executing process > 'transfer (1)'

Caused by: Can't stage file quilt+s3://quilt-example#package=examples%2fhurdat -- file does not exist

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

-- Check '.nextflow.log' file for details java.lang.InterruptedException at java.base/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1048) at java.base/java.util.concurrent.Semaphore.acquire(Semaphore.java:475) at groovyx.gpars.dataflow.operator.ForkingDataflowOperatorActor.forwardPoisonPill(ForkingDataflowOperatorActor.java:69) at groovyx.gpars.dataflow.operator.DataflowProcessorActor.checkPoison(DataflowProcessorActor.java:114) at groovyx.gpars.dataflow.operator.DataflowOperatorActor.onMessage(DataflowOperatorActor.java:83) at groovyx.gpars.actor.impl.SDAClosure$1.call(SDAClosure.java:43) at groovyx.gpars.actor.AbstractLoopingActor.runEnhancedWithoutRepliesOnMessages(AbstractLoopingActor.java:293) [- ] process > transfer - done ERROR ~ Error executing process > 'transfer (1)'

Caused by: Can't stage file quilt+s3://quilt-example#package=examples%2fhurdat -- file does not exist

Tip: when you have fixed the problem you can continue the execution adding the option -resume to the run command line

-- Check '.nextflow.log' file for details

make: *** [pkg-test] Error 1

[drernie]

Amusingly, it looks like we broke "CopyMoveHelper#copyToForeignTarget" -- which this comment promised to prevent!

class QuiltFileAttributesView implements BasicFileAttributeView {
    //....
    /**
     * This API is implemented is not supported but instead of throwing an exception just do nothing
     * to not break the method {@link java.nio.file.CopyMoveHelper#copyToForeignTarget(...)}
     *
     * @param lastModifiedTime
     * @param lastAccessTime
     * @param createTime
     * @throws IOException
     */
    @Override
    void setTimes(FileTime lastModifiedTime, FileTime lastAccessTime, FileTime createTime) throws IOException {
    // TBD
    }

}

[drernie]

Log file

Sep-05 13:43:39.872 [FileTransfer-1] DEBUG nextflow.file.FilePorter - Copying foreign file quilt+s3://quilt-example#package=examples%2fhurdat to work dir: /Users/quilt/Documents/GitHub/nf-quilt/work/stage-a2996f23-58a3-45ae-a7fa-be35c661b9a4/0b/36d858ee5bd1f81c562742c10902fd/quilt-example#package=examples%2fhurdat
Sep-05 13:43:40.079 [FileTransfer-1] DEBUG nextflow.quilt.jep.QuiltPackage - QuiltPackage.install: already installed: /var/folders/rr/hp1w0hxd07lgq1y8k9dmnrwr0000gq/T/QuiltPackage7609402337477599622/QuiltPackage.quilt_example_examples_hurdat
Sep-05 13:43:40.080 [FileTransfer-1] DEBUG nextflow.quilt.jep.QuiltPackage - QuiltPackage.install: already installed: /var/folders/rr/hp1w0hxd07lgq1y8k9dmnrwr0000gq/T/QuiltPackage7609402337477599622/QuiltPackage.quilt_example_examples_hurdat
Sep-05 13:43:40.080 [FileTransfer-1] DEBUG n.quilt.nio.QuiltFileSystemProvider -    Reading from: /var/folders/rr/hp1w0hxd07lgq1y8k9dmnrwr0000gq/T/QuiltPackage7609402337477599622/QuiltPackage.quilt_example_examples_hurdat/data/atlantic-storms.csv
Sep-05 13:43:40.080 [FileTransfer-1] DEBUG n.quilt.nio.QuiltFileSystemProvider - FileChannel.open: sun.nio.ch.FileChannelImpl@55543023
Sep-05 13:43:40.084 [Actor Thread 5] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for
  task: name=transfer (1); work-dir=null
  error [nextflow.exception.ProcessStageException]: Can't stage file quilt+s3://quilt-example#package=examples%2fhurdat -- file does not exist
Sep-05 13:43:40.103 [Actor Thread 5] ERROR nextflow.processor.TaskProcessor - Error executing process > 'transfer (1)'

[drernie]

Looks like I need to have an explicit download method to AVOID the copyToForeignTarget

            if( sourceProvider instanceof FileSystemTransferAware && sourceProvider.canDownload(source, target)){
                sourceProvider.download(source, target, options)
            }
            else if( targetProvider instanceof FileSystemTransferAware && targetProvider.canUpload(source, target)) {
                targetProvider.upload(source, target, options)

if your FS provider implements that interface then it will call the download method, otherwise it will call copyToForeignTarget() , which you can keep following to see which methods they use from your provider

[drernie]

Fixed. At least well enough to move forward.