MCMICRO on NF Tower error with incorrect path to raw files

I am setting up MCMICRO in a deployment of Nextflow Tower.

I am running MCMICRO on exemplar-001, staged to a s3 bucket, using a AWS batch spot compute instance with the following resolved configuration - note I am not using the AWS config preset and we should likely work to add a specific Tower config where needed

params {
   in = 's3://mc2-mcmicro-project-tower-bucket/staging/exemplar-001'
   roadie = 'labsyspharm/roadie:2022-06-06'
   contPfx = ''
}

docker {
   enabled = true
   runOptions = '--cpus 0.000 -u root'
}

timeline {
   enabled = true
   file = 'timeline-25ynmqcuxBvTtS.html'
}

aws {
   region = 'us-east-1'
   client {
      uploadChunkSize = 10485760
   }
   batch {
      cliPath = '/home/ec2-user/miniconda/bin/aws'
      jobRole = 'arn:aws:iam::728882028485:role/mc2-mcmicro-project-TowerForgeBatchWorkJobRole-5KCRB8ULUE68'
      executionRole = 'arn:aws:iam::728882028485:role/mc2-mcmicro-project-TowerForgeBatchExecutionRole-19PO6Q0IID42G'
   }
}

process {
   executor = 'awsbatch'
   queue = 'TowerForge-13OhRvFugwOqYkiM2X3d8M'
}

workDir = 's3://mc2-mcmicro-project-tower-scratch/work'
runName = 'jolly_lavoisier'

tower {
   enabled = true
   endpoint = 'https://tower.sagebionetworks.org/api'
}

However the run fails with the following Nextlow console output:

N E X T F L O W  ~  version 22.04.3
Pulling labsyspharm/mcmicro ...
 downloaded from https://github.com/labsyspharm/mcmicro.git
Launching `https://github.com/labsyspharm/mcmicro` [jolly_lavoisier] DSL2 - revision: 8165b1ac3a [master]
Downloading plugin xpack-amzn@1.2.0
Monitor the execution with Nextflow Tower using this url https://tower.sagebionetworks.org/orgs/Sage-Bionetworks/workspaces/mc2-mcmicro-project/watch/25ynmqcuxBvTtS
WARN: Unable to get file attributes file: s3:/exemplar-001-cycle-06.ome.tiff -- Cause: java.lang.IllegalArgumentException: The bucket name parameter must be specified when listing objects in a bucket
Error executing process > 'registration:ashlar'

Caused by:
  Oops.. something wrong happened while creating task 'registration:ashlar' unique id -- Offending keys: [
 - type=java.util.UUID value=c3ec993a-1739-4dc1-827b-a355a6ccca8a, 
 - type=java.lang.String value=registration:ashlar, 
 - type=java.lang.String value=def imgs = lrelPath.collect{ Util.escapeForShell(it) }.join(" ")
def ilp = "--ffp $lffp --dfp $ldfp"
if (ilp == '--ffp  --dfp ') ilp = ''
"""
    ashlar $imgs ${Opts.moduleOpts(module, mcp)} $ilp -o ${sampleName}.ome.tif
    """
, 
 - type=java.lang.String value=labsyspharm/ashlar:1.17.0, 
 - type=java.lang.String value=mcp, 
 - type=java.util.LinkedHashMap value=[workflow:[start-at:registration, stop-at:quantification, qc-files:copy, tma:false, viz:false, multi-formats:{.xdce,.nd,.scan,.htd}, single-formats:{.ome.tiff,.ome.tif,.rcpnl,.btf,.nd2,.tif,.czi}, segmentation:unmicst, segmentation-recyze:false, downstream:scimap], options:[ashlar:-m 30, cypository:--model zeisscyto, ilastik:--num_channels 1, mcquant:--masks cell*.tif, naivestates:-p png], modules:[illumination:[name:basic, container:labsyspharm/basic-illumination, version:1.1.0], registration:[name:ashlar, container:labsyspharm/ashlar, version:1.17.0], dearray:[name:coreograph, container:labsyspharm/unetcoreograph, version:2.2.8, cmd:python /app/UNetCoreograph.py --outputPath ., input:--imagePath], segmentation:[[name:unmicst, container:labsyspharm/unmicst, version:2.7.4, cmd:python /app/unmicstWrapper.py --stackOutput --outputPath ., input:, channel:--channel, idxbase:1, watershed:yes]], watershed:[name:s3seg, container:labsyspharm/s3segmenter, version:1.3.18, channel:--probMapChan, idxbase:1], quantification:[name:mcquant, container:labsyspharm/quantification, version:1.5.2], downstream:[[name:scimap, container:labsyspharm/scimap, version:0.17.7, cmd:scimap-mcmicro -o ., input:]], viz:[name:autominerva, container:labsyspharm/auto-minerva, version:2022-06-06]]], 
 - type=java.lang.String value=module, 
 - type=java.util.LinkedHashMap value=[name:ashlar, container:labsyspharm/ashlar, version:1.17.0], 
 - type=java.lang.String value=sampleName, 
 - type=java.lang.String value=exemplar-001, 
 - type=java.lang.String value=lraw, 
 - type=nextflow.util.ArrayBag value=[FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/raw/exemplar-001-cycle-06.ome.tiff, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/raw/exemplar-001-cycle-06.ome.tiff, stageName:exemplar-001-cycle-06.ome.tiff), FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/raw/exemplar-001-cycle-07.ome.tiff, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/raw/exemplar-001-cycle-07.ome.tiff, stageName:exemplar-001-cycle-07.ome.tiff), FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/raw/exemplar-001-cycle-08.ome.tiff, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/raw/exemplar-001-cycle-08.ome.tiff, stageName:exemplar-001-cycle-08.ome.tiff)], 
 - type=java.lang.String value=lrelPath, 
 - type=java.util.ArrayList value=[exemplar-001-cycle-06.ome.tiff, exemplar-001-cycle-07.ome.tiff, exemplar-001-cycle-08.ome.tiff], 
 - type=java.lang.String value=lffp, 
 - type=nextflow.util.ArrayBag value=[FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-06-ffp.tif, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-06-ffp.tif, stageName:exemplar-001-cycle-06-ffp.tif), FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-07-ffp.tif, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-07-ffp.tif, stageName:exemplar-001-cycle-07-ffp.tif), FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-08-ffp.tif, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-08-ffp.tif, stageName:exemplar-001-cycle-08-ffp.tif)], 
 - type=java.lang.String value=ldfp, 
 - type=nextflow.util.ArrayBag value=[FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-06-dfp.tif, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-06-dfp.tif, stageName:exemplar-001-cycle-06-dfp.tif), FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-07-dfp.tif, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-07-dfp.tif, stageName:exemplar-001-cycle-07-dfp.tif), FileHolder(sourceObj:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-08-dfp.tif, storePath:/mc2-mcmicro-project-tower-bucket/staging/exemplar-001/illumination/exemplar-001-cycle-08-dfp.tif, stageName:exemplar-001-cycle-08-dfp.tif)], 
 - type=java.lang.String value=$, 
 - type=java.lang.Boolean value=true]

From the warning on line 7 (WARN: Unable to get file attributes file: s3:/exemplar-001-cycle-06.ome.tiff) the ashlar process is receiving as input the basename of the first tile preffixed with s3:/, when I think it should be receiving the full uri of the tile (s3://mc2-mcmicro-project-tower-bucket/staging/exemplar-001/raw/exemplar-001-cycle-06.ome.tiff)

I think this might be related to lines 70-72 of main.nf where some path construction is performed.

Here we assemble tuples of 1) path to stage for each raw image (might be a directory) and 2) relative path to the main file for each image. Processes must input the first as a path and the second as a val to avoid incorrect or redundant file staging. They must also only use the second (relative) path to construct pathnames for scripts etc. mcmicro.Util.escapePathForShell must be used when interpolating these paths into script strings, as we are bypassing the normal way that paths are passed to channels which handles this escaping automatically.

Looking forward to working with you on a fix... and to contributing a user-guide for setup on Nextflow Tower once working.

labsyspharm / mcmicro

MCMICRO on NF Tower error with incorrect path to raw files #425