bioformats2raw breaks with docker (24.0.2) in Nextflow

[BioinfoTongLI]

bioformats2raw conversion with --compression error in the latest nextflow update (22.10.+). Temporary solution: Specify NXF_VER=22.04.5 in the ENV to enforce nextflow version for as walk-around.

[davehorsfall]

Hey @BioinfoTongLI, just chatting with Daniela about this. We're not using flags in the pipeline source, so we wondered how is the --compression flag being added in your use case?

[BioinfoTongLI]

Oh yea, wanted to save some space while converting... But actually it was to record which command line went wrong. But I've just tested again and it bugs with and without --compression. It's more a general bioformats2raw + 22.10.+ problem

[prete]

I fail to see how bioformats2raw and nextflow may not be working. What error are you getting?

[BioinfoTongLI]

It seems to me the last line

WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Unknown option: '-ue'

and bf2raw just shows the help message and this is bf2rwa option irrelevant.

Usage: <main class> [-p] [--no-hcs] [--[no-]nested] [--no-ome-meta-export]
                    [--no-root-group] [--overwrite]
                    [--use-existing-resolutions] [--version] [--debug
                    [=<logLevel>]] [--extra-readers[=<extraReaders>[,
                    <extraReaders>...]]]... [--options[=<readerOptions>[,
                    <readerOptions>...]]]... [-s[=<seriesList>[,
                    <seriesList>...]]]...
                    [--additional-scale-format-string-args=<additionalScaleForma
                    tStringArgsCsv>] [-c=<compressionType>]
                    [--dimension-order=<dimensionOrder>]
                    [--downsample-type=<downsampling>]
                    [--fill-value=<fillValue>] [-h=<tileHeight>]
                    [--max_cached_tiles=<maxCachedTiles>]
                    [--max_workers=<maxWorkers>]
                    [--memo-directory=<memoDirectory>]
                    [--pixel-type=<outputPixelType>]
                    [--pyramid-name=<pyramidName>] [-r=<pyramidResolutions>]
                    [--scale-format-string=<scaleFormatString>]
                    [--target-min-size=<minSize>] [-w=<tileWidth>]
                    [-z=<chunkDepth>]
                    [--compression-properties=<String=Object>]...
                    [--output-options=<String=String>[\|<String=String>...]]...
                    <inputPath> <outputLocation>

[prete]

What compressionare you using --compression=xxx ? Do you have libblosc installed in your OS? (sudo apt-get install libblosc1)

I just tried:

process bioformats2raw {
  debug true

  shell:
  '''
  bioformats2raw --compression=zlib test.ndpi out_zlib.zarr
  '''
}

workflow {
    bioformats2raw()
}

and got

N E X T F L O W  ~  version 23.04.1
Launching `main.nf` [big_mcclintock] DSL2 - revision: 94a0abcd3d
executor >  local (1)
[c9/3a2a24] process > bioformats2raw [100%] 1 of 1 ✔

Completed at: 22-May-2023 23:59:39
Duration    : 1m 3s
CPU hours   : (a few seconds)
Succeeded   : 1

Additional info:

bioformats2raw version = 0.4.0 Bio-Formats version = 6.8.0 OpenJDK Runtime Environment (build 17.0.3-internal+0-adhoc..src)

[BioinfoTongLI]

Actually, I have the exact same setting as you. I just dug into the detail and realised that it's not the compression or any bf2raw option, but docker related. when I switch to singularity, it works just fine.

[prete]

Is this still an issue or we can close it?

[BioinfoTongLI]

it still is an issue. And will potentilly stop docker user from running the image conversion step of the pipeline. I would leave it here for now

[davehorsfall]

I can't replicate this using the latest release of the webatlas-pipeline from last week (0.4.0).

I'm using:

• Docker version 24.0.7, build afdd53b
• nextflow version 23.10.0.5889
• openjdk version "11.0.20.1" 2023-08-24

@dannda made significant changes and updates to the docker image and dependencies. Could you please try again with the latest version of check if this is still an issue?

[prete]

Can't be reprouced, I'll close this we can re-open it if someone else runs into it.