Expand add_layer_from_images to cubing script

[jstriebel]

After having #632 (PR #741) in place, add_layer_from_images is a good start to replace/update the current cubing.py and related scripts.

The following functionality is lacking in comparison to the current cubing.py script:

• z offset (start_z)
• skip_first_z_slices
• allow smaller batch-sizes if not compressing data
• channel_index
• sample_index
• target_mag (downsampling initial layer)
• manual dtype selection,
• missing formats (monkey-patching extensions, adding czi, dm3 and dm4 readers, see below)

Other functionality, that might get into it:

• use typer, add docs-generation
• general offset

• 340

• xy-grid (see tile_cubing.py), might also fix #44
• allow multiple layers / image stacks, might fix #285, #426
• add xy-batching (fixes #442)
• default to chunk-size 1 in z when converting 2D images to zarr (#322)

More issues which might get fixed:

729, #273 (via pims_nd2), #171 (via pims), #51, #39, #289 (via bioformats), #427, #282, #281, #280 (via bioformats), large step for #188

Missing formats

# current formats we support
cuber = {"tif", "tiff", "jpg", "jpeg", "jpe", "jfif", "jif", "jp2", "j2k", "jpf", "jpm", "jpg2", "j2c", "jpc", "jpx", "mj2", "png", "bmp", "dm3", "dm4", "czi"}
# pims provides supported extensions for the readers
pims_all = {"koala", "g3", "mpeg", "pfm", "ico", "targa", "psd", "ras", "dm3", "seq", "tif", "lsm", "tiff", "pgm", "lei", "pbm", "raw", "ffmpeg", "pgmraw", "cine", "spe", "avbin", "mp4", "jp2", "wbmp", "wmv", "ipw", "fits", "gif", "cut", "exr", "j2k", "pbmraw", "ppm", "npz", "avi", "gdal", "jpg", "iff", "jng", "webp", "ics", "fi_tiff", "xpm", "dicom", "dds", "stk", "itk", "bmp", "mkv", "hdr", "ts", "ipl", "mov", "ids", "jpeg", "lif", "pcx", "pcd", "sgi", "dummy", "swf", "ppmraw", "png", "nd2", "xbm", "pict"}
# this is excluding the bioformats reader extensions:
pims = {"mp4", "hdr", "fi_tiff", "wbmp", "npz", "psd", "ppmraw", "dummy", "swf", "g3", "avi", "wmv", "jp2", "koala", "pbmraw", "bmp", "gif", "mkv", "spe", "webp", "xpm", "ras", "jpeg", "pgm", "pbm", "iff", "jng", "exr", "sgi", "dds", "gdal", "pcx", "ppm", "j2k", "raw", "dicom", "cine", "pict", "png", "seq", "fits", "itk", "ts", "mpeg", "pcd", "targa", "ffmpeg", "mov", "pfm", "avbin", "tif", "pgmraw", "xbm", "cut", "lsm", "ico", "tiff", "jpg", "stk"}
# bioformats actually supports much more extensions than pims annotates
bioformats = {"sld", "sldy", "tif", "aim", "al3d", "gel", "am", "amiramesh", "grey", "hx", "labels", "cif", "img", "hdr", "sif", "png", "afi", "svs", "svs", "htd", "pnl", "avi", "arf", "exp", "tif", "spc", "sdt", "xml", "h5", "1sc", "pic", "raw", "xml", "scn", "ims", "img", "cr2", "crw", "ch5", "c01", "dib", "vsi", "xml", "tif", "wpi", "tif", "dv", "r3d", "rcpnl", "dcm", "dicom", "v", "eps", "epsi", "ps", "flex", "mea", "res", "img", "tiff", "fits", "dm3", "dm4", "dm2", "vff", "gif", "naf", "his", "ndpi", "ndpis", "vms", "txt", "tif", "bmp", "jpg", "i2i", "ics", "ids", "fff", "seq", "ipw", "hed", "img", "mod", "liff", "raw", "tif", "obf", "msr", "xdce", "tif", "frm", "inr", "hdr", "tif", "tiff", "ipl", "ipm", "dat", "img", "par", "jpg", "jp2", "j2k", "jpf", "jpk", "jpx", "klb", "xv", "bip", "fli", "msr", "lei", "tif", "lif", "lof", "scn", "xlef", "sxm", "tif", "tiff", "l2d", "tif", "scn", "lim", "tiff", "stk", "nd", "htd", "tif", "tiff", "tif", "tif", "txt", "xml", "tif", "tiff", "mnc", "mrw", "mng", "stp", "mrc", "st", "ali", "map", "rec", "mrcs", "nef", "tif", "img", "hdr", "nii", "nii.gz", "tiff", "tiff", "nd2", "nrrd", "nhdr", "raw", "txt", "apl", "mtb", "tnb", "tif", "obsep", "oib", "oif", "tif", "oir", "omp2info", "xml", "dat", "tif", "tiff", "ome.tiff", "ome.tif", "ome.tf2", "ome.tf8", "ome.btf", "ome", "ome.xml", "top", "pcoraw", "rec", "pcx", "pds", "xml", "csv", "tif", "im3", "tiff", "xml", "tif", "pbm", "pgm", "ppm", "psd", "tif", "tiff", "bin", "pict", "png", "tif", "xml", "cfg", "spe", "afm", "mov", "sm2", "sm3", "xqd", "xqf", "cxd", "tiff", "spi", "stk", "tga", "db", "tif", "txt", "tiff", "tif", "tf2", "tf8", "btf", "vws", "tfr", "ffr", "zfr", "zfp", "2fl", "tif", "sld", "jpg", "pr3", "dat", "hdr", "fdf", "tif", "qptiff", "hdf", "bif", "dti", "xys", "html", "mvd2", "acff", "wat", "bmp", "wlz", "lms", "xml", "tif", "zvi", "czi", "lsm", "mdb"}

Most of the missing formats are just missing extension that need to get monkey-patched into pims ({'jfif', 'mj2', 'jpe', 'jpg2', 'jif', 'jpm', 'jpc', 'jpf', 'jpx', 'j2c'}. From the non-bioformats python implementations czi, dm3 and dm4 are missing, but all of those are included in bioformats. Before relying too much on it, we should compare the memory usage. Depending on this, we should still add our czi, dm3/4 adapters to the pims readers. To use bioformats, we should also adapt the docker image to include a JVM.

---

Affected libraries: webknossos & wkcuber

[jstriebel]

Some current features seem like they could be ignored atm, according to @normanrz:

• a CLI in general, we can just start out with a function in the wk-libs
• cubing tiled stacks does not seem to be important anymore

Both of those can be incorporated later.

For a first version that should be able to supersede wkcuber the following features are needed:

• z offset (start_z)
• skip_first_z_slices
• channel_index
• sample_index
• missing formats (monkey-patching extensions, adding czi, dm3 and dm4 readers)

[normanrz]

• cubing stacks does not seem to be important anymore

what stacks do you mean?

[jstriebel]

• cubing stacks does not seem to be important anymore

what stacks do you mean?

Having multiple stacks across x-y. We discussed that briefly during the last sprint, and you were mentioning that we don't have such datasets anymore usually.

[normanrz]

• cubing stacks does not seem to be important anymore

what stacks do you mean?

Having multiple stacks across x-y. We discussed that briefly during the last sprint, and you were mentioning that we don't have such datasets anymore usually.

Ah ok, you mean tiled stacks. I agree with that.

[jstriebel]

Superseded by #820 and #825