Spots extractation workflow: KeyError pixelResolution

[erkinacar5]

Hello! I have successfully ran the demo_tiny.sh and I got the output files (and folders) as intended. Now, I want to use only the spot extraction (airlocalize) for my own files. I have run the following command:

#!/bin/bash
DIR="/scratch/erkin/projects/vatsi_multifish"
${DIR}/bin/multifish/main-spots-extraction.nf \
    --runtime_opts "-B ${DIR}/inputs"  \
    --stitchdir "${DIR}/inputs/N5Try/Ki67" \
    --outdir "${DIR}/inputs/Vatsi_Files/out_try1"\
    --airlocalize_xy_stride 2048\
    --airlocalize_xy_overlap 5\
    --airlocalize_z_stride 32\
    --airlocalize_z_overlap 5\
    --airlocalize_cpus 10\
    --airlocalize_memory 20G

The file was originally .tif file and we exported it as an n5 file using Fiji. However, this gives the following error:

N E X T F L O W  ~  version 21.10.6
Launching `/scratch/erkin/projects/vatsi_multifish/bin/multifish/main-spots-extraction.nf` [thirsty_hypatia] - revision: 76379d3fd6
executor >  local (1)
executor >  local (1)
[b9/164e55] process > airlocalize:cut_tiles (1)   [100%] 1 of 1, failed: 1 ✘
[-        ] process > airlocalize:run_airlocalize -
[-        ] process > airlocalize:merge_points    -
Error executing process > 'airlocalize:cut_tiles (1)'

Caused by:
  Process `airlocalize:cut_tiles (1)` terminated with an error exit status (1)

Command executed:

  umask 0002
  mkdir -p /scratch/erkin/projects/vatsi_multifish/inputs/Vatsi_Files/out_try1/tiles
  echo "python /app/airlocalize/scripts/cut_tiles.py /scratch/erkin/projects/vatsi_multifish/inputs/N5Try/Ki67 /c2/s0 /scratch/erkin/projects/vatsi_multifish/inputs/Vatsi_Files/out_try1/tiles 2048 5 32 5
  python /app/airlocalize/scripts/cut_tiles.py /scratch/erkin/projects/vatsi_multifish/inputs/N5Try/Ki67 /c2/s0 /scratch/erkin/projects/vatsi_multifish/inputs/Vatsi_Files/out_try1/tiles 2048 5 32 5
  CUT_TILES_RES=`ls -d /scratch/erkin/projects/vatsi_multifish/inputs/Vatsi_Files/out_try1/tiles/*[0-9]`

Command exit status:
  1

Command output:
  python /app/airlocalize/scripts/cut_tiles.py /scratch/erkin/projects/vatsi_multifish/inputs/N5Try/Ki67 /c2/s0 /scratch/erkin/projects/vatsi_multifish/inputs/Vatsi_Files/out_try1/tiles 2048 5 32 5
  Reading voxel grid as 'dimensions' from /scratch/erkin/projects/vatsi_multifish/inputs/N5Try/Ki67//c2/s0
  voxel grid [2048 2048   39]
  voxel stride [2048 2048   32]
  voxel overlap [5 5 5]
  tile grid [1, 1, 1]
  Reading vox as 'pixelResolution.dimensions' from /scratch/erkin/projects/vatsi_multifish/inputs/N5Try/Ki67//c2/s0

Command error:
  INFO:    Could not find any nv files on this host!
  WARNING: While bind mounting '/local1/scratch/erkin/projects/vatsi_multifish/bin/work/b9/164e55b3ef2b9724822a2e730e4a39:/local1/scratch/erkin/projects/vatsi_multifish/bin/work/b9/164e55b3ef2b9724822a2e730e4a39': destination is already in the mount point list
  Traceback (most recent call last):
    File "/app/airlocalize/scripts/cut_tiles.py", line 37, in <module>
      vox           = n5mu.read_voxel_spacing(ref_img_path, ref_img_subpath)
    File "/app/airlocalize/scripts/n5_metadata_utils.py", line 22, in read_voxel_spacing
      vox = np.absolute(atts['pixelResolution']['dimensions'])
  KeyError: 'pixelResolution'

Work dir:
  /local1/scratch/erkin/projects/vatsi_multifish/bin/work/b9/164e55b3ef2b9724822a2e730e4a39

Tip: view the complete command output by changing to the process work dir and entering the command `cat .command.out`

Seeing this, I tried to run the workflow with the output from the tiny demo (since it ran everything, and stichting was done with an export of n5). However, I ran the command which was adapted from the examples folder, to get a file not found error. At this point I am a bit puzzled to figure out what I am doing wrong. Any help is appreciated. I am attaching the output from the tiny demo try, as well as the command I ran to prevent the text from getting too long :) error_demo_file.txt run_spot_extraction_default.sh.txt

[cgoina]

I think the process was adapted to use the output generated by the stitching pipeline which generates an n5 output (exports.n5). Take a look at the metadata and the scale levels generated by the stitching in exports.n5 and try to generate a similar n5 container - in terms of channels and scale levels.

[erkinacar5]

Thank you for the reply! I thought that might be the case and that's why I tried to run the workflow with demo_tiny as well. If I run it, I get a different error (it is looking for a c2 folder that wasn't created by stiching pipeline). You can see the command I ran for the export.n5 folder (from the stiching pipeline) that was made from demo_tiny in my initial post, as well as the error. I also want to mention that I can run the whole pipeline with demo_tiny.sh script, it is just that if I want to run the spots extraction from the stiching output of this successfully ran script, spots extraction still doesn't work (even though it worked within the demo_tiny.sh).

I hope my point is clear enough, if not please let me know and I will do my best to clarify.

[cgoina]

By default spot_extraction workflow uses the dapi_channel parameter to tile the volume. By default the dapi_channel is 'c2'. That is why your tile command uses 'c2/s0'. For demo_tiny dataset that is adjusted and it is set to c1 because the tiny dataset only has 2 channels. You can override the dapi_channel parameter using '--dapi_channel '. Also you need to pass in the correct channels using the --channels argument which default to 'c0,c1,c2,c3' All these are taken care of by the demo_tiny.sh, but if you want to invoke the standalone spot extraction you need to take care of that. Keep in mind that the standalone commands have not been kept up-to-date. The full pipeline offers 2 options for spot extraction - airlocalize and rs-fish (a spark based implementation) but main_spots_extraction only uses airlocalize.

[erkinacar5]

Thank you for the explanation. I see that I cannot pass the channels argument with the standalone spot extraction, but if they are not kept up-to-date then it makes sense. I just wanted to make sure if I am doing anything wrong or not, so I think we can say the issue is resolved. Once again thank you for your kind and patient explanation! I will stick to the main workflow if I can!

[cgoina]

You should be able to pass --channels parameters to main-spots-extraction.nf. Have you tried? If that fails please let me know because it is a bug. Also if it is really important for you and/or you prefer using the standalone tools please let us know - Feel free to open a ticket and explain us your use case and we will try to address it. The reason we have not kept those tools up to date was because our internal users could just use the main pipeline and use the skip parameters.