list index out of range error when trying to save registered slides

[ciaobladoo]

Hey Chandler,

I just made it work on Databricks! However, I am when I try to save registered slides with registrar.warp_and_save_slides(registered_slide_dst_dir, crop="overlap") I get this error:

Saving images: 0%| | 0/2 [00:14<?, ?image/s] Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/valis/valtils.py", line 30, in wrapper return f(*args, **kwargs) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/valis/registration.py", line 4518, in warp_and_save_slides slide_obj.warp_and_save_slide(dst_f=dst_f, level=level, File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/valis/valtils.py", line 30, in wrapper return f(*args, **kwargs) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/valis/registration.py", line 1004, in warp_and_save_slide ome_xml_obj = slide_io.update_xml_for_new_img(current_ome_xml_str=slide_meta.original_xml, File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/valis/valtils.py", line 30, in wrapper return f(*args, **kwargs) File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.10/site-packages/valis/slide_io.py", line 2674, in update_xml_for_new_img img = og_ome.images[series] IndexError: list index out of range

Could you advice what could be going on here? Thanks!

[cdgatenbee]

Hi @ciaobladoo,

Congrats on getting valis to work on Databricks! I'll have to check it out. Regarding this error, it looks like there is a mismatch between the series that was read, and the series that is being used to access the image metadata. I've made some updates that should prevent this from happening, so hopefully it should be fixed in the next update (1.0.3), which I think should be ready sometime next week. However, if you're able to share the images that are causing the issue, I could test before pushing the update to make sure that this won't be an issue in the future. If you'd prefer to share them privately, you can email me at Chandler.Gatenbee@moffitt.org.

Best, -Chandler

[ciaobladoo]

Hi Chandler,

Do you mean this error is caused by the particular images I am registering? My colleague have previously made it work on different set of images on a local machine. I will double check with my manager if I could send you the images. If so I will send to your email.

Thank you so much!

Best, Chao

[ciaobladoo]

Hey @cdgatenbee, just wanted to give you a quick update - it seems like the error may not be because of the image metadata. I tried to register on Databricks images my colleague Daniel (who you may know) successfully registered on a local machine and am getting this same error. We will find time to discuss and see what is going on and get back to you.

[unnir]

The same error.

Previously it worked perfectly, I guess something changes in the dependencies...

JVM has been initialized. Be sure to call registration.kill_jvm() or slide_io.kill_jvm() at the end of your script.

==== Converting images

Converting images: 100%|██████████| 3/3 [00:05<00:00,  1.96s/image]

==== Processing images

Processing images : 100%|██████████| 3/3 [00:06<00:00,  2.15s/image]
Normalizing images: 100%|██████████| 3/3 [00:00<00:00,  8.90image/s]
Denoising images  : 100%|██████████| 3/3 [00:00<00:00,  3.05image/s]

==== Rigid registration

Detecting features   : 100%|██████████| 3/3 [00:18<00:00,  6.25s/image]
Matching images      : 100%|██████████| 3/3 [00:01<00:00,  1.52image/s]
Finding transforms   : 100%|██████████| 2/2 [00:00<00:00, 1370.69image/s]
Finalizing           : 100%|██████████| 3/3 [00:00<00:00, 2830.80image/s]

======== Rigid registration complete in 24.707 seconds

==== Non-rigid registration

Preparing images for non-rigid registration: 100%|██████████| 3/3 [00:03<00:00,  1.08s/image]
Finding non-rigid transforms: 100%|██████████| 2/2 [00:08<00:00,  4.14s/image]

======== Non-rigid registration complete in 10.528 seconds

/usr/local/lib/python3.10/dist-packages/interpolation/splines/eval_splines.py:318: NumbaPendingDeprecationWarning: Code using Numba extension API maybe depending on 'old_style' error-capturing, which is deprecated and will be replaced by 'new_style' in a future release. See details at https://numba.readthedocs.io/en/latest/reference/deprecation.html#deprecation-of-old-style-numba-captured-errors
Exception origin:
  File "/usr/local/lib/python3.10/dist-packages/interpolation/splines/eval_splines.py", line 116, in __eval_spline
    kk = (order).literal_value

  return _eval_cubic(*args)

==== Measuring error

Measuring error: 100%|██████████| 3/3 [00:04<00:00,  1.51s/image]
Saving images:   0%|          | 0/3 [00:05<?, ?image/s]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
[<ipython-input-5-0175eabb0ab3>](https://localhost:8080/#) in <cell line: 6>()
      4 
      5 # Save all registered slides as ome.tiff
----> 6 registrar.warp_and_save_slides(registered_slide_dst_dir, crop="overlap")

5 frames
[/usr/local/lib/python3.10/dist-packages/valis/valtils.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
     28         def wrapper(*args, **kwargs):
     29             rename_kwargs(f.__name__, kwargs, aliases)
---> 30             return f(*args, **kwargs)
     31 
     32         return wrapper

[/usr/local/lib/python3.10/dist-packages/valis/registration.py](https://localhost:8080/#) in warp_and_save_slides(self, dst_dir, level, non_rigid, crop, colormap, interp_method, tile_wh, compression, Q)
   4516             dst_f = os.path.join(dst_dir, slide_obj.name + ".ome.tiff")
   4517 
-> 4518             slide_obj.warp_and_save_slide(dst_f=dst_f, level=level,
   4519                                           non_rigid=non_rigid,
   4520                                           crop=crop,

[/usr/local/lib/python3.10/dist-packages/valis/valtils.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
     28         def wrapper(*args, **kwargs):
     29             rename_kwargs(f.__name__, kwargs, aliases)
---> 30             return f(*args, **kwargs)
     31 
     32         return wrapper

[/usr/local/lib/python3.10/dist-packages/valis/registration.py](https://localhost:8080/#) in warp_and_save_slide(self, dst_f, level, non_rigid, crop, src_f, channel_names, colormap, interp_method, tile_wh, compression, Q)
   1002         bf_dtype = slide_io.vips2bf_dtype(warped_slide.format)
   1003         out_xyczt = slide_io.get_shape_xyzct((warped_slide.width, warped_slide.height), warped_slide.bands)
-> 1004         ome_xml_obj = slide_io.update_xml_for_new_img(current_ome_xml_str=slide_meta.original_xml,
   1005                                                       new_xyzct=out_xyczt,
   1006                                                       bf_dtype=bf_dtype,

[/usr/local/lib/python3.10/dist-packages/valis/valtils.py](https://localhost:8080/#) in wrapper(*args, **kwargs)
     28         def wrapper(*args, **kwargs):
     29             rename_kwargs(f.__name__, kwargs, aliases)
---> 30             return f(*args, **kwargs)
     31 
     32         return wrapper

[/usr/local/lib/python3.10/dist-packages/valis/slide_io.py](https://localhost:8080/#) in update_xml_for_new_img(current_ome_xml_str, new_xyzct, bf_dtype, is_rgb, series, pixel_physical_size_xyu, channel_names, colormap)
   2672             if colormap is None:
   2673                 # Get original channel colors
-> 2674                 img = og_ome.images[series]
   2675                 colormap = {c.name: c.color.as_rgb_tuple() for c in img.pixels.channels}
   2676                 all_rgb = set(list(colormap.values()))

IndexError: list index out of range

[cdgatenbee]

Thanks for the updates @ciaobladoo and @unnir! I think I have the fix for this in the next update (1.0.3), but before pushing I'll test using updated dependencies to make sure everything works as expected.

Best, -Chandler

[unnir]

great! Thank you!

[ciaobladoo]

Hi, @cdgatenbee. I talked to Daniel and the only reason I can think of that for the same pair of images he can register on a local machine and I cannot on Databricks is that Docker does something that prevents this error (I cannot use Docker on Databricks). Hope your fix can help. I sent you a pair of images last week via email. Feel free to use them for testing. Let me know if you need anything else. Thank you so much for your diligent help!

[cdgatenbee]

Hi @ciaobladoo, Could you try updating valis to 1.0.3 (pushed today) to see if that fixes the issue?

Best, -Chandler

[ciaobladoo]

Will do, Chandler! Thank you for the hard work!

[unnir]

thank you for update it solved the issue!

[ciaobladoo]

Hey Chandler, I can confirm the issue is resolved for us as well. Thank you so much!

[cdgatenbee]

Great! Glad this fixed the issue :)