Issue saving the ome.tiff while merging 2channel tiffs.

[andreevg04]

Dear Chandler, I am using the following script to try merge 3 2-channel images, whereas channel 1 should always be the same. The images are tiffs where they were RGBs that I converted to 2channel images using ImageJ. import os

from valis import registration, valtils

slide_src_dir = "/media/burgera/Samsung_T5/merged cores" results_dst_dir = "./expected_results/registration" merged_slide_dst_f = "./slide_merging_example/merged_slides.ome.tiff" # Where to save merged slide

Create a Valis object and use it to register the slides in slide_src_dir

registrar = registration.Valis(slide_src_dir, results_dst_dir) rigid_registrar, non_rigid_registrar, error_df = registrar.register()

Merge registered channels

def cnames_from_filename(src_f): """Get channel names from file name Note that the DAPI channel is not part of the filename but is always the first channel. """ f = valtils.get_name(src_f) return ["DAPI"] + f.split(" ")

channel_name_dict = {f: cnames_from_filename(f) for f in registrar.original_img_list}

Use the correctly defined path for merging slides

start = time.time() merged_img, channel_names, ome_xml = registrar.warp_and_merge_slides(merged_slide_dst_f, channel_name_dict=channel_name_dict, drop_duplicates=True)

registration.kill_jvm()

The error is this one === Measuring error

Measuring error: 100%|███████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 3.14image/s] merging DAPI, PDL1 merging FoxP3 merging CD57 Traceback (most recent call last): File "/home/burgera/slides/test_merge.py", line 28, in merged_img, channel_names, ome_xml = registrar.warp_and_merge_slides(merged_slide_dst_f, File "/home/burgera/.local/lib/python3.10/site-packages/valis/valtils.py", line 30, in wrapper return f(*args, *kwargs) File "/home/burgera/.local/lib/python3.10/site-packages/valis/registration.py", line 4664, in warp_and_merge_slides ome_xml_obj = slide_io.create_ome_xml(out_xyczt, bf_dtype, is_rgb=False, File "/home/burgera/.local/lib/python3.10/site-packages/valis/valtils.py", line 30, in wrapper return f(args, **kwargs) File "/home/burgera/.local/lib/python3.10/site-packages/valis/slide_io.py", line 2593, in create_ome_xml new_img.pixels.physical_size_x_unit = phys_u File "pydantic/main.py", line 384, in pydantic.main.BaseModel.setattr pydantic.error_wrappers.ValidationError: 1 validation error for Pixels physical_size_x_unit value is not a valid enumeration member; permitted: 'Å', 'ua', 'am', 'cm', 'dam', 'dm', 'Em', 'fm', 'ft', 'Gm', 'hm', 'in', 'km', 'ly', 'li', 'Mm', 'm', 'µm', 'mi', 'mm', 'nm', 'pc', 'Pm', 'pm', 'pixel', 'pt', 'reference frame', 'Tm', 'thou', 'yd', 'ym', 'Ym', 'zm', 'Zm' (type=type_error.enum; enum_values=[<UnitsLength.ANGSTROM: 'Å'>, <UnitsLength.ASTRONOMICALUNIT: 'ua'>, <UnitsLength.ATTOMETER: 'am'>, <UnitsLength.CENTIMETER: 'cm'>, <UnitsLength.DECAMETER: 'dam'>, <UnitsLength.DECIMETER: 'dm'>, <UnitsLength.EXAMETER: 'Em'>, <UnitsLength.FEMTOMETER: 'fm'>, <UnitsLength.FOOT: 'ft'>, <UnitsLength.GIGAMETER: 'Gm'>, <UnitsLength.HECTOMETER: 'hm'>, <UnitsLength.INCH: 'in'>, <UnitsLength.KILOMETER: 'km'>, <UnitsLength.LIGHTYEAR: 'ly'>, <UnitsLength.LINE: 'li'>, <UnitsLength.MEGAMETER: 'Mm'>, <UnitsLength.METER: 'm'>, <UnitsLength.MICROMETER: 'µm'>, <UnitsLength.MILE: 'mi'>, <UnitsLength.MILLIMETER: 'mm'>, <UnitsLength.NANOMETER: 'nm'>, <UnitsLength.PARSEC: 'pc'>, <UnitsLength.PETAMETER: 'Pm'>, <UnitsLength.PICOMETER: 'pm'>, <UnitsLength.PIXEL: 'pixel'>, <UnitsLength.POINT: 'pt'>, <UnitsLength.REFERENCEFRAME: 'reference frame'>, <UnitsLength.TERAMETER: 'Tm'>, <UnitsLength.THOU: 'thou'>, <UnitsLength.YARD: 'yd'>, <UnitsLength.YOCTOMETER: 'ym'>, <UnitsLength.YOTTAMETER: 'Ym'>, <UnitsLength.ZEPTOMETER: 'zm'>, <UnitsLength.ZETTAMETER: 'Zm'>])

I know it is long.....Also you said in the paper that you merged multiple IHCs and they are RGBs. How did you manage to register and merge them in one multiplexed image?

[andreevg04]

Just a small correction, I managed to deal with the problem of the physical size by switching from pixels to micrometers in ImageJ and scaling it accordingly. Nevertheless the result is not a multiplexed IF but a strange mix of color channels. So any tips will be hearthly welcomed. I can also provide the images which I would like to register and merge together. It is basically cyclic IHC methon with RGB images, which I deconvoluted to two channel image and then merged the channles and inverted the image in ImageJ. Those images I want to merge then to get the multiplexed IF image. If you would have any other sugestion I would love to get your help!

[xb-li]

Hi @andreevg04 Is it reasonable that the ome.tiff files generated by running the program are all above 1GB. Later, when I used CLAM for segmentation, it displayed: level_dim 46208 x 23808 is likely too large for successful segmentation, aborting I hope to receive your help.

[andreevg04]

Hi @xb-li I wasnt able to see how big the ome.tiff was but it was around 300-400mb. I ran htop and my CPU was at 100% at all 15 cores (I have 8 and another virtual 8). RAM was bit overloaded around 13.8GB from 14.5GB possible. I managed to get 4 images fused but not more and my theroy at the moment is that our PC is far too weak to handle this load. What types of PCs do you guys use and is there any script possibilities to overcome this issue???

Thank you in advance!!!!

[xb-li]

Hi @andreevg04 I used two V100 images, but the results I got also seem to be incorrect because the display was too large during the later segmentation.

[andreevg04]

Hi @xb-li What type of PC do you use? On my behalf it crashes during Matching probably because of the listed CPU usage being 100%

[cdgatenbee]

Just a small correction, I managed to deal with the problem of the physical size by switching from pixels to micrometers in ImageJ and scaling it accordingly. Nevertheless the result is not a multiplexed IF but a strange mix of color channels. So any tips will be hearthly welcomed. I can also provide the images which I would like to register and merge together. It is basically cyclic IHC methon with RGB images, which I deconvoluted to two channel image and then merged the channles and inverted the image in ImageJ. Those images I want to merge then to get the multiplexed IF image. If you would have any other sugestion I would love to get your help!

Hi @andreevg04, Apologies for taking so long to get back to you, I kind of got caught up trying to finish this next update. Regarding your question though, the way I generated the multiplex image from cyclic IHC was sort of from the opposite direction, where I used valis to align the RGB images, separated the stains in the registered images, and then merged those stain channels into a single image. One benefit of this approach is that if you're doing your stain separation in Python, you won't need to save any intermediate files (i.e. the separated stain images or aligned RGB images), since you can work directly with the warped pyvips Images (i.e. warped_slide = slide_obj.warp_slide(level=0), or slide_obj.warp_slide(level=0, crop=tile_bbox_xywh) if you need to process in tiles). Still, the approach you're taking of aligning the 2 channel images should work OK too, especially if the first channel is always the same. If you're still able share the images, I'd be more than happy to take a look to see why merged image isn't turning out as expected. If you'd prefer to keep the images private, you can email me a link (just send to Chandler[dot]Gatenbee[at]moffitt[dot]org). If I can get that working, I can also let you and @xb-li know how big the images turn out to be and what memory usage is like (running a 2021 M1 macbook pro).

Best, -Chandler

[andreevg04]

Dear @cdgatenbee thank you for the response. I will give you a link with the images I wanted to align and merge. I dont know if its because of my computers hardware that it stop processing the image. May I ask you for a script on how to you did the RGB registration, separation and merging, so that I can test it myself on RGB images? The images have always the same first image from the channel split. Will let you see them through.

If there is a way that wont be too process intensive would love to try the script for the RGB images if possible?

[cdgatenbee]

Hi @andreevg04,

Unfortunately, I don't have a relevant script on hand, but maybe I could put something together. However, it might be good to create such an example, since I think others may be interested in taking this approach, too. Working with the pyvips.Image objects might not be too memory intensive, because you can avoid loading the entire image into memory. I'd still like to see if I can figure out why your images aren't being saved properly, but I'll also try to put together an example script showing how to separate the stains in the aligned images, and then merge those together into a single image.

Best, -Chandler

[andreevg04]

I will send you the link tomorrow with some images.

Best wishes and much thanks Chandler, Grigor

[andreevg04]

@cdgatenbee @xb-li I sent Chandler a link with some grayscale images. Would love to get your feedback on why it crashed during the matching step. My guess is that there were too much markers. So at 4 images (4 markers it worked without crashing) above that, it shuts down completely.

I would be open to a collab Chandler for the script so write me an email back and we can talk :)

Best wishes, Grigor

[cdgatenbee]

Hi @andreevg04, Thanks again for sharing the images, I don't think I would have figured out this issue without them. It looks like the reason valis was crashing on the image matching step is because the images look so similar that the number of matches was huge. To avoid this, I created a feature detector that is the same as the default one, but has a lower maximum number of features. When using this feature detector, the average memory usage for all 3 samples is 744MB, the average time taken is 1.1minutes, and the average estimated error is 1.21um. In the script below, I've also added a bit to merge and save the cores into a single ome.tiff. Based on the images and filenames, I assumed the first channel was always DAPI/nuclear, and the second channel was needed to be merged. Below is a screenshoot of the aligned image in QuPath (sorry the thumbnail is distorted, I need to figure out what's going on), and the code used to align, merge, and save the results. Please try it out and let me know if it resolves the issue for you too.

Best,

• Chandler

Code

import pathlib
import os
from time import time
import tracemalloc
import cv2
import numpy as np
from valis import registration, feature_detectors

MAX_FEATURES = 1000
# Create feature detector that has a lower maximum feature count
class OrbVggFD_low_fd(feature_detectors.FeatureDD):
    """Uses ORB for feature detection and VGG for feature description"""
    def __init__(self,  kp_detector=cv2.ORB_create(nfeatures=MAX_FEATURES, fastThreshold=0), kp_descriptor=cv2.xfeatures2d.VGG_create(scale_factor=0.75)):
        super().__init__(kp_detector=kp_detector, kp_descriptor=kp_descriptor)

all_src_dir = "path/to/folder/with/cores" 
dir_list = [d for d in pathlib.Path(all_src_dir).iterdir() if os.path.isdir(d)] # each d is a folder containing the images to be aligned (e.g. 1Adeno_G-11)

n = len(dir_list)
mem_used = [None] * n
time_taken = [None] * n
acc = [None] * n
for i, d in enumerate(dir_list):

    print(os.path.split(d)[1])
    tracemalloc.start()
    start = time()
    # Set `feature_detector_cls` to feature detector that uses fewer features
    registrar = registration.Valis(str(d), dst_dir, feature_detector_cls=OrbVggFD_low_fd)
    rigid_registrar, non_rigid_registrar, error_df = registrar.register()
    stop = time()
    mem_trace = tracemalloc.get_traced_memory()
    tracemalloc.stop()

    # Merge channels and save as ome.tiff
    # Valis did not detect channel names in the slide metadata, so create channel name dictionary
    channel_name_dict = {}
    for slide_obj in registrar.slide_dict.values():
        channel_names = ["DAPI", slide_obj.name.split("_")[0]]
        channel_name_dict[slide_obj.src_f] = channel_names

    merged_f = os.path.join(registrar.dst_dir, registrar.name + "_merged.ome.tiff")
    registrar.warp_and_merge_slides(dst_f=merged_f, channel_name_dict=channel_name_dict, tile_wh=128)

    # registrar.draw_matches(registrar.dst_dir) # Uncomment to see which matched features were used for rigid registration
    mem_used[i] = mem_trace[1]
    time_taken[i] = stop - start
    acc[i] = error_df["mean_non_rigid_D"][0]

mean_time_min = np.mean(time_taken)/60
mean_peak_mem_mb = np.mean(mem_used)/10**6
mean_est_acc_um = np.mean(acc)

[andreevg04]

@cdgatenbee Dear Chandler, a big thanks!!! I will try it out later today or first thing tomorrow. Will this work on other tissues or larger cores? Also the green channel does look like CK and not DAPI. About the thumbnail, this doenst mean that the merging failed correc?

Best wishes Grigor

[cdgatenbee]

@andreevg04, I believe this should work on larger images too, but if you run into a problem let me know. I checked the individual channels, and I think the green looking like CK and not DAPI was just due to the way the colors were mixed. Here are the individual channels:

DAPI

CK

Regarding the distorted thumbnail, I think it may be an issue with the lower resolution pyramid levels either not being saved correctly, or with QuPath not displaying them correctly. It's only started happening recently, so I need to explore it further, but the the full resolution image (e.g. pyramid level 0) seems to display and read OK.

Best, -Chandler