Precomputed rigid transformation passed as a dictionary in the "do_rigid" argument

[juanenofbit]

Hello, Chandler, You told me about the possibility of passing pre-computed rigid transformations by email. I am having trouble with the formatting required, I am opening an issue here so I can share it with the community, rather than continuing with the post.

I have built a very simple phantom with a known rigid transformation, to validate that valis can correctly read the transformation.

I attach the target (or reference) image "image_target.png", and the moving image "image_moving.png" to be registered. The images are of different size (which is usual in my histopathology images). The rigid transformation from the target to the moving image is the following 3x3 matrix M ( a rotation of 10 degrees plus a translation, no scaling). The upper right corner of the images are taken as the origin of coordinates. I have also changed the colors in the moving image for clarity.

M= [[ 9.84807753e-01 -1.73648178e-01 -200] [ 1.73648178e-01 9.84807753e-01 -100] [ 0 0 1]]

Target image: (1400x800 pixels)

Moving image: (1100x650 pixels)

Note that there is no scaling, the color object is rotated and translated, but it has the same pixels, it's just inscribed in different size images, so it looks bigger in the moving image when reading the post on github.

It can be checked that the centroids of the four small black squares of the target image, with coordinates [600 200; 600 500 ; 1000 200 ; 1000 500 ] when multiplied by the matrix M, they fit the corresponding positions in moving image, with coordinates
[356.15 201.15; 304.06 496.59; 750.07 270.60; 697.98 566.05]

To register the moving image, the inverse matrix of M , naturally, would be used.

In valis, when I use this matrix M in the dictionary passed as a parameter to the "do_rigid" argument, I am getting a scaled result (with different size of the colored object), not aligned with the original target image:

I am also trying to use the “transformation_src_shape_rc” and “transformation_dst_shape_rc” parameters to control the size of both source and destination images but I am not able to adjust to the desired result. The "do_rigid" dictionary: 'Image_moving.png': {'M': array([[ 9.84807753e-01, -1.73648178e-01, -2.00000000e+02], [ 1.73648178e-01, 9.84807753e-01, -1.00000000e+02], [ 0.00000000e+00, 0.00000000e+00, 1.00000000e+00]]), 'transformation_src_shape_rc': array([ 650., 1100.]), 'transformation_dst_shape_rc': array([ 800., 1400.])}}

And the commands for registration: registrar = registration.Valis(dir_input, dir_output, img_list=files_selected, imgs_ordered=True, align_to_reference=True, reference_img_f=reference_image, do_rigid=rigid_dict, max_image_dim_px=MAX_PROCESSED_IMAGE_DIM, max_processed_image_dim_px=MAX_PROCESSED_IMAGE_DIM, thumbnail_size=THUMBNAIL_SIZE) rigid_registrar, non_rigid_registrar, error_df = registrar.register() registrar.warp_and_save_slides(dir_output, level=0, non_rigid=False, compression="lzw", crop="reference")

[cdgatenbee]

Hi @juanenofbit, Good to hear from you, but sorry this isn't working as expected. Thanks for putting together this nice example dataset. I'll download the images and see if I can figure out what is going wrong.

Best, -Chandler

[juanenofbit]

Hi, I've tried an even simpler example, and I'm seeing a scaling factor depending on the relative size of the images.

This time I have drawn the same object in the reference and moving images, in the same position (measured in pixels) with respect to the origin of coordinates (upper left corner of the image, I suppose). I enlarge the size of the moving image on the right, and change the color for clarity

Reference image: 1000x1000 pixels:

Moving image: 1000x2000 pixels:

This time the transformation passed to the "do_rigid" argument is the identity matrix [[1 0 0][0 1 0][0 0 1]], So the object should remain in the starting position, without changing size.

But the result transformed by valis results in a 1000x1000 image (because I am using crop="reference") but with the object reduced to half the size (click on the images or download them to compare them properly):

I don't quite understand the influence of the other parameters of the "do_rigid" dictionary either (transformation_src_shape_rc and transformation_dst_shape_rc). I don't see that they affect the transformation when I change their values (although here I haven't tested too much).

Regards, Juan

[cdgatenbee]

Hi @juanenofbit Thanks for catching this and putting these examples together. In order to use the provided transformations, valis has to scale them to work with the smaller images used throughout the rest of the pipeline, but it looks like it wasn't being done correctly. You were passing in everything correctly though. I believe i was able to fix it in the most recent version of valis, 1.0.0rc11. Here are what the overlaps of the moving and target images, although it seems that the non-rigid registration did a few odd things:

and the second example

The transformation_src_shape_rc parameter is supposed to be the shape of the image used to find the transformation, which may or may not be the same as the image that is being warped. Similarly, transformation_dst_shape_rc is the shape of the image after being warped, which may be different than the shape of the warped images valis uses. They're both used to scale the user provided transformations. If transformation_src_shape_rc isn't provided, valis assumes it to be the shape of the full resolution moving image. If transformation_dst_shape_rc isn't provided, but reference_img_f is, valis will assume it to be the shape of the full resolution reference image. In this example those assumptions seem to be met so the arguments could probably be left out, but that may not always be the case.

Here is the code I used to perform the registration using the provided transforms. It's pretty much the same as you had provided, but I thought it still might be useful to share.

from valis import registration, valtils, warp_tools

if img_dir == "phantom_images":
    # First example
    moving_M = np.array([[ 9.84807753e-01, -1.73648178e-01, -200], [1.73648178e-01, 9.84807753e-01, -100], [0, 0, 1]])
    transformation_src_shape_rc = (650, 1100) # Shape of moving image
    transformation_dst_shape_rc = (800, 1400) # Shape of target image
elif img_dir == "phantom_images2":
    # Second example
    moving_M = np.eye(3)
    transformation_src_shape_rc = (1000, 2000) # Shape of moving image
    transformation_dst_shape_rc = (1000, 1000) # Shape of target image

moving_image_f = os.path.join(dir_input, "Image_moving.png")
reference_image_f =  os.path.join(dir_input, "Image_target.png")

moving_transform = {"M":moving_M, "transformation_src_shape_rc": transformation_src_shape_rc, "transformation_dst_shape_rc": transformation_dst_shape_rc}
# moving_transform = {"M":moving_M} # Could use this too because assumptions are met
rigid_transform_dict = {moving_image_f: moving_transform}

registrar = registration.Valis(dir_input, dir_output, imgs_ordered=True, align_to_reference=True, reference_img_f=reference_image_f, do_rigid=rigid_transform_dict)
rigid_registrar, non_rigid_registrar, error_df = registrar.register()

# Warp the full image. Small enough to save as png
moving_slide = registrar.get_slide(moving_image_f)
warped_moving = moving_slide.warp_img()
target_slide = registrar.get_slide(reference_image_f)
img_list = [skcolor.rgb2gray(warped_moving), skcolor.rgb2gray(target_slide.image)]
overlap_img = registrar.draw_overlap_img(img_list)
warp_tools.save_img(os.path.join(registrar.dst_dir, "overlap.png"), overlap_img)
warp_tools.save_img(os.path.join(registrar.dst_dir, "moving_warped.png"), warped_moving)
warp_tools.save_img(os.path.join(registrar.dst_dir, "target.png"), target_slide.image)

Thanks again posting the issue. Please let me know if you're also able to get it to work correctly, and/or if you run into more issues.

Best, -Chandler

[juanenofbit]

Thank you for the update, Chandler!

After installing the release 1.0.0rc11, it works pefectly with these examples. This week I will try with the real histopathological images and if there are any problems I will let you know.

Regards, Juan

[juanenofbit]

I tested the precalculated rigid option with a set of histopathological images. In a high percentage of cases, the code has worked perfectly. However, in some cases the following error appeared:

I can provide more details of a case that failed to see what might be going on.

==== Rigid registraration

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:09<00:00,  2.36s/it]

======== Detecting features

100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:23<00:00,  5.87s/it]

======== Matching images

100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:00<00:00, 47304.18it/s]
/home/juan/anaconda3/envs/valis/lib/python3.9/site-packages/valis/valtils.py:75: UserWarning: unable to call extract_area
  extract_area: bad extract area

  warnings.warn(warning_msg, warning_type)
Traceback (most recent call last):
  File "/home/juan/anaconda3/envs/valis/lib/python3.9/site-packages/valis/registration.py", line 3359, in register
    rigid_registrar = self.rigid_register()
  File "/home/juan/anaconda3/envs/valis/lib/python3.9/site-packages/valis/registration.py", line 2494, in rigid_register
    self.rigid_overlap_img = warp_tools.crop_img(self.rigid_overlap_img, overlap_mask_bbox_xywh)
  File "/home/juan/anaconda3/envs/valis/lib/python3.9/site-packages/valis/warp_tools.py", line 1318, in crop_img
    cropped = img.extract_area(*xywh[:2], *wh)
  File "/home/juan/anaconda3/envs/valis/lib/python3.9/site-packages/pyvips/vimage.py", line 1347, in call_function
    return pyvips.Operation.call(name, self, *args, **kwargs)
  File "/home/juan/anaconda3/envs/valis/lib/python3.9/site-packages/pyvips/voperation.py", line 305, in call
    raise Error('unable to call {0}'.format(operation_name))
pyvips.error.Error: unable to call extract_area
  extract_area: bad extract area

[cdgatenbee]

Hi @juanenofbit,

Glad it's working most of the time, but sorry to hear this is happening now. The error seems to be because valis is trying to crop the image but some part of the region's bounding box is outside of the image. I have some ideas as to why this might be happening, but it would definitely be helpful if you could provide a few more details, assuming it isn't too much trouble. In particular, if you could pickle the registrar after the registration fails that would also be extremely helpful. Even though there was an error, the registrar should still contain useful info, like the "do_rigid" dictionary, thumbnails of the images (processed and unprocessed), slide image dimensions, etc... With that file I think I should be able to replicate this particular error and then figure out what's going on. Thanks again for the feature suggestions and for your patience :)

Best, Chandler

[juanenofbit]

Hi Chandler,

I get the following error when I try to pickle the registrar:

file_registrar_pickle =  dir_output + "registrar.pickle"
pickle.dump(registrar, open(file_registrar_pickle, "wb"))    
pickle.close()

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_3877196/2274835269.py in <module>
      2 
      3 file_registrar_pickle =  dir_output + "registrar.pickle"
----> 4 pickle.dump(registrar, open(file_registrar_pickle, "wb"))
      5 pickle.close()

TypeError: cannot pickle 'cv2.BRISK' object

How to perform only the rigid transformation, without calling the non-rigid part? Thanks again for your support!

Best, Juan

[cdgatenbee]

Hi Juan,

Sorry, I hadn't considered that the registrar would still contain un-picklable objects. But you can remove those by calling registrar.cleanup() before pickling. Hopefully that will work, but if not Iet me know.

To skip non-rigid registration, you can set non_rigid_registrar_cls=None when you initialize the Valis object.

Also, I should mention that I'll be traveling for work over the next 2 weeks, so it me take me a little longer than usual to address this issue, but I'll try to take care of it ASAP.

Best, -Chandler

[juanenofbit]

I Chandler,

I attach a link to the pickle file,

https://drive.google.com/file/d/1PXnn6Qp1kpsUzr_7zZUeXLv_xbtKHhdf/view?usp=sharing

Best, Juan

[cdgatenbee]

Thanks Juan! I'll take a look and see if I can figure out what's going on. I'll keep you posted.

Best, -Chandler

[cdgatenbee]

Hi @juanenofbit, So sorry it took so long to address this issue. Has been a very busy few months :( Anyhow, I believe I've fixed the issue, which was related to rounding errors in estimating the resized registered image's shape. Hopefully it will work now, but if not let me know.

Also, out of curiosity I wanted to know how valis would do without the manually provided transforms, and at least in this case it seemed to do pretty well. I just used the default setup, but increased the size of the images used for non-rigid registration to match something similar to what you had:

registrar = registration.Valis(src_dir=src_dir, dst_dir=dst_dir, reference_img_f=ref_img_name, max_non_rigid_registartion_dim_px=3000)
registrar.register()

And here are the results I got: Original

Rigid

Non-rigid

At least in this case, it looks like valis could align these automatically. Just thought I would share since it might save you some trouble of having to manually find the rigid transformations (at least for this image set). Do you think this parameterization would work some of your other difficult images?

Best, Chandler