"ERROR: Could not consume arg: --im1"

[asmagen]

I got the following error related to the command setup:

[magena01@li03c03 liver]$ pathflow-mixmatch --im1 Li63TCD20_cropped_mask.png --im2 Li63TCD3_cropped_mask.png --fix_rotation False --output_dir ./ --gpu_device -1 --transform_type similarity --lr 0.01 --iterations 1000 --min_object_size 50000 --no_segment_analysis True --black_background True
ERROR: Could not consume arg: --im1
Usage: pathflow-mixmatch <command>
  available commands:    register_images

For detailed information on this command, run:
  pathflow-mixmatch --help

Trying to add 'register_images' didn't work:

[magena01@li03c03 liver]$ pathflow-mixmatch register_images --im1 Li63TCD20_cropped_mask.png --im2 Li63TCD3_cropped_mask.png --fix_rotation False --output_dir ./ --gpu_device -1 --transform_type similarity --lr 0.01 --iterations 1000 --min_object_size 50000 --no_segment_analysis True --black_background True
Loading images.
Traceback (most recent call last):
  File "/hpc/users/magena01/.local/bin/pathflow-mixmatch", line 10, in <module>
    sys.exit(main())
  File "/hpc/users/magena01/.local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 386, in main
    fire.Fire(Commands)
  File "/hpc/packages/minerva-centos7/py_packages/3.7/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/hpc/packages/minerva-centos7/py_packages/3.7/lib/python3.7/site-packages/fire/core.py", line 471, in _Fire
    target=component.__name__)
  File "/hpc/packages/minerva-centos7/py_packages/3.7/lib/python3.7/site-packages/fire/core.py", line 675, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/hpc/users/magena01/.local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 383, in register_images
    max_rotation_vertical_px=max_rotation_vertical_px)
  File "/hpc/users/magena01/.local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 248, in register_images_
    im1=np.load(npy1)
  File "/hpc/packages/minerva-centos7/py_packages/3.7/lib/python3.7/site-packages/numpy/lib/npyio.py", line 428, in load
    fid = open(os_fspath(file), "rb")
FileNotFoundError: [Errno 2] No such file or directory: 'A.npy'

[jlevy44]

It seems like you do not have the latest version of pathflow_mixmatch.

Can you try:

pip install git+https://github.com/jlevy44/PathFlow-MixMatch.git

[jlevy44]

I think you installed our official distribution, but not the dev version.

[asmagen]

I had to uninstall and reinstall. It worked after adding 'register_images' to the command, so overall: pathflow-mixmatch register_images --im1 Li63TCD20_cropped_mask.png --im2 Li63TCD3_cropped_mask.png --fix_rotation False --output_dir ./ --gpu_device -1 --transform_type similarity --lr 0.01 --iterations 1000 --min_object_size 50000 --no_segment_analysis True --black_background True I tried running it in a low and high RAM environment (~4-8GB vs 64GB). Somehow I got a the message 'libgomp: Thread creation failed: Resource temporarily unavailable' in the high environment -- not sure what it does and what are the consequences of this issue. I'll see if it runs and keep you posted. It didn't produce any other message than loading images for 10 minutes already. Is that okay?

[asmagen]

So the results of the description above are that I could run it on a local machine but it failed with the following message after running for about 10-15 minutes:

pathflow-mixmatch register_images --im1 CD20_test.png --im2 CD3_test.png --fix_rotation False --output_dir ./ --gpu_device -1 --transform_type similarity --lr 0.01 --iterations 1000 --min_object_size 50000 --no_segment_analysis True --black_background True
Loading images.
^[[A^[[BTraceback (most recent call last):
  File "/usr/local/bin/pathflow-mixmatch", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 466, in main
    fire.Fire(Commands)
  File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 463, in register_images
    black_background=black_background)
  File "/usr/local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 407, in register_images_
    print("[{}/{}] - Writing registered sections to file.".format(idx+1,N))
UnboundLocalError: local variable 'idx' referenced before assignment

[asmagen]

Trying to run a part of the code shows an issue in

>>> transformation.init_translation(fixed_image)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/airlab/transformation/pairwise.py", line 180, in init_translation
    fixed_image_center_mass_x = th.sum(fixed_image.image.squeeze() * self._grid[..., 0]) / intensity_sum
RuntimeError: The size of tensor a (2846) must match the size of tensor b (2928) at non-singleton dimension 1

I ran:

import SimpleITK as sitk
import torch as th
import matplotlib.pyplot as plt
import time
import sys
import os
import airlab as al
import cv2
from skimage.transform import resize

start = time.time()

dtype = th.float32
device = th.device("cpu")

iterations=1000
lr=0.01
transform_type='similarity'
gpu_device=0
opt_cm=True
loss_fn='mse'

im1 = cv2.imread("/Users/Assaf/Dropbox/shared_imaging/CD20_test.png")
im1.shape
im2 = cv2.imread("/Users/Assaf/Dropbox/shared_imaging/CD3_test.png")
im2.shape

fixed_image = al.utils.image.create_tensor_image_from_itk_image(sitk.GetImageFromArray(cv2.cvtColor(im1, cv2.COLOR_BGR2GRAY)), dtype=th.float32, device=device)
moving_image = al.utils.image.create_tensor_image_from_itk_image(sitk.GetImageFromArray(cv2.cvtColor(im2, cv2.COLOR_BGR2GRAY)), dtype=th.float32, device=device)

fixed_image, moving_image = al.utils.normalize_images(fixed_image, moving_image)

# fixed_image.image = 1 - fixed_image.image
# moving_image.image = 1 - moving_image.image

registration = al.PairwiseRegistration()
transforms=dict(similarity=al.transformation.pairwise.SimilarityTransformation,
                   affine=al.transformation.pairwise.AffineTransformation,
                   rigid=al.transformation.pairwise.RigidTransformation)

# choose the affine transformation model
transformation = transforms[transform_type](moving_image, opt_cm=opt_cm)
# initialize the translation with the center of mass of the fixed image
transformation.init_translation(fixed_image)

[jlevy44]

I'm assuming you were able to install Airlab, but you can try:

pip install git+https://github.com/airlab-unibas/airlab.git

If the installation was unsuccessful. Setting gpu_device to 0 should work and actually consume lower memory than the CPU process, just make sure you're in the corect computing environment.

The index error was a bug:

 File "/usr/local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 407, in register_images_
    print("[{}/{}] - Writing registered sections to file.".format(idx+1,N))
UnboundLocalError: local variable 'idx' referenced before assignment

I pushed the code and it should be functional now. By the way, the fact that this flagged indicates that the registration did actually run, it just bugged before writing to file, so you can try the command line again. Try reinstalling:

pip install git+https://github.com/jlevy44/PathFlow-MixMatch.git --no-deps

You can see if GPU works better than CPU by changing gpu_device from -1 to 0.

In any case, try this command again:

pathflow-mixmatch register_images --im1 CD20_test.png --im2 CD3_test.png --fix_rotation False --output_dir ./ --gpu_device -1 --transform_type similarity --lr 0.01 --iterations 1000 --min_object_size 50000 --no_segment_analysis True --black_background True --verbose True

Hope some of that helps.. If you want to debug the code you had pasted above, you'd want to add this:

im1,im2=match_image_size(im1,im2,black_background=black_background)

[jlevy44]

Not sure about libgomp, might be something you want to discuss with your cluster admin?

[asmagen]

Thanks. I got an output (that doesn't seem very aligned well yet) and an error so I believe that output is not the registered image:

Loading images.
Performing regitration.
Writing registered section to file.
Traceback (most recent call last):
  File "/usr/local/bin/pathflow-mixmatch", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 472, in main
    fire.Fire(Commands)
  File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 138, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
  File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire
    target=component.__name__)
  File "/usr/local/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 469, in register_images
    verbose=verbose)
  File "/usr/local/lib/python3.7/site-packages/pathflow_mixmatch/cli.py", line 414, in register_images_
    cv2.imwrite(os.path.join(output_dir,os.path.basename(im2_fname).replace(file_ext,'_registered{}'.format(file_ext))),cv2.cvtColor(new_image,cv2.COLOR_BGR2RGB))
NameError: name 'new_image' is not defined

I also tried running the following for a mini image example (600x900), which worked but showed that the similarity and affine transformations aren't aligning the the cell groups (not individual cells) well enough:

import SimpleITK as sitk
import torch as th
import matplotlib.pyplot as plt
import time
import sys
import os
import airlab as al
import cv2
import numpy as np
from skimage.transform import resize

def match_image_size(img1,img2,black_background=False):
    white=int(black_background==False)
    fill_color=(np.array([255,255,255])*white).astype(int).tolist()
    dh=int(np.abs((img1.shape[0]-img2.shape[0])))
    if img1.shape[0]>img2.shape[0]:
        img2 = cv2.copyMakeBorder(img2, dh//2+dh%2, dh//2, 0, 0, cv2.BORDER_CONSTANT, value=fill_color)
    elif img1.shape[0]<img2.shape[0]:
        img1 = cv2.copyMakeBorder(img1, dh//2+dh%2, dh//2, 0, 0, cv2.BORDER_CONSTANT, value=fill_color)
    dw=int(np.abs((img1.shape[1]-img2.shape[1])))
    if img1.shape[1]>img2.shape[1]:
        img2 = cv2.copyMakeBorder(img2, 0, 0, dw//2+dw%2, dw//2, cv2.BORDER_CONSTANT, value=fill_color)
    elif img1.shape[1]<img2.shape[1]:
        img1 = cv2.copyMakeBorder(img1, 0, 0, dw//2+dw%2, dw//2, cv2.BORDER_CONSTANT, value=fill_color)
    return img1, img2

def displace_image(img, displacement, gpu_device):
    channels=[]
    for i in range(3):
        im=sitk.GetImageFromArray(img[...,i])
        im=al.utils.image.create_tensor_image_from_itk_image(im, dtype=th.float32, device=('cuda:{}'.format(gpu_device) if gpu_device>=0 else 'cpu'))
        channels.append(al.transformation.utils.warp_image(im, displacement).numpy())
    return np.uint8(np.stack(channels).transpose((1,2,0)))

start = time.time()

dtype = th.float32
device = th.device("cpu")

iterations=200
lr=0.01
transform_type='affine'
gpu_device=-1
opt_cm=True
loss_fn='mse'

im1 = cv2.imread("/Users/Assaf/Dropbox/shared_imaging/mini_CD20.png")
im1.shape
im2 = cv2.imread("/Users/Assaf/Dropbox/shared_imaging/mini_CD3.png")
im2.shape

im1,im2 = match_image_size(im1,im2,black_background=True)

fixed_image = al.utils.image.create_tensor_image_from_itk_image(sitk.GetImageFromArray(cv2.cvtColor(im1, cv2.COLOR_BGR2GRAY)), dtype=th.float32, device=device)
moving_image = al.utils.image.create_tensor_image_from_itk_image(sitk.GetImageFromArray(cv2.cvtColor(im2, cv2.COLOR_BGR2GRAY)), dtype=th.float32, device=device)

fixed_image, moving_image = al.utils.normalize_images(fixed_image, moving_image)

# fixed_image.image = 1 - fixed_image.image
# moving_image.image = 1 - moving_image.image

registration = al.PairwiseRegistration()
transforms=dict(similarity=al.transformation.pairwise.SimilarityTransformation,
                   affine=al.transformation.pairwise.AffineTransformation,
                   rigid=al.transformation.pairwise.RigidTransformation)

# choose the affine transformation model
transformation = transforms[transform_type](moving_image, opt_cm=opt_cm)
# initialize the translation with the center of mass of the fixed image
transformation.init_translation(fixed_image)

registration.set_transformation(transformation)

loss_fns=dict(mse=al.loss.pairwise.MSE,
            ncc=al.loss.pairwise.NCC,
            lcc=al.loss.pairwise.LCC,
            mi=al.loss.pairwise.MI,
            mgf=al.loss.pairwise.NGF,
            ssim=al.loss.pairwise.SSIM)

# choose the Mean Squared Error as image loss
image_loss = loss_fns[loss_fn](fixed_image, moving_image)

registration.set_image_loss([image_loss])

# choose the Adam optimizer to minimize the objective
optimizer = th.optim.Adam(transformation.parameters(), lr=lr, amsgrad=True)

registration.set_optimizer(optimizer)
registration.set_number_of_iterations(iterations)

# start the registration
registration.start()

# set the intensities back to the original for the visualisation
fixed_image.image = 1 - fixed_image.image
moving_image.image = 1 - moving_image.image

# warp the moving image with the final transformation result
displacement = transformation.get_displacement()
warped_image = al.transformation.utils.warp_image(moving_image, displacement)

end = time.time()

print("=================================================================")

print("Registration done in:", end - start, "s")
print("Result parameters:")
transformation.print()

new_img=displace_image(im2,displacement,gpu_device=gpu_device) # new tri, output he as well

fig, axes = plt.subplots(2, 2, figsize=(7, 6), sharex=True, sharey=True)
ax = axes.ravel()

ax[0].imshow(im1)
ax[0].set_title("REF")

ax[1].imshow(im2)
ax[1].set_title("SRC")

ax[2].imshow(warped_image.numpy(), cmap='gray')
ax[2].set_title("registered")

ax[3].imshow(new_img)
ax[3].set_title("registered")

for a in ax.ravel():
    a.axis('off')

fig.tight_layout()
fig.show()

See example output and note the group of cells in the red circle which are very far apart:

Thanks for your help

[asmagen]

It may be also very valuable to add the integration with drop2 deformation fields and allow to compare the results from the two registration pipelines. Also is there a non linear registration option in AirLab that we can use?

[jlevy44]

Ok, I fixed a bug that was not outputting the final registered image. I also pushed major updates to add the following non-linear registration techniques: 'non_parametric','bspline','wendland' , but these are untested, and also seeking to add masking and a preliminary linear alignment step before performing the non-linear methods. I think the soonest that I'll be able to get to these is this weekend, but you are welcome to try. Same for the deformation fields, it will probably take me about a week to push the code for those, but you are welcome to submit a PR. Please see some of the new parameters I have added and happy to chat more.

[jlevy44]

I would also experiment with the learning rate, number of training iterations and different loss functions.

[jlevy44]

Seems like as expected, the rigid/affine transforms were not able to completely fit your dataset.

[jlevy44]

Let's move this conversation over to #1. Closing this issue because --im1 works as intended.