POE analysis fails on run_starfysh

[Jorges1000]

Great package! When trying to run

model, loss = utils.run_starfysh(visium_args,
                                 n_repeats=n_repeats,
                                 epochs=epochs,
                                 poe=True, 
                                 # Turn on/off for model with/without histology integration
                                 device=device) # 16'

I get the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/starfysh/utils.py", line 440, in run_starfysh
    result = train_func(model, trainloader, device, optimizer)
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/starfysh/starfysh.py", line 821, in train_poe
    img_outputs = model.predictor_img(img)  # inference & generative for 2D img. data
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/starfysh/starfysh.py", line 507, in predictor_img
    hidden_z = self.img_z_enc(y_n)
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
    return forward_call(*args, **kwargs)
  File "/shared/apps/anaconda3/lib/python3.9/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
RuntimeError: mat1 and mat2 shapes cannot be multiplied (32x3072 and 1024x256)

When running with poe = False it works. I attach the model and the img_metadata. model_img.zip

Thanks for any help!

[Jorges1000]

setting up hchannel=True on preprocess_image (#31 run_starfysh error with poe argument) works but the image looks horrible:

[YinuoJin]

Hi @Jorges1000,

Thanks for using Starfysh! We'll try to reproduce the RGB dimension error you reported and get back later (will release a new version if it's a confirmed package error).

For your follow-up question with 1-D PoE, Starfysh performs H&E color deconvolution to pick only hematoxylin channels via histomicstk link, where we empirically found the positive correlations between H-channel and cell density.

From the uploaded image, the color segmentation picks up a lot of background noise distribution (repeated white blobs) + fiducial points (boundary black dots). However, Starfysh only input pixel patches mapped with valid Visium coordinates (img_metadata['map_info'][['imagerow', 'imagecol']]) while most of the background values should be dropped. Does your raw tissue consist of four separate tissues?

[Jorges1000]

Hi, thanks for the reply, it seems to be enhancing small differences in the background of the slide. Here is the source image:

[Jorges1000]

And here is the estimated tissue density; it seems that the background is affecting it more than cell density

[YinuoJin]

Thanks for the follow-up. We fixed the RGB issues and you can install through the GitHub version to run histology integration. I also looked into the 1D PoE: the Haematoxylin channel might either captures nucleus or the background depending on the color comparison w/ background (examples attached). We might extend it as a user option whether to flip the 1D intensities.

• Provided histology:

• Our sample histology vs. smoothed library size:

Please let me know which intensity preprocessing setup makes more sense to you?

[Jorges1000]

Hi @YinuoJin, many thanks for the improvements, i will try soon. I think there may be a benefit of applying a threshold to the H-channel to eliminate low counts, as it appears that the nuclei intensities are well above the background.