search

QI2lab / merfish3d-analysis

3D MERFISH data processing
BSD 3-Clause "New" or "Revised" License
0 stars 0 forks source link

'registered_decon_data' key error #8

Closed julianantolin closed 3 months ago

julianantolin commented 3 months ago

Description

I am running test_pixeldecode.py and expect to have pixel-based decoding of an image stack.

What I Did

Run test_pixeldecode.py
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
File /home/julian/wf-merfish/test_pixeldecode.py:10
      [7](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:7) # dataset_path = Path("/home/julian//Downloads/50_mhd4_4bit_mk2/results/_b_x_rna_50_bg_0.01_temp_all/compatibility/processed/")
      [9](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:9) dataset_path = Path('/home/julian/Documents/rnaFISH/runs/20240329_zhuang_MOP_data/qi2lab_merfish/')
---> [10](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:10) decode_factory = PixelDecoder(dataset_path=dataset_path,
     [11](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:11)                               global_normalization_limits=[.1,80.0], # clipping to have data in [0,100%]
     [12](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:12)                               overwrite_normalization=True,
     [13](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:13)                               exp_type='3D',
     [14](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:14)                               merfish_bits=[16](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:16))
     16 tile_ids = decode_factory._tile_ids
     [18](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/test_pixeldecode.py:18) del decode_factory

File ~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:99, in PixelDecoder.__init__(self, dataset_path, tile_idx, exp_type, use_mask, scale_factors, global_normalization_limits, overwrite_normalization, z_range, include_blanks, merfish_bits, verbose)
     [97](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:97) else:
     [98](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:98)     if global_normalization_limits is not None:
---> [99](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:99)         self._global_normalization_factors(low_percentile_cut=global_normalization_limits[0],
    [100](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:100)                                            high_percentile_cut=global_normalization_limits[1],
    [101](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:101)                                            overwrite=overwrite_normalization)
    [102](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:102)     else:
    [103](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:103)         self._global_normalization_factors(overwrite=overwrite_normalization)

File ~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:219, in PixelDecoder._global_normalization_factors(self, low_percentile_cut, high_percentile_cut, camera_background, hot_pixel_threshold, overwrite)
    [216](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:216) current_bit = zarr.open(bit_dir_path,mode='r')
    [217](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:217) print(current_bit.keys())
    [218](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:218) current_image = cp.where(cp.asarray(current_bit["registered_ufish_data"], dtype=cp.float32) > 0.01, 
--> [219](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:219)                          cp.asarray(current_bit["registered_decon_data"], dtype=cp.float32), 
    [220](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:220)                          0.)
    [221](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:221) current_image[current_image<camera_background] = cp.median(current_image[current_image.shape[0]//2,:,:]).astype(cp.float32)
    [222](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/wf_merfish/postprocess/PixelDecoder.py:222) current_image[current_image>hot_pixel_threshold] = cp.median(current_image[current_image.shape[0]//2,:,:]).astype(cp.float32)

File ~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/zarr/hierarchy.py:511, in Group.__getitem__(self, item)
    [509](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/zarr/hierarchy.py:509)         raise KeyError(item)
    [510](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/zarr/hierarchy.py:510) else:
--> [511](https://file+.vscode-resource.vscode-cdn.net/home/julian/wf-merfish/~/miniconda3/envs/wf-merfish/lib/python3.10/site-packages/zarr/hierarchy.py:511)     raise KeyError(item)

KeyError: 'registered_decon_data'
dpshepherd commented 3 months ago

We need to generate the deconvolved data for this dataset again. Do you have a GPU?

julianantolin commented 3 months ago

Oh, ok. Yes, I have a GPU. Do I run the Zhuang data PIL file?

dpshepherd commented 3 months ago

Let me double check it is current and then yes.

dpshepherd commented 3 months ago

Should be good with b2b7ddc

You don't actually have to deconvolve again, because the data on BIL is already deconvolved. It just needs to be rewritten and the U-fish predictions run.

dpshepherd commented 3 months ago

To explain a bit better. For decoding the Zhuang data we don't need to generate registered and deconvolved data on our own, because the provided data is already registered and deconvolved.

What we did previously for Zhuang lab data was modify the PixelDecoder class to use the Zhuang provided data and registered_ufish_data for decoding. Traditionally, we have to generate registered and deconvolved data. So PixelDecoder expects there to also be an array registered_decon_data for that step.

To keep things consistent, I propose that you regenerate a data structure so that it matches the library's current data array expectations. This is a bit wasteful on space, but should be relatively fast to do.

julianantolin commented 3 months ago

I am running postprocess_zhuang_BIL.py to redo the file structure. Is this correct?

On Mon, Jun 10, 2024 at 3:36 PM Doug Shepherd @.***> wrote:

To explain a bit better - for Zhuang decoding don't need to generate deconvolved data on our own, because the provided data is already deconvolved.

What we did previously for Zhuang lab data was modify the PixelDecoder class to use their provided data and registered_ufish_data for decoding.

To keep things consistent, I propose that you regenerate the data structure so that it matches the library's current data array expectations. This is a bit wasteful on space, but should be relatively fast to do.

— Reply to this email directly, view it on GitHub https://github.com/QI2lab/wf-merfish/issues/8#issuecomment-2159424475, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABHWX2ITT4AGUONDUWFJRNLZGYS75AVCNFSM6AAAAABJDDD4GGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNJZGQZDINBXGU . You are receiving this because you authored the thread.Message ID: @.***>

dpshepherd commented 3 months ago

Yes. Please open a new issue with further questions.