search

arayabrain / barebone-studio

pipeline tool
GNU General Public License v3.0
1 stars 3 forks source link

[104][inscopix]tiff画像 処理ノード投入検証 #216

Closed itutu-tienday closed 10 months ago

itutu-tienday commented 11 months ago

Inscopixデータファイル（.isxd）からexportされたtiffファイルを、suite2p, caiman で処理できるように調整する。

hunghongnam commented 11 months ago

Đối ứng thêm ảnh suite2p inscopix tiff

Điều chỉnh file TIFF được export từ ​​file data Inscopix (.isxd) để có thể được xử lý với Suite2p và caiman.

Input sample date

https://drive.google.com/drive/u/0/folders/1fDYGtumb8h3lZBcFI5MB8ifDH0ximbKH ISXD-movie-Export- result.tif

Bổ sung cho data

Thông tin đã điều tra trước

Với suite2p, có vẻ như không thể xử lý. Nội dung

Caima có vẻ OK. Tuy nhiên, kiểm tra trạng thái support chính xác.

Goal Đưa file Inscopix tiff vào Workflow với suite2p, caiman và xử lý thành công tới khi xuất ROI. Xác nhận thông số kỹ thuật cho trạng thái đối ưứn Bigtiff của suite2p, caiman

Liên quan khác

[104][Inscopix]データ仕様把握 #197 [104][Inscopix]Workflow Data Node組み込み #200

quanpython commented 11 months ago

According to my investigation, the results achieved are different from yours. Please test it again and provide the results along with your environment information.

As a result of running the workflow with tiff image input isxd-movie-export-test-result.tif, the execution process appears different between OS, in my case is Linux (CPU i5 gen10, RAM 16GB) and MacOS (M1):

Compare with your debug result in suite2p/io/tiff.py:
tif, Ltif = open_tiff(file, use_sktiff)
My debug results: tif.asarray().shape = (2996, 474, 538) and Ltif = 2996

itutu-tienday commented 11 months ago

@quanpython Thank you.

Sorry, the data confirming the situation I was describing in my "pre-researched information" is

Also, the operating environment should be Mac.

@ReiHashimoto Regarding the above operation check, was your environment Mac?

@quanpython Could you reconfirm the above data (it is large, about 3 GB)? *Note that "isxd-movie-export-test-result.tif" is also the original data from Inscopix, so it is necessary to make it available as well.

ReiHashimoto commented 11 months ago

@quanpython CC: @itutu-tienday

Though I'm running again, I leave some comments for now.

Regarding the above operation check, was your environment Mac?

Yes, I checked in MacOS (Apple Silicon, M2, RAM 16GB).

Compare with your debug result in suite2p/io/tiff.py: tif, Ltif = open_tiff(file, use_sktiff) My debug results: tif.asarray().shape = (2996, 474, 538) and Ltif = 2996

Apologies, I should have used permanent link. Some codes changed since I checked. This is the code I referred to. https://github.com/MouseLand/suite2p/blob/118901ac15c6881502c65e011a46fbca16e7a52d/suite2p/io/tiff.py#L183-L184

At this line, array im's all data got to 0.

FYI, I tried using notebook(suite2p.ipynb in GoogleDrive shared folder) on local to debug this process. (with conda env python=3.8, installed suite2p by pip.)

ReiHashimoto commented 11 months ago

@quanpython I tried suite2p_file_convert with recording_20160613_105808-PP-PP-BP-MC-DFF.tiff again, but it got the same result as recording_20160613_105808-PP-PP-BP-MC-DFF-shrink_suite2p

Screenshot 2023-12-08 at 15 37 24

Running with recording_20160613_105808-PP-PP-BP-MC-DFF.tiff takes time, I added reduced image recording_20160613_105808-PP-PP-BP-MC-DFF-shrink to Google Drive.

quanpython commented 11 months ago

@itutu-tienday @ReiHashimoto Thank you for the additional information you provided.

I have debugged at function tiff_to_binary.The values in the im array are not within the valid range for the np.int16 data type, so they are getting rounded down to zero during the conversion. If they are outside this range, it is necessary to normalize or scale it before converting it to np.int16 to ensure that it fits within the valid range. I adjusted the code as follows and got the expected output:

# check if uint16
if im.dtype.type == np.uint16:
    im = (im // 2).astype(np.int16)
elif im.dtype.type == np.int32:
    im = (im // 2).astype(np.int16)
+ elif im.dtype.type == np.float32:
+   im = (im - im.min()) / (im.max() - im.min())
+   im = (im * np.iinfo(np.int16).max).astype(np.int16)
elif im.dtype.type != np.int16:
    im = im.astype(np.int16)

Because this is a problem with the suite2p library, I would like to ask you if this is the right way to solve it in the current system: Apply above modification and move all content of tiff_to_binary method to suite2p_file_convert is a reasonable solution, isn't it? If you have a better idea, please let me know

ReiHashimoto commented 11 months ago

@quanpython Thank you for your debugging and suggestion.

How about converting image data's array itself to np.int16 before passing data to suite2p? It means, implementation as following comment outs.

studio/app/optinist/wrappers/suite2p/file_convert.py

def suite2p_file_convert(
    image: ImageData, output_dir: str, params: dict = None, **kwargs
) -> dict(ops=Suite2pData):

    ...

    data_path_list = []
    data_name_list = []
    for file_path in image.path:

        # Check file_path's array dtype.
        # If dtype is float32, create converted image data here
        # use converted file's path for suite2p

        data_path_list.append(os.path.dirname(file_path))
        data_name_list.append(os.path.basename(file_path))
quanpython commented 11 months ago

@ReiHashimoto Yes, that is also a reasonable solution, but reading and writing files there will increase processing time