search

AllenInstitute / AllenSDK

code for reading and processing Allen Institute for Brain Science data
https://allensdk.readthedocs.io/en/latest/
Other
343 stars 149 forks source link

Troubleshoot file splitting on MESO.2 #1771

Closed wbwakeman closed 3 years ago

wbwakeman commented 3 years ago

For the new Mesoscope/Multiscope rig MESO.2, there is an updated version of the ScanImage software SI2020. One effect of this is that the TIFF header information has changed relative to the previous version of Mesoscope (MESO.1).

ScanImage 2016 TIFF Specification

ScanImage 2019 TIFF Specification

From Vidrio:

these looks like structural differences between our current SI implementation and 2016. The SI structure embedded into the TIff file is actually the saveable properties of the hSI object in ScanImage. Since we recently overhauled the structure of the SI object, it will break parsers which are used to earlier SI versions.

SI.hFastZ.userZs -> SI.hStackManager.arbitraryZs SI.hFastZ.numFramesPerVolume -> SI.hStackManager.numFramesPerVolume SI.hFastZ.numVolumes -> SI.hStackManager.numVolumes SI.hStackmanager.zs -> SI.hStackManager.hZs

I am not familiar with the modifications implemented by Georg on this system so let me know if I made a bad assumption with these mappings.

The python reader should work but I believe you will face the same issue because it works around the metadata changes by simply spitting the structure out. That said, I do not believe anyone in Vidrio is currently maintaining this code so support will be limited there.

Can e-mail support@vidriotech.com with questions

Made one attempt to run a file splitting job with the version check disabled.  Below is the stack trace for that run:

Traceback (most recent call last):
  File "/allen/aibs/technology/conda/shared/miniconda/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/allen/aibs/technology/conda/shared/miniconda/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/allen/aibs/informatics/kschelon/wayne-mesoscope-test/wayne-meso/lib/python3.6/site-packages/ophys_etl/pipelines/brain_observatory/scripts/run_mesoscope_splitting.py", line 261, in <module>
    main()
  File "/allen/aibs/informatics/kschelon/wayne-mesoscope-test/wayne-meso/lib/python3.6/site-packages/ophys_etl/pipelines/brain_observatory/scripts/run_mesoscope_splitting.py", line 216, in main
    out, meta = split_z(localz, exp, **h5_opts)
  File "/allen/aibs/informatics/kschelon/wayne-mesoscope-test/wayne-meso/lib/python3.6/site-packages/ophys_etl/pipelines/brain_observatory/scripts/run_mesoscope_splitting.py", line 66, in split_z
    mt = MesoscopeTiff(input_tif)
  File "/allen/aibs/informatics/kschelon/wayne-mesoscope-test/wayne-meso/lib/python3.6/site-packages/ophys_etl/transforms/mesoscope_2p/tiff.py", line 122, in __init__
    self._frame_data, self._roi_data = tiff_header_data(source_tiff)
  File "/allen/aibs/informatics/kschelon/wayne-mesoscope-test/wayne-meso/lib/python3.6/site-packages/ophys_etl/transforms/mesoscope_2p/metadata.py", line 99, in tiff_header_data
    frame_data, roi_data = _tiff_header_data_2017b_v1(f)
  File "/allen/aibs/informatics/kschelon/wayne-mesoscope-test/wayne-meso/lib/python3.6/site-packages/ophys_etl/transforms/mesoscope_2p/metadata.py", line 82, in _tiff_header_data_2017b_v1
    stripnull(fp.read(frame_data_size)).decode('utf-8'))
  File "/allen/aibs/technology/conda/shared/miniconda/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/allen/aibs/technology/conda/shared/miniconda/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/allen/aibs/technology/conda/shared/miniconda/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
wbwakeman commented 3 years ago

Ross from MPE will look at this starting January 11, 2021

rhytnen commented 3 years ago

Wayne reported that new commits to my PR have resolved this issue.

djkapner commented 3 years ago

@wbwakeman can you link here to some non-archived mesoscope so I can check this PR does not break MESO1 splitting?

wbwakeman commented 3 years ago

@djkapner 

/allen/aibs/technology/waynew/allen/programs/braintv/production/incoming/neuralcoding/1053297451_timeseries.tiff
/allen/aibs/technology/waynew/allen/programs/braintv/production/incoming/neuralcoding/1081677132_timeseries.tiff
/allen/programs/braintv/production/visualbehavior/prod0/specimen_840542948/ophys_session_914224851/914224851_timeseries.tiff