Failed to load mrc

[jdeschamps]

Hi!

Warning: I have a complete lack of understanding of how mrc files are read/written. I also understand that the plugin is in development!

When trying to open an mrc file from a collaborator, I get the following error:

ValueError                                Traceback (most recent call last)
File src/psygnal/_signal.py:912, in psygnal._signal.SignalInstance._run_emit_loop()

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py:207, in FunctionGui.__init__.<locals>._disable_button_and_call()
    206 try:
--> 207     self.__call__()
        self = <FunctionGui file_reader(files: List[pathlib.Path] = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex: List[str] = [], names: List[str] = [], as_dask_array: bool = True) -> <function NewType.<locals>.new_type at 0x7f376cfd7f70>>
    208 finally:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/magicgui/widgets/_function_gui.py:318, in FunctionGui.__call__(self=<FunctionGui file_reader(files: List[pathlib.Pat... = True) -> <function NewType.<locals>.new_type>>, update_widget=False, *args=(), **kwargs={})
    317 with _function_name_pointing_to_widget(self):
--> 318     value = self._function(*bound.args, **bound.kwargs)
        self = <FunctionGui file_reader(files: List[pathlib.Path] = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex: List[str] = [], names: List[str] = [], as_dask_array: bool = True) -> <function NewType.<locals>.new_type at 0x7f376cfd7f70>>
        bound = <BoundArguments (files=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex=[], names=[], as_dask_array=True)>
        self._function = <function file_reader at 0x7f36fcc02670>
    320 self._call_count += 1

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/blik/widgets/file_reader.py:29, in file_reader(files=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), name_regex=[], names=[], as_dask_array=True)
     23 """
     24 Read files with blik.
     25 
     26 name_regex: a regex string. Matching text will be used as name for the piece of data
     27 names: only load data matching this comma separated list of names
     28 """
---> 29 return read_layers(*files, name_regex=name_regex or None, names=names or None, lazy=as_dask_array)
        files = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),)
        name_regex or None = None
        names or None = None
        as_dask_array = True
        name_regex = []
        names = []

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/blik/reader.py:73, in read_layers(*paths=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), **kwargs={'lazy': True, 'name_regex': None, 'names': None})
     72 def read_layers(*paths, **kwargs):
---> 73     data_list = read(*paths, **kwargs)
        paths = (PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),)
        kwargs = {'name_regex': None, 'names': None, 'lazy': True}
     74     layers = []

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/naaf/reading/main.py:76, in read(name_regex=None, names=None, strict=False, lazy=True, *paths=(PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'),), **kwargs={})
     75 try:
---> 76     data.extend(read_file(file, name_regex=name_regex, lazy=lazy, **kwargs))
        data = []
        file = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
        name_regex = None
        lazy = True
        kwargs = {}
     77 except ParseError:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/naaf/reading/main.py:35, in read_file(file_path=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), **kwargs={'lazy': True, 'name_regex': None})
     34 try:
---> 35     data = listify(func(file_path, **kwargs))
        func = <function read_mrc at 0x7f36fc695790>
        file_path = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
        kwargs = {'name_regex': None, 'lazy': True}
     36     for d in data:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/naaf/reading/mrc.py:16, in read_mrc(image_path=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), name_regex=None, lazy=True, **kwargs={})
     14 name = guess_name(image_path, name_regex)
---> 16 with mrcfile.mmap(image_path) as mrc:
        image_path = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
     17     if lazy:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/load_functions.py:208, in mmap(name=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), mode='r', permissive=False)
    185 """Open a memory-mapped MRC file.
    186 
    187 This allows much faster opening of large files, because the data is only
   (...)
    206     An :class:`~mrcfile.mrcmemmap.MrcMemmap` object.
    207 """
--> 208 return MrcMemmap(name, mode=mode, permissive=permissive)
        name = PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc')
        mode = 'r'
        permissive = False

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcfile.py:114, in MrcFile.__init__(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'), name=PosixPath('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc'), mode='r', overwrite=False, permissive=False, header_only=False, **kwargs={})
    113     else:
--> 114         self._read(header_only)
        self = MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r')
        header_only = False
    115 except Exception:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcfile.py:130, in MrcFile._read(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'), header_only=False)
    129 self._iostream.seek(0)
--> 130 super(MrcFile, self)._read(header_only)
        header_only = False
        self = MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r')
    132 # Check if the file is the expected size.

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcinterpreter.py:173, in MrcInterpreter._read(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'), header_only=False)
    154 """Read the header, extended header and data from the I/O stream.
    155 
    156 Before calling this method, the stream should be open and positioned at
   (...)
    171          as a valid MRC file and ``permissive`` is :data:`True`.
    172 """
--> 173 self._read_header()
        self = MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r')
    174 self._read_extended_header()

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcinterpreter.py:215, in MrcInterpreter._read_header(self=MrcMemmap('/home/joran.deschamps/Desktop/projects/Helen/blik/21.mrc', mode='r'))
    214 try:
--> 215     byte_order = utils.byte_order_from_machine_stamp(header.machst)
        header = rec.array((1024, 1024, 512, 2, 0, 0, 0, 1024, 1024, 512, (1024., 1024., 512.), (90., 90., 90.), 1, 2, 3, -995.9688, 480.2282, -2.2351742e-07, 0, 0, b'\x00\x00\x00\x00\x00\x00\x00\x00', b'', 0, b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', (0., 0., 0.), b'MAP ', [68, 32, 32, 32], -1., 0, [b'', b'', b'', b'', b'', b'', b'', b'', b'', b'']),
          dtype=[('nx', '<i4'), ('ny', '<i4'), ('nz', '<i4'), ('mode', '<i4'), ('nxstart', '<i4'), ('nystart', '<i4'), ('nzstart', '<i4'), ('mx', '<i4'), ('my', '<i4'), ('mz', '<i4'), ('cella', [('x', '<f4'), ('y', '<f4'), ('z', '<f4')]), ('cellb', [('alpha', '<f4'), ('beta', '<f4'), ('gamma', '<f4')]), ('mapc', '<i4'), ('mapr', '<i4'), ('maps', '<i4'), ('dmin', '<f4'), ('dmax', '<f4'), ('dmean', '<f4'), ('ispg', '<i4'), ('nsymbt', '<i4'), ('extra1', 'V8'), ('exttyp', 'S4'), ('nversion', '<i4'), ('extra2', 'V84'), ('origin', [('x', '<f4'), ('y', '<f4'), ('z', '<f4')]), ('map', 'S4'), ('machst', 'u1', (4,)), ('rms', '<f4'), ('nlabl', '<i4'), ('label', 'S80', (10,))])
        utils = <module 'mrcfile.utils' from '/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/utils.py'>
    216 except ValueError as err:

File ~/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/utils.py:199, in byte_order_from_machine_stamp(machst=<class 'numpy.ndarray'> (4,) uint8)
    198 pretty_bytes = pretty_machine_stamp(machst)
--> 199 raise ValueError("Unrecognised machine stamp: " + pretty_bytes)
        pretty_bytes = '0x44 0x20 0x20 0x20'
        "Unrecognised machine stamp: " + pretty_bytes = 'Unrecognised machine stamp: 0x44 0x20 0x20 0x20'

ValueError: Unrecognised machine stamp: 0x44 0x20 0x20 0x20

The above exception was the direct cause of the following exception:

EmitLoopError                             Traceback (most recent call last)
File ~/miniconda3/envs/blik/lib/python3.9/site-packages/magicgui/widgets/_bases/value_widget.py:57, in ValueWidget._on_value_change(self=PushButton(value=False, annotation=None, name='call_button'), value=False)
     55 if value is self.null_value and not self._nullable:
     56     return
---> 57 self.changed.emit(value)
        value = False
        self.changed = <SignalInstance 'changed' on PushButton(value=False, annotation=None, name='call_button')>
        self = PushButton(value=False, annotation=None, name='call_button')

File src/psygnal/_signal.py:849, in psygnal._signal.SignalInstance.emit()

File src/psygnal/_signal.py:891, in psygnal._signal.SignalInstance._run_emit_loop()

File src/psygnal/_signal.py:892, in psygnal._signal.SignalInstance._run_emit_loop()

File src/psygnal/_signal.py:914, in psygnal._signal.SignalInstance._run_emit_loop()

EmitLoopError: calling <function FunctionGui.__init__.<locals>._disable_button_and_call at 0x7f36fc5c7790> with args=() caused ValueError in emit loop.
^CTraceback (most recent call last):
  File "/home/joran.deschamps/miniconda3/envs/blik/bin/napari", line 8, in <module>
    sys.exit(main())
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/__main__.py", line 447, in main
    _run()
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/__main__.py", line 336, in _run
    run(gui_exceptions=True)
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/_qt/qt_event_loop.py", line 402, in run
    app.exec_()
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/contextlib.py", line 126, in __exit__
    next(self.gen)
  File "/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/napari/_qt/utils.py", line 459, in _maybe_allow_interrupt
    old_sigint_handler(*handler_args)

I installed the env following https://github.com/gutsche-lab/blik/issues/130, with naaf==0.2.3, which may or may not explain the problem.

[brisvag]

Hi! This seems to be unrelated to #130. The issue is deeper (with mrcfile):

ValueError: Unrecognised machine stamp: 0x44 0x20 0x20 0x20

Something seems to be off with the header of your mrc file, which is not recognized by mrcfile. Can you try to manually open the file by doing this in ipython?:

with mrcfile.open('/path/to/file.mrc', 'r', header_only=True) as mrc:
    print(mrc.header)

If this fails again, try:

with mrcfile.open('/path/to/file.mrc', 'r', header_only=True, permissive=True) as mrc:
    print(mrc.header)

I didn't add any machinery to allow permissive=True for now; if that solves it, maybe I should consider setting it to True by default.

[jdeschamps]

It fails without permissive=True, but loads with it:

/home/joran.deschamps/miniconda3/envs/blik/lib/python3.9/site-packages/mrcfile/mrcinterpreter.py:219: 
RuntimeWarning: Unrecognised machine stamp: 0x44 0x20 0x20 0x20
  warnings.warn(str(err), RuntimeWarning)
(1024, 1024, 512, 2, 0, 0, 0, 1024, 1024, 512, (1024., 1024., 512.), (90., 90., 90.), 1, 2, 3, -995.9688, 480.2282, 
-2.2351742e-07, 0, 0, b'\x00\x00\x00\x00\x00\x00\x00\x00', b'', 0, b'\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00', (0., 0., 0.), b'MAP ', [68, 32, 32, 32], -1., 0, 
[b'', b'', b'', b'', b'', b'', b'', b'', b'', b''])

[brisvag]

Yeah, this happens if the header is a bit messed up for some reason. Corrupted file, or maybe it was written by a software that wasn't very careful about writing correct information in the header. If you want a quick fix, you can edit the code in naaf for the mrc reader function and add the permissive=True argument to the mrcfile.mmap call. I will try to release a half update to naaf and blik to fix both this issue and the other you raised.

[jdeschamps]

Thanks a lot! I will let you be the judge of whether to keep this issue open!

FYI the file was saved using imod, and there is no issue opening it there.

[brisvag]

You might want to bring this issue up to the mrcfile repo then, since this can only be solved there.

In my experience, imod is super resilient to broken headers, but because of this it might be less careful in writing correct ones :P

[jdeschamps]

The joy of open source dependencies :D

https://github.com/ccpem/mrcfile/issues/47

[brisvag]

@jdeschamps I just released v0.3.2 as an intermediate version before napari 0.4.17. Feel free to try it out and let me know if things work :)

[brisvag]

@jdeschamps update: v0.3.3 is actually fixed, sorry :P

[jdeschamps]

Awesome thanks, I'll let you know if anything arises!