search

ml-struct-bio / cryodrgn

Neural networks for cryo-EM reconstruction
http://cryodrgn.cs.princeton.edu
GNU General Public License v3.0
322 stars 76 forks source link

cryodrgn downsample crashes when trying to read a .cs file with a large header #418

Open Guillawme opened 3 weeks ago

Guillawme commented 3 weeks ago

Describe the bug With CryoDRGN 3.4.1, all these commands crash when trying to read a .cs file with a large header:

To Reproduce Run any of these three commands on a .cs file with 100 alignments_class3D_* entries in the header. These entries result from having run a 3D classification with 100 classes earlier in the job graph. They are no longer relevant to the final refinement job, but CryoSPARC keeps them in the particle metadata, and I don't think it provides a way to remove these entries.

I don't know what threshold of header size triggers this behavior, and can't test it easily.

Expected behavior This shouldn't cause a crash. The irrelevant header entries should be dropped and the command should complete normally.

Additional context Initially, this happened with a very large .cs file (~6.5 GB) because it's a large particle set. I thought the total file size was the problem, so I repeated this on a random subset of 100 000 particles, which should be workable and gave a much smaller .cs file (~850 MB), but I got the same error.

I prepared a .cs file with only 100 particles (~950 KB) that also causes the crash because it has a full header. I attached this .cs file in case this is helpful for debugging (I needed to add a .txt extension for GitHub to accept to upload it, you might need to rename the file to remove this extension). J156_split_0_exported.cs.txt

Error for downsample:

Traceback (most recent call last):
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/bin/cryodrgn", line 8, in <module>
    sys.exit(main_commands())
             ^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/command_line.py", line 81, in main_commands
    _get_commands(
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/command_line.py", line 76, in _get_commands
    args.func(args)
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/commands/downsample.py", line 241, in main
    src = ImageSource.from_file(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/source.py", line 172, in from_file
    return CsSource(
           ^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/source.py", line 559, in __init__
    csdata = np.load(filepath)
             ^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/npyio.py", line 456, in load
    return format.read_array(fid, allow_pickle=allow_pickle,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/format.py", line 784, in read_array
    shape, fortran_order, dtype = _read_array_header(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/format.py", line 604, in _read_array_header
    raise ValueError(
ValueError: Header info length (80116) is large and may not be safe to load securely.
To allow loading, adjust `max_header_size` or fully trust the `.npy` file using `allow_pickle=True`.
For safety against large resource use or crashes, sandboxing may be necessary.

Error for parse_pose_csparc:

Traceback (most recent call last):
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/bin/cryodrgn", line 8, in <module>
    sys.exit(main_commands())
             ^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/command_line.py", line 81, in main_commands
    _get_commands(
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/command_line.py", line 76, in _get_commands
    args.func(args)
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/commands/parse_pose_csparc.py", line 39, in main
    data = np.load(args.input)
           ^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/npyio.py", line 456, in load
    return format.read_array(fid, allow_pickle=allow_pickle,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/format.py", line 784, in read_array
    shape, fortran_order, dtype = _read_array_header(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/format.py", line 604, in _read_array_header
    raise ValueError(
ValueError: Header info length (80116) is large and may not be safe to load securely.
To allow loading, adjust `max_header_size` or fully trust the `.npy` file using `allow_pickle=True`.
For safety against large resource use or crashes, sandboxing may be necessary.

Error for parse_pose_ctf:

Traceback (most recent call last):
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/bin/cryodrgn", line 8, in <module>
    sys.exit(main_commands())
             ^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/command_line.py", line 81, in main_commands
    _get_commands(
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/command_line.py", line 76, in _get_commands
    args.func(args)
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/cryodrgn/commands/parse_ctf_csparc.py", line 32, in main
    metadata = np.load(args.cs)
               ^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/npyio.py", line 456, in load
    return format.read_array(fid, allow_pickle=allow_pickle,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/format.py", line 784, in read_array
    shape, fortran_order, dtype = _read_array_header(
                                  ^^^^^^^^^^^^^^^^^^^
  File "/home/guillaume/opt/miniconda3/envs/cryodrgn-3.4.1/lib/python3.11/site-packages/numpy/lib/format.py", line 604, in _read_array_header
    raise ValueError(
ValueError: Header info length (80116) is large and may not be safe to load securely.
To allow loading, adjust `max_header_size` or fully trust the `.npy` file using `allow_pickle=True`.
For safety against large resource use or crashes, sandboxing may be necessary.
kvr2007 commented 3 weeks ago

I have to say that the csparc's data model is quite messy, and cannot exclude this is done on purpose :/

So a few workarounds to try to remove 3D classification metadata: 1) use the particle stack before 3D classification, cryoDRGN only needs particle images and CTF information, but not the class assignment 2) run csparc restacking job on the output of 3D classification and see if *.cs files from restacking work 3) convert to Relion STAR files and then import to cryoDRGN

I would also recommend to start with a smaller subset, say 100-200k particles. This way you can try different cryoDRGN settings relatively quick.

Guillawme commented 3 weeks ago

Yes, I could also come up with some workarounds (haven't had time to test any of them yet). But regardless, it seems to me like a .cs file smaller than 1 MB crashing three commands is a bug worth fixing.