Issue with the pickle protocol for big images

[FiReTiTi]

Hi, I'm running CellPose on really big images (about 30K x 30K), and I get an error coming likely from the pickle protocol:

Traceback (most recent call last):
  File "/home/groups/graylab_share/OMERO.rdsStore/thibaulg/Python/Python38_Test/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/home/groups/graylab_share/OMERO.rdsStore/thibaulg/Python/Python38_Test/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/groups/graylab_share/OMERO.rdsStore/thibaulg/Python/Python38_Test/lib/python3.8/site-packages/cellpose/__main__.py", line 458, in <module>
    main()
  File "/home/groups/graylab_share/OMERO.rdsStore/thibaulg/Python/Python38_Test/lib/python3.8/site-packages/cellpose/__main__.py", line 342, in main
    io.masks_flows_to_seg(image, masks, flows, diams, image_name, channels)
  File "/home/groups/graylab_share/OMERO.rdsStore/thibaulg/Python/Python38_Test/lib/python3.8/site-packages/cellpose/io.py", line 285, in masks_flows_to_seg
    np.save(base+ '_seg.npy',
  File "<__array_function__ internals>", line 5, in save
  File "/home/groups/graylab_share/OMERO.rdsStore/thibaulg/Python/Python38_Test/lib/python3.8/site-packages/numpy/lib/npyio.py", line 529, in save
    format.write_array(fid, arr, allow_pickle=allow_pickle,
  File "/home/groups/graylab_share/OMERO.rdsStore/thibaulg/Python/Python38_Test/lib/python3.8/site-packages/numpy/lib/format.py", line 680, in write_array
    pickle.dump(array, fp, protocol=3, **pickle_kwargs)
OverflowError: serializing a bytes object larger than 4 GiB requires pickle protocol 4 or higher

Is there a workaround?

I run CellPose on a cluster running on CentOS with Slurm, on P100 or V100 GPUs with 700Go or RAM.

[winfrees]

I was having the same error on mesoscale datasets until I disabled npy output with --no_npy.

[FiReTiTi]

Thank you, it worked.