search

nyukat / mammography_metarepository

Meta-repository of screening mammography classifiers
https://arxiv.org/abs/2108.04800
BSD 2-Clause "Simplified" License
65 stars 11 forks source link

Unsupported pickle protocol: 5 #1

Closed jamesjjcondon closed 3 years ago

jamesjjcondon commented 3 years ago 
drjc@home:~/mydev/mammography_metarepository$ sudo bash run.sh nyu_gmic test01 '/data1/NYU_retrain/train_ims_mini/gmic_pngs' '/data1/NYU_metarepo/cropped_pngs' '/data1/NYU_metarepo/gmic_exam_list_before_cropping.pkl' '/data1/NYU_metarepo/predictions' gpu 0
Expand preprocessed image path
Creating subdirectory for experiment
Directory already exists
Expanding path to images

Used model: nyu_gmic.
Device: gpu, number: 0.
Name of the experiment: test01
Path to images: /data1/NYU_retrain/train_ims_mini/gmic_pngs
Path to the pickle file: /data1/NYU_metarepo/gmic_exam_list_before_cropping.pkl
Path for saving predictions: /data1/NYU_metarepo/predictions

Model already built, running.
Model extra arguments:
 - MODEL_INDEX=1
 - NUM_PROCESSES=2
 - PREPROCESS_FLAG=True

Starting docker container for nyu_gmic model.
f86481e97b030180c15841437715c912b70066c9a745fc7978e6f18d480a4dbe

Running predict.sh for nyu_gmic model.
Stage 1: Crop Mammograms
Traceback (most recent call last):
  File "src/cropping/crop_mammogram.py", line 365, in <module>
    buffer_size=args.buffer_size,
  File "src/cropping/crop_mammogram.py", line 254, in crop_mammogram
    exam_list = pickling.unpickle_from_file(exam_list_path)
  File "/home/gmic/GMIC/src/utilities/pickling.py", line 34, in unpickle_from_file
    return pickle.load(handle)
ValueError: unsupported pickle protocol: 5

Maybe upgrade to pickle5?

I'll try saving differently...

jamesjjcondon commented 3 years ago

Saving my exam_list.pkl with the default protocol kwarg (4) makes no difference. Any ideas?

jwitos commented 3 years ago

Thanks for the report @jamesjjcondon. I can reproduce this when saving our sample_data pickle file with protocol version 5. I guess this is expected as GMIC supports Python 3.6, and protocol version 5 was only introduced in Python 3.8. However, when saving the pickle file with protocol version 4 it works fine. Could you double check if you are saving with protocol=4 argument? If that doesn't work, please let me know how your environment looks like so I can try to reproduce it. Also, lower protocol versions (0-3) should be compatible with very old Python releases, although at the cost of performance. I tried protocol versions 0 and 3 and it worked fine too.

jwitos commented 3 years ago

cc @chledowski @bstadnick

jamesjjcondon commented 3 years ago

Yep, sorry, #path_fail, it's cropping...

jwitos commented 3 years ago

No problem, this is very useful. We will include information about the pickle protocol version (probably version 3 is a safe bet... will discuss) in the README.