search

modestyachts / CIFAR-10.1

Release of CIFAR-10.1, a new test set for CIFAR-10.
MIT License
220 stars 19 forks source link

"Failed to interpret file %s as a pickle" % repr(file)) #1

Closed SivanDoveh closed 4 years ago

SivanDoveh commented 5 years ago

HI,

I'm trying to load 'cifar10.1_v4_data.npy' with np.load, but got the following message: OSError: Failed to interpret file '/home/sivan/darts/cnn/data_cifar_10_1/cifar10.1_v4_data.npy' as a pickle

The answers I found on google is to save differently the files. I wanted to ask, how do you load the datasets? looking in utils it seems you also use np.load. Did you get this message before?

Thank you! Sivan

ludwigschmidt commented 5 years ago

Hi Sivan,

I don't recall seeing this error before. What version of Python and NumPy are you using?

Thanks, Ludwig

SivanDoveh commented 5 years ago

Thank you for the quick reply!

Python 3.5.5 :: Anaconda custom (64-bit) numpy: 1.14.3

ludwigschmidt commented 5 years ago

Can you past the exact code you used here? I just executed the cells in the inspect_dataset_simple.ipynb notebook locally and didn't get an error. I used Python 3.7, but that probably shouldn't make a difference. Just to be safe though: can you see if you get the same error with Python 3.6 or 3.7?

SivanDoveh commented 5 years ago

I tried to run this with python 3.6.8 and 3.7.2 and got the same error.

This is the code: data_10_1 = np.load("/home/sivan/darts/cnn/data_cifar_10_1/cifar10.1_v4_data.npy") labels_10_1 = np.load('/home/sivan/darts/cnn/data_cifar_10_1/cifar10.1_v4_labels.npy')

Thank you!

ludwigschmidt commented 5 years ago

Very strange. I just tested loading the files again in a separate environment (the AWS Deep Learning AMI) and it also works there. Can you reproduce your error in a different environment?

gabriel175 commented 5 years ago

I got the same exact error with python 3.6 and 3.7 :\ Any ideas how can be solved?

Vaishaal commented 5 years ago

What version of numpy are you running?

gabriel175 commented 5 years ago

The numpy version is 1.14.2 Which version did you used to save the set?

ludwigschmidt commented 5 years ago

xxd cifar10.1_v4_labels.npy

gives

00000000: 934e 554d 5059 0100 4600 7b27 6465 7363 .NUMPY..F.{'desc

as the first line. So according to https://docs.scipy.org/doc/numpy/reference/generated/numpy.lib.format.html#module-numpy.lib.format

the file format is version 1.0. I'm not sure if we can determine at this point what version of numpy we used.

Did you try if you can load the file on a different machine / numpy installation?

rolloff commented 4 years ago

I updated the file format version to 2.0. Hopefully, this solves the issue.

alexlioralexli commented 3 years ago

I think the problem arises if you wget the wrong link, which downloads the github page instead of the npy file itself.