search

jeffheaton / t81_558_deep_learning

T81-558: Keras - Applications of Deep Neural Networks @Washington University in St. Louis
https://sites.wustl.edu/jeffheaton/t81-558/
Other
5.71k stars 3.04k forks source link

"pickle data was truncated" error in models trained using GCE #136

Closed baristerzi closed 2 years ago

baristerzi commented 2 years ago

Hello Mr Heaton, I am trying to train the model using "t81_558_class_07_2_train_gan.ipynb". While there is no problem with the model when I connect to Google Colab internal servers, when I work with a VM I rented via Google Cloud, half of my data turns black as in the image. If I train with this dataset, I cannot use the pickle to generate images later. Model gives the error "pickle data was truncated".

I also add GPU features to the image. I had this problem with v100 and t4 GPUs that I rented through Google Cloud. Unfortunately, I cannot use Colab Pro due to location, and Colab Free is not enough to train my model. I would appreciate it if you could give your opinion on the problem. Have a nice day. error

jeffheaton commented 2 years ago

I am not sure on this one. I do not even really know how I would recreate it. The problem is likely in the way the Pickle file is being generated inside of StyleGAN, so it is probably a better question for them.

baristerzi commented 2 years ago

Thanks for the comment. I better ask this question to the Stylegan team. If I can find a solution, I'll post it here as well.

jeffheaton commented 2 years ago

Okay, thanks, sounds good.