search

NVIDIA / DIGITS

Deep Learning GPU Training System
https://developer.nvidia.com/digits
BSD 3-Clause "New" or "Revised" License
4.12k stars 1.38k forks source link

Nvidia Digits not saving Jobs after training the model and throwing Pickle error in log #2215

Open spurani opened 4 years ago

spurani commented 4 years ago

I have successfully created the dataset and trained the model but while saving the model it throws an Caught PicklingError while saving job 20200229-132633-c74a182b8700cc886a1b9e86: Can't pickle <class 'caffe.proto.caffe_pb2.SolverParameter'>: it's not the same object as caffe.proto.caffe_pb2.SolverParameter. I looked it up online and found couple of resource. But I am not able to decide which one should I go with.

https://blog.csdn.net/u014285910/article/details/86521518 NVIDIA#32 (comment) NVIDIA#306 NVIDIA#2024

Can anyone please help me solve this issue so that I can continue my deep learning? Thanks a bunch My system is Ubuntu 16.04 LTS 64bit. @lukeyeager

x4nny commented 4 years ago

i have the same issue

spurani commented 4 years ago

So after implementing changes mention in this commit. I retrained my model and created new dataset then I didn't get any errors or warning but after the model was successfully trained and completed all its process for saving that model. I am not able to test the model it just keeps loading the page with no errors or warnings. But will update here soon If I get it through lets see how it goes https://github.com/uhfband/DIGITS/commit/59faee35442191356e43f51df87bd9253c1f6f1f