mgermain
commented
9 years ago @andyyuan78 thanks for the report.
Looks like a file in the experiment you are trying to resume may be corrupted.
Have you tried to remove the ~/github/MADE/experiments/mnist_from_paper folder and try again? You will lose the progress you've made on the training, but it should fix the problem. If you have any more information please let me know.
ubgpu@ubgpu:~/github/MADE$ sudo python -u trainMADE.py --name mnist_from_paper binarized_mnist 0.01 0 -1 32 Full 300 100 30 False 0 adagrad 0 [8000,8000] 1234 False Output False hinge Orthogonal 0 Using gpu device 0: GeForce GTX 970
Resuming experiment (mnist_from_paper).
Loading dataset [binarized_mnist] ... (Dim:784 Train:50000 Valid:10000 Test:10000) DONE in 0.3176 seconds.
Initializing MADE ... DONE in 947.4197 seconds.
Traceback (most recent call last): File "trainMADE.py", line 293, in
load_model_params(model, save_path_experiment)
File "trainMADE.py", line 230, in load_model_params
model.update_rule.parameters[i].set_value(param.get_value())
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/var.py", line 132, in set_value
self.container.value = value # this will copy a numpy ndarray
File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 343, in set
**kwargs)
File "/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/type.py", line 94, in filter_inplace
strict, old_data)
ValueError: ('object too deep for desired array', 'Container name "mean_square_grad_b0"')
ubgpu@ubgpu:~/github/MADE$