search

spoonsso / dannce

MIT License
214 stars 30 forks source link

ValueError: no field of name val_loss #67

Closed Spartan859 closed 3 years ago

Spartan859 commented 3 years ago

Epoch 600/600 20/20 [==============================] - ETA: 0s - loss: 179.8598 - euclidean_distance_3D: 11.5716WARNING:tensorflow:Can save best model only with val_loss available, skipping. Saving predictions on train and validation data, after epoch 599 20/20 [==============================] - 20s 998ms/step - loss: 179.8598 - euclidean_distance_3D: 11.5716 Renaming weights file with best epoch description Traceback (most recent call last): File "/home/xuchun/.conda/envs/dannce/bin/dannce-train", line 33, in sys.exit(load_entry_point('dannce', 'console_scripts', 'dannce-train')()) File "/home/xuchun/dannce/dannce/cli.py", line 66, in dannce_train_cli dannce_train(params) File "/home/xuchun/dannce/dannce/interface.py", line 1249, in dannce_train processing.rename_weights(dannce_train_dir, kkey, mon) File "/home/xuchun/dannce/dannce/engine/processing.py", line 485, in rename_weights q = r[mon] ValueError: no field of name val_loss

When I finished training 3cam AVG net of dannce, this happened. How can I solve it?

Spartan859 commented 3 years ago

Sorry for troubling. The problem is caused by CUPTI module of CUDA. I fixed it by copying the required .so file to cuda11.1/lib64.