Closed rafaelblevin821 closed 2 years ago
Hey @rafaelblevin821
So I went back locally and tried to run it again on my end. This was the command that worked for me locally:
python scripts/test.py --weights models/test --classes 3 --folds 1
Explanation:
weights
: I think the defaults may actually put your weights here from the training scriptclasses
: The defaults in the training script actually use three classes 'covid', 'pneumonia', 'regular'
but they might be experimenting with four classes to allow a rejection of the image 'covid', 'pneumonia', 'regular', 'uninformative'
folds
: I can't remember if I chose this or ran it by default but I actually had to adjust the code to get this to workBefore running this try ls models/test
to make sure that you have models there. I think it assumes 5 folds. Do you remember what you trained with?
Hi @nickdnickd,
RE: python scripts/test.py --weights models/test --classes 3 --folds 1
So I ran the code and seem to have got results, but also some errors.
Any help is greatly appreciated.
Results:
------------- SPLIT 0 -------------------
2022-02-05 18:07:00.159015: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/david/venv/lib/python3.8/site-packages/cv2/../../lib64:
2022-02-05 18:07:00.160538: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-02-05 18:07:00.163429: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (JXVL203): /proc/driver/nvidia/version does not exist
2022-02-05 18:07:00.164869: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Model restored. Class mappings are ['covid', 'pneumonia', 'regular']
testing on n_files: 625
precision recall f1-score support
covid 0.60 0.39 0.48 147
pneumonia 0.43 1.00 0.60 206
regular 1.00 0.19 0.32 272
accuracy 0.51 625
macro avg 0.68 0.53 0.47 625
weighted avg 0.72 0.51 0.45 625
Traceback (most recent call last):
File "scripts/test.py", line 240, in <module>
main()
File "scripts/test.py", line 227, in main
evaluate_logits(
File "scripts/test.py", line 72, in evaluate_logits
gt_s = saved_gt[s]
IndexError: list index out of range
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_1
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_2
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.decay
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.learning_rate
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.0.total
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.0.count
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.1.total
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.1.count
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-12.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-12.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-13.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-13.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-14.gamma
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-14.beta
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-15.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-15.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-12.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-12.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-13.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-13.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-14.gamma
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-14.beta
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-15.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-15.bias
yep! This is expected. It seems like with folds not equal to 5 you will run into this error. I have a 1 line change that enabled this that I can submit as a PR. It uses a heuristic only loop over the length of the logits instead of 5 by default.
yep! This is expected. It seems like with folds not equal to 5 you will run into this error. I have a 1 line change that enabled this that I can submit as a PR. It uses a heuristic only loop over the length of the logits instead of 5 by default.
Great, thank you. Please let me know when you have submitted the PR. :)
Submitted the below PR. Even if it's not totally acceptable, you can get the idea for how to push through with only one fold:
https://github.com/jannisborn/covid19_ultrasound/pull/104/files
Submitted the below PR. Even if it's not totally acceptable, you can get the idea for how to push through with only one fold:
https://github.com/jannisborn/covid19_ultrasound/pull/104/files
Thank @nickdnickd. I updated the test.py with your fix and seem to have gotten more results, but still some errors, which may be unrelated to your code and possibly related to plugins on my laptop.
------------- SPLIT 0 -------------------
2022-02-05 18:31:12.619038: W tensorflow/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/david/venv/lib/python3.8/site-packages/cv2/../../lib64:
2022-02-05 18:31:12.619849: W tensorflow/stream_executor/cuda/cuda_driver.cc:269] failed call to cuInit: UNKNOWN ERROR (303)
2022-02-05 18:31:12.621459: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:156] kernel driver does not appear to be running on this host (JXVL203): /proc/driver/nvidia/version does not exist
2022-02-05 18:31:12.624992: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
Model restored. Class mappings are ['covid', 'pneumonia', 'regular']
testing on n_files: 625
precision recall f1-score support
covid 0.60 0.39 0.48 147
pneumonia 0.43 1.00 0.60 206
regular 1.00 0.19 0.32 272
accuracy 0.51 625
macro avg 0.68 0.53 0.47 625
weighted avg 0.72 0.51 0.45 625
Average scores in cross validation:
Precision Recall F1-score MCC Specificity Accuracy Balanced
covid 0.604 0.395 0.477 0.371 0.921 0.506 0.529
pneumonia 0.432 1.000 0.603 0.391 0.353 0.506 0.529
regular 1.000 0.191 0.321 0.343 1.000 0.506 0.529
Standard deviations:
Precision Recall F1-score MCC Specificity Accuracy Balanced
covid 0.0 0.0 0.0 0.0 0.0 0.0 0.0
pneumonia 0.0 0.0 0.0 0.0 0.0 0.0 0.0
regular 0.0 0.0 0.0 0.0 0.0 0.0 0.0
WARNING:tensorflow:Detecting that an object or model or tf.train.Checkpoint is being deleted with unrestored values. See the following logs for the specific values in question. To silence these warnings, use `status.expect_partial()`. See https://www.tensorflow.org/api_docs/python/tf/train/Checkpoint#restorefor details about the status object returned by the restore function.
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.iter
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_1
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.beta_2
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.decay
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer.learning_rate
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.0.total
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.0.count
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.1.total
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).keras_api.metrics.1.count
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-12.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-12.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-13.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-13.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-14.gamma
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-14.beta
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-15.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'm' for (root).layer_with_weights-15.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-12.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-12.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-13.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-13.bias
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-14.gamma
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-14.beta
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-15.kernel
WARNING:tensorflow:Value in checkpoint could not be found in the restored object: (root).optimizer's state 'v' for (root).layer_with_weights-15.bias
@rafaelblevin821 Any time! I see these too and am also unsure.
Thanks @nickdnickd for proposing this fix!
@rafaelblevin821 the remaining entries in the log trace are warnings not errors. The stds are all zeros in the final evaluation since you only have one fold.
Hi, I'm running into an error when trying to test the model.
test.py: error: unrecognized arguments: [-h] [--data DATA] [--weights WEIGHTS] [--m_id M_ID] [--classes CLASSES] [--folds FOLDS] [--save_path SAVE_PATH]
I tried python scripts/test.py but got more errors as per @nickdnickd suggestion and got the following error:
I wonder is it necessary to have a dedicated GPU for this?
Any ideas on how I can rectify this? Thanks.