search

Adamouization / Breast-Cancer-Detection-Mammogram-Deep-Learning

Master's dissertation for breast cancer detection in mammograms using deep learning techniques in Tensorflow. Contains the final report and source code.
https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0280841
BSD 2-Clause "Simplified" License
97 stars 26 forks source link

Unble to run the code #33

Closed raghavzns closed 1 year ago

raghavzns commented 1 year ago 
python3.7 main.py  -d mini-MIAS  -m basic -r train -lr .001 -b 2 -e1 100 -e2 50 -False -v
2023-08-19 18:45:21.582524: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
usage: main.py [-h] -d DATASET -m MODEL [-r RUNMODE] [-i IMAGESIZE] [-v]
main.py: error: unrecognized arguments: -lr .001 -b 2 -e1 100 -e2 50 -False
Adamouization commented 1 year ago

Try replacing basic with one of the models in the instructions and remove -False as the usage of a region of interest (ROI) is False by default:

python3.7 main.py -d mini-MIAS -m VGG -r train -lr 0.001 -b 2 -e1 100 -e2 50 -v

You can also check out the updated code here: https://github.com/Adamouization/Breast-Cancer-Detection-Mammogram-Deep-Learning-Publication

And the publication here: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0280841

raghavzns commented 1 year ago

Will check, but still getting this error :-


python3.7 main.py -d mini-MIAS -m VGG -r train -lr 0.001 -b 2 -e1 100 -e2 50 -v
2023-08-21 15:42:16.878596: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
usage: main.py [-h] -d DATASET -m MODEL [-r RUNMODE] [-i IMAGESIZE] [-v]
main.py: error: unrecognized arguments: -lr 0.001 -b 2 -e1 100 -e2 50
Adamouization commented 1 year ago

Any luck using the latest code @raghavzns?

raghavzns commented 1 year ago

Getting this error

python3 main.py -d CBIS-DDSM -mt all -m MobileNet -r train -lr 0.0001

2023-08-31 16:09:43.971961: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-08-31 16:09:45.045876: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-08-31 16:09:45.046453: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2023-08-31 16:09:45.073711: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:941] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2023-08-31 16:09:45.073868: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1720] Found device 0 with properties: 
pciBusID: 0000:09:00.0 name: NVIDIA GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.6575GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2023-08-31 16:09:45.073889: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2023-08-31 16:09:45.074006: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/cv2/../../lib64:
2023-08-31 16:09:45.074046: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/cv2/../../lib64:
2023-08-31 16:09:45.075419: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2023-08-31 16:09:45.075615: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2023-08-31 16:09:45.076805: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
2023-08-31 16:09:45.076867: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/cv2/../../lib64:
2023-08-31 16:09:45.076908: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/cv2/../../lib64:
2023-08-31 16:09:45.076916: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1757] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Number of GPUs Available:  0
-- Training model --

Importing CBIS-DDSM training set
2023-08-31 16:09:45.086900: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-08-31 16:09:45.087426: I tensorflow/compiler/jit/xla_gpu_device.cc:99] Not creating XLA devices, tf_xla_enable_xla_devices not set
2023-08-31 16:09:45.087452: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1261] Device interconnect StreamExecutor with strength 1 edge matrix:
2023-08-31 16:09:45.087457: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1267]      
2023-08-31 16:09:45.261180: I tensorflow_io/core/kernels/cpu_check.cc:128] Your CPU supports instructions that this TensorFlow IO binary was not compiled to use: AVX2 FMA
WARNING:tensorflow:`input_shape` is undefined or non-square, or `rows` is not in [96, 128, 160, 192, 224]. Weights for input shape (224, 224) will be loaded as the default.
2023-08-31 16:09:46.164286: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 671088640 exceeds 10% of free system memory.
2023-08-31 16:09:46.269639: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 671088640 exceeds 10% of free system memory.
2023-08-31 16:09:46.408658: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 671088640 exceeds 10% of free system memory.
2023-08-31 16:09:46.638900: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:116] None of the MLIR optimization passes are enabled (registered 2)
2023-08-31 16:09:46.659940: I tensorflow/core/platform/profile_utils/cpu_utils.cc:112] CPU Frequency: 4050125000 Hz
Epoch 1/100
2023-08-31 16:09:47.188207: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 671088640 exceeds 10% of free system memory.
2023-08-31 16:09:47.321841: W tensorflow/core/framework/cpu_allocator_impl.cc:80] Allocation of 671088640 exceeds 10% of free system memory.
2023-08-31 16:09:48.852421: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00018_RIGHT_CC/07-20-2016-DDSM-NA-18937/1.000000-full\ mammogram\ images-64601/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852421: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00218_LEFT_CC/07-20-2016-DDSM-NA-27758/1.000000-full\ mammogram\ images-85171/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852421: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00328_LEFT_MLO/07-20-2016-DDSM-NA-39090/1.000000-full\ mammogram\ images-56790/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852453: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_00260_RIGHT_MLO/08-07-2016-DDSM-NA-07680/1.000000-full\ mammogram\ images-93179/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852425: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00421_LEFT_CC/07-20-2016-DDSM-NA-30153/1.000000-full\ mammogram\ images-58874/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852442: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_00750_RIGHT_MLO/08-07-2016-DDSM-NA-61043/1.000000-full\ mammogram\ images-43503/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852452: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00119_LEFT_CC/07-20-2016-DDSM-NA-00683/1.000000-full\ mammogram\ images-43211/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852500: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_00049_RIGHT_MLO/08-07-2016-DDSM-NA-75796/1.000000-full\ mammogram\ images-09467/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852466: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00421_LEFT_MLO/07-20-2016-DDSM-NA-26126/1.000000-full\ mammogram\ images-11853/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852476: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_00047_LEFT_CC/08-07-2016-DDSM-NA-73583/1.000000-full\ mammogram\ images-60300/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852477: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_01712_RIGHT_CC/07-20-2016-DDSM-NA-45871/1.000000-full\ mammogram\ images-50889/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852501: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_00926_LEFT_MLO/08-07-2016-DDSM-NA-46329/1.000000-full\ mammogram\ images-70494/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852441: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_01184_LEFT_MLO/08-07-2016-DDSM-NA-66018/1.000000-full\ mammogram\ images-24998/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852433: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_01621_RIGHT_CC/07-20-2016-DDSM-NA-57866/1.000000-full\ mammogram\ images-65010/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852513: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00354_LEFT_CC/07-20-2016-DDSM-NA-35595/1.000000-full\ mammogram\ images-42349/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852521: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_00531_RIGHT_CC/08-07-2016-DDSM-NA-91043/1.000000-full\ mammogram\ images-95436/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852701: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_00609_RIGHT_CC/08-07-2016-DDSM-NA-62499/1.000000-full\ mammogram\ images-91897/1-1.dcm; No such file or directory
2023-08-31 16:09:48.852729: W tensorflow/core/framework/op_kernel.cc:1763] OP_REQUIRES failed at whole_file_read_ops.cc:116 : Not found: /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Calc-Training_P_01784_LEFT_CC/08-07-2016-DDSM-NA-60540/1.000000-full\ mammogram\ images-76397/1-1.dcm; No such file or directory
Traceback (most recent call last):
  File "main.py", line 264, in <module>
    main()
  File "main.py", line 116, in main
    model.train_model(train_dataset, validation_dataset, None, None, class_weights)
  File "/media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/src/cnn_models/cnn_model.py", line 85, in train_model
    self.fit_model(X_train, X_val, y_train, y_val, class_weights, is_frozen_layers=True)
  File "/media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/src/cnn_models/cnn_model.py", line 183, in fit_model
    ReduceLROnPlateau(patience=int(patience / 2))
  File "/home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1095, in fit
    tmp_logs = self.train_function(iterator)
  File "/home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 828, in __call__
    result = self._call(*args, **kwds)
  File "/home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 888, in _call
    return self._stateless_fn(*args, **kwds)
  File "/home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 2943, in __call__
    filtered_flat_args, captured_inputs=graph_function.captured_inputs)  # pylint: disable=protected-access
  File "/home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1919, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 560, in call
    ctx=ctx)
  File "/home/aitest/.conda/envs/tftest/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.NotFoundError:  /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00218_LEFT_CC/07-20-2016-DDSM-NA-27758/1.000000-full\ mammogram\ images-85171/1-1.dcm; No such file or directory
     [[{{node ReadFile}}]]
     [[IteratorGetNext]] [Op:__inference_train_function_11582]

Function call stack:
train_function

But i double checked, and the files are there at the location 


(tftest) aitest@aitest-System-Product-Name:/media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/src$ ls /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00018_RIGHT_CC/07-20-2016-DDSM-NA-18937/1.000000-full\ mammogram\ images-64601/1-1.dcm
'/media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00018_RIGHT_CC/07-20-2016-DDSM-NA-18937/1.000000-full mammogram images-64601/1-1.dcm'
(tftest) aitest@aitest-System-Product-Name:/media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/src$ ls /media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00218_LEFT_CC/07-20-2016-DDSM-NA-27758/1.000000-full\ mammogram\ images-85171/1-1.dcm
'/media/aitest/01D6F64296B8DBA0/Breast_cancer/BCDC2/data/CBIS-DDSM/manifest-ZkhPvrLo5216730872708713142/CBIS-DDSM/Mass-Training_P_00218_LEFT_CC/07-20-2016-DDSM-NA-27758/1.000000-full mammogram images-85171/1-1.dcm'