search

MStarmans91 / WORC

Workflow for Optimal Radiomics Classification
Other
69 stars 19 forks source link

[BUG] missing Key: extraction_mode in WORC config file #68

Closed Douwe-Spaanderman closed 2 years ago

Douwe-Spaanderman commented 2 years ago

Describe the bug WORC seems incompatible with PREDICTFastr 3.1.14+ release, due to missing key: extraction_mode introduced in PREDICTFastr 3.1.14.

WORC configuration SimpleWORC config file, here shown only the ImageFeatures:

[ImageFeatures]
shape = True
histogram = True
orientation = True
texture_Gabor = True
texture_LBP = True
texture_GLCM = True
texture_GLCMMS = True
texture_GLRLM = False
texture_GLSZM = False
texture_NGTDM = False
coliage = False
vessel = True
log = True
phase = True
image_type = CT
gabor_frequencies = 0.05, 0.2, 0.5
gabor_angles = 0, 45, 90, 135
GLCM_angles = 0, 0.79, 1.57, 2.36
GLCM_levels = 16
GLCM_distances = 1, 3
LBP_radius = 3, 8, 15
LBP_npoints = 12, 24, 36
phase_minwavelength = 3
phase_nscale = 5
log_sigma = 1, 5, 10
vessel_scale_range = 1, 10
vessel_scale_step = 2
vessel_radius = 5
dicom_feature_tags = 0010 1010, 0010 0040
dicom_feature_labels = age, sex

fastr trace

[WARNING]  __init__:0084 >> Not running in a production installation (branch "develop" from source code)
Tracing errors for sample Lipo-100 from sink features_train_CT_0_predict
Located result pickle: /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/featureconverter_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/__fastr_result__.yaml

===== JOB WORC_LIPO_classification_minus___featureconverter_train_predict_CalcFeatures_1_0_CT_0___Lipo-100 =====
Network: WORC_LIPO_classification_minus
Run: WORC_LIPO_classification_minus_2022-04-01T14-49-29
Node: featureconverter_train_predict_CalcFeatures_1_0_CT_0
Sample index: (0)
Sample id: Lipo-100
Status: JobState.cancelled
Timestamp: 2022-04-01 12:57:00.622826
Job file: /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/featureconverter_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/__fastr_result__.yaml

----- ERRORS -----
- FastrError: job cancelled by fastr DRMAA plugin (/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/fastr/resources/plugins/executionplugins/drmaaexecution.py:300)
------------------

No process information:
Cannot find process information in Job information, processing probably got killed.
If there are no other errors, this is often a result of too high memory use or
exceeding some other type of resource limit.

Output data:
{}

Status history:
2022-04-01 12:57:00.622843: JobState.created
2022-04-01 13:08:31.203427: JobState.queued
2022-04-01 13:11:35.373959: JobState.cancelled
Located result pickle: /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/features_train_CT_0_predict/Lipo-100_0/__fastr_result__.yaml

===== JOB WORC_LIPO_classification_minus___features_train_CT_0_predict___Lipo-100___0 =====
Network: WORC_LIPO_classification_minus
Run: WORC_LIPO_classification_minus_2022-04-01T14-49-29
Node: features_train_CT_0_predict
Sample index: (0)
Sample id: Lipo-100
Status: JobState.cancelled
Timestamp: 2022-04-01 12:57:05.028554
Job file: /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/features_train_CT_0_predict/Lipo-100_0/__fastr_result__.yaml

----- ERRORS -----
- FastrError: job cancelled by fastr DRMAA plugin (/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/fastr/resources/plugins/executionplugins/drmaaexecution.py:300)
------------------

No process information:
Cannot find process information in Job information, processing probably got killed.
If there are no other errors, this is often a result of too high memory use or
exceeding some other type of resource limit.

Output data:
{}

Status history:
2022-04-01 12:57:05.028571: JobState.created
2022-04-01 13:08:36.418976: JobState.queued
2022-04-01 13:11:35.436503: JobState.cancelled
Located result pickle: /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/calcfeatures_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/__fastr_result__.yaml

===== JOB WORC_LIPO_classification_minus___calcfeatures_train_predict_CalcFeatures_1_0_CT_0___Lipo-100 =====
Network: WORC_LIPO_classification_minus
Run: WORC_LIPO_classification_minus_2022-04-01T14-49-29
Node: calcfeatures_train_predict_CalcFeatures_1_0_CT_0
Sample index: (0)
Sample id: Lipo-100
Status: JobState.execution_failed
Timestamp: 2022-04-01 12:51:50.614785
Job file: /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/calcfeatures_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/__fastr_result__.yaml

----- ERRORS -----
- FastrOutputValidationError: Output value [HDF5] "vfs://tmp/WORC_LIPO_classification_minus/calcfeatures_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/features_0.hdf5" not valid for datatype "'HDF5'" (/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/fastr/execution/job.py:1151)
- FastrOutputValidationError: The output "features" is invalid! (/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/fastr/execution/job.py:1099)
- FastrErrorInSubprocess: Traceback (most recent call last):
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/WORC/resources/fastr_tools/predict/bin/CalcFeatures_tool.py", line 72, in <module>
    main()
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/WORC/resources/fastr_tools/predict/bin/CalcFeatures_tool.py", line 68, in main
    semantics_file=args.sem)
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/PREDICT/CalcFeatures.py", line 74, in CalcFeatures
    config = config_io.load_config(parameters)
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/PREDICT/IOparser/config_io_CalcFeatures.py", line 94, in load_config
    settings['ImageFeatures']['extraction_mode']
  File "/cm/shared/apps/python3/3.6.8/lib/python3.6/configparser.py", line 1233, in __getitem__
    raise KeyError(key)
KeyError: 'extraction_mode'
 (/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/fastr/execution/executionscript.py:111)
- FastrValueError: Output values are not valid! (/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/fastr/execution/job.py:830)
------------------

Command:
List representation: ['python', '/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/WORC/resources/fastr_tools/predict/bin/CalcFeatures_tool.py', '--im', '/scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/preprocessing_train_CT_0/Lipo-100/image_0.nii.gz', '--out', '/scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/calcfeatures_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/features_0.hdf5', '--seg', '/scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/segmentix_train_CT_0/Lipo-100/segmentation_out_0.nii.gz', '--para', '/scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/config_CT_0/id_0/result/config_WORC_LIPO_classification_minus_0.ini']

String representation: python /home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/WORC/resources/fastr_tools/predict/bin/CalcFeatures_tool.py --im /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/preprocessing_train_CT_0/Lipo-100/image_0.nii.gz --out /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/calcfeatures_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/features_0.hdf5 --seg /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/segmentix_train_CT_0/Lipo-100/segmentation_out_0.nii.gz --para /scratch/dspaanderman/tmp/WORC_LIPO_classification_minus/config_CT_0/id_0/result/config_WORC_LIPO_classification_minus_0.ini

Output data:
{'features': [<HDF5: 'vfs://tmp/WORC_LIPO_classification_minus/calcfeatures_train_predict_CalcFeatures_1_0_CT_0/Lipo-100/features_0.hdf5'>]}

Status history:
2022-04-01 12:51:50.614800: JobState.created
2022-04-01 13:11:07.530470: JobState.running
2022-04-01 13:11:28.013769: JobState.execution_failed

----- STDOUT -----

------------------

----- STDERR -----
Traceback (most recent call last):
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/WORC/resources/fastr_tools/predict/bin/CalcFeatures_tool.py", line 72, in <module>
    main()
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/WORC/resources/fastr_tools/predict/bin/CalcFeatures_tool.py", line 68, in main
    semantics_file=args.sem)
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/PREDICT/CalcFeatures.py", line 74, in CalcFeatures
    config = config_io.load_config(parameters)
  File "/home/dspaanderman/WORCTutorial/env/lib/python3.6/site-packages/PREDICT/IOparser/config_io_CalcFeatures.py", line 94, in load_config
    settings['ImageFeatures']['extraction_mode']
  File "/cm/shared/apps/python3/3.6.8/lib/python3.6/configparser.py", line 1233, in __getitem__
    raise KeyError(key)
KeyError: 'extraction_mode'

------------------

To Reproduce Fresh Install with the newest version of WORC and PREDICTFastr.

Expected behavior WORC config file should include extraction_mode in order to determine whether to use 2D, 2.5D or 3D feature extraction.

Desktop:

MStarmans91 commented 2 years ago

Thanks for pointing this out, this should be automatically solved by the new WORC 3.6.0 that was just released, which updates the PREDICT dependency to the most recent release.