Closed drcandacemakedamoore closed 5 months ago
Thanks for preparing a continuous integration script. My main concern is that running the full test suite currently requires at least 15 minutes on a high-end system (excluding setting up the environment). GitHub will likely take longer.
Currently, the test suite is run as part of version releases, see #64 for example. However, I can see the benefit of additionally running CI on (a subset of) the full test suite. This will also help identify issues when dependencies are making changes to their API.
I have opened a new branch for the PR.
List of test durations:
183.87s call test/documentation_example_test.py::test_extract_features_examples
133.61s call test/perturbation_test.py::test_translation_perturbation_multiple
110.72s call test/dicom_mr_test.py::test_bias_field_correction_t1
88.12s call test/ibsi_2_feature_test.py::test_ibsi_2_config_gabor_filter
58.53s call test/readme_example_test.py::test_example_feature_extraction
35.92s call test/perturbation_test.py::test_rotation_perturbation_multiple
34.10s call test/ibsi_1_test.py::test_ibsi_1_chest_config_a
31.68s call test/perturbation_test.py::test_perturbation_roi_randomisation
31.49s call test/parallel_feature_extraction_test.py::test_parallel_feature_extraction
29.80s call test/perturbation_test.py::test_pertubation_distance_change_multiple
28.02s call test/perturbation_test.py::test_pertubation_fraction_change_multiple
23.70s call test/perturbation_test.py::test_perturbation_roi_randomisation_rotation
23.58s call test/ibsi_1_test.py::test_ibsi_1_chest_config_b
22.24s call test/perturbation_test.py::test_noise_perturbation
18.44s call test/basic_test.py::test_xml_configurations
18.23s call test/ibsi_2_feature_test.py::test_ibsi_2_config_simoncelli_filter
17.00s call test/ibsi_2_feature_test.py::test_ibsi_2_config_daubechies_filter
16.63s call test/basic_test.py::test_edge_cases_basic_pipeline
16.15s call test/ibsi_1_test.py::test_ibsi_1_chest_config_c
16.12s call test/dicom_mr_test.py::test_bias_field_correction_t2
15.91s call test/ibsi_1_test.py::test_ibsi_1_chest_config_e
14.17s call test/ibsi_1_test.py::test_ibsi_1_chest_config_d
13.49s call test/perturbation_test.py::test_perturbation_distance_grow
12.54s call test/perturbation_test.py::test_perturbation_distance_shrink
12.50s call test/perturbation_test.py::test_perturbation_fraction_shrink
12.49s call test/perturbation_test.py::test_translation_perturbation
12.26s call test/perturbation_test.py::test_rotation_perturbation
11.89s call test/perturbation_test.py::test_perturbation_fraction_growth
11.56s call test/dl_preprocessing_test.py::test_extract_image_crop
9.49s call test/read_image_and_mask_test.py::test_read_dicom_image_and_mask_stack
8.07s call test/intensity_scaling_test.py::test_intensity_scaling
7.91s call test/read_image_and_mask_test.py::test_read_dicom_image_and_mask_modality_specific
7.19s call test/ibsi_2_feature_test.py::test_ibsi_2_config_laws_filter
7.18s call test/parallel_feature_extraction_test.py::test_parallel_dl_preprocessing
6.70s call test/ibsi_2_feature_test.py::test_ibsi_2_config_laplacian_of_gaussian_filter
6.67s call test/dicom_rtdose_test.py::test_basic_rtdose_feature_extraction
6.06s call test/ibsi_2_feature_test.py::test_ibsi_2_config_mean_filter
5.95s call test/ibsi_2_feature_test.py::test_ibsi_2_config_none
5.66s call test/dicom_pet_test.py::test_basic_pet_feature_extraction
5.63s call test/dl_preprocessing_test.py::test_normalisation_quantile_range
5.55s call test/dl_preprocessing_test.py::test_normalisation_range
5.51s call test/dl_preprocessing_test.py::test_normalisation_relative_range
5.24s call test/extract_image_parameters_test.py::test_extract_image_parameters_dicom
4.61s call test/import_image_and_mask_test.py::test_multiple_image_and_mask_import_flat
4.04s call test/dl_preprocessing_test.py::test_normalisation_standardisation
3.67s call test/import_image_and_mask_test.py::test_multiple_image_and_mask_import
3.06s call test/basic_test.py::test_orientation
3.02s call test/ibsi_2_response_map_test.py::test_ibsi_2_gabor_filter
2.96s call test/import_image_test.py::test_multiple_image_import_flat
2.64s call test/dicom_mr_test.py::test_basic_mr_t1_feature_extraction
2.18s call test/dicom_mr_test.py::test_basic_mr_t2_feature_extraction
2.06s call test/import_mask_test.py::test_multiple_mask_import_flat
2.06s call test/documentation_example_test.py::test_deeplearning_preprocessing
2.05s call test/import_image_and_mask_test.py::test_single_image_and_mask_import_flat
1.97s call test/import_image_and_mask_test.py::test_failure_multiple_image_and_mask_import_data_xml
1.96s call test/import_image_and_mask_test.py::test_single_image_and_mask_import
1.94s call test/import_image_and_mask_test.py::test_failure_multiple_image_and_mask_import
1.92s call test/read_image_and_mask_test.py::test_read_dicom_image_and_mask_data_xml
1.91s call test/readme_example_test.py::test_example_deep_learning_preprocessing
1.85s call test/ibsi_1_test.py::test_ibsi_1_digital_phantom
1.76s call test/import_image_test.py::test_multiple_image_import
1.70s call test/import_image_test.py::test_single_image_import_flat
1.55s call test/non_ibsi_compliant_filter_test.py::test_logarithm_transformation_filter
1.54s call test/non_ibsi_compliant_filter_test.py::test_square_transformation_filter
1.43s call test/dicom_ct_test.py::test_basic_ct_feature_extraction
1.42s call test/non_ibsi_compliant_filter_test.py::test_exponential_transformation_filter
1.41s call test/non_ibsi_compliant_filter_test.py::test_square_root_transformation_filter
1.29s call test/import_image_test.py::test_single_image_import
1.07s call test/read_image_and_mask_test.py::test_read_numpy_image_and_mask_stack
0.88s call test/readme_example_test.py::test_example_image_metadata
0.84s call test/import_mask_test.py::test_multiple_mask_import
0.70s call test/ibsi_2_response_map_test.py::test_ibsi_2_laws_filter
0.51s call test/read_image_and_mask_test.py::test_read_itk_image_and_mask
0.34s call test/import_mask_test.py::test_single_mask_import
0.34s call test/read_image_and_mask_test.py::test_read_numpy_image_and_mask
0.30s call test/extract_mask_labels_test.py::test_extract_mask_labels_rtstruct
0.26s call test/import_mask_test.py::test_single_mask_import_flat
0.24s call test/ibsi_2_response_map_test.py::test_ibsi_2_haar_filter
0.23s call test/ibsi_2_response_map_test.py::test_ibsi_2_coifflet_filter
0.22s call test/ibsi_2_response_map_test.py::test_ibsi_2_daubechies_filter
0.22s call test/read_image_and_mask_test.py::test_read_generic_image_and_mask_modality_specific
0.21s call test/documentation_example_test.py::test_image_metadata_extraction
0.19s call test/ibsi_2_response_map_test.py::test_ibsi_2_simoncelli_filter
0.15s call test/ibsi_2_response_map_test.py::test_ibsi_2_mean_filter
0.15s call test/ibsi_2_response_map_test.py::test_ibsi_2_log_filter
0.13s call test/extract_mask_labels_test.py::test_extract_mask_labels_generic
0.12s call test/read_image_and_mask_test.py::test_read_numpy_image_and_mask_online
0.10s call test/readme_example_test.py::test_example_retrieve_mask_labels
0.08s call test/configuration_test.py::test_image_transformation_settings_configuration
0.06s call test/documentation_example_test.py::test_mask_label_extraction
0.05s call test/import_image_test.py::test_image_import_flat_poor_naming
0.03s call test/export_image_and_mask_test.py::test_basic_ct_feature_extraction
0.03s call test/extract_image_parameters_test.py::test_extract_image_parameters_default
0.01s call test/mask_processing_test.py::test_mask_processing
0.01s call test/import_mask_test.py::test_mask_import_flat_poor_naming
0.01s call test/configuration_test.py::test_general_settings_configuration
I think it's fine to not test everything, but there should be some critical group of tests that get run on auto- otherwise it's too easy, especially if someone is running away with a fork of their own, to just start slowly breaking everything without realizing it.
I took the liberty of writing a CI workflow based on a standard GitHub workflow: .github/workflows/auto-test-package.yml.
This also checks against different OS and python versions (currently limited to 3.10 and 3.11 due to limitations in the ray
package).
A Pull Request is no longer required.
Great but if I recall Python 3.11 should throw you some problems, and Python 3.12 definitely did when I tested it. Look very carefully at each package, and the requirements before adding Python 3.11.
Python 3.12 likely throws errors because its not supported by ray
yet. Other versions seem to build correctly and test correctly: https://github.com/oncoray/mirp/actions/runs/8433681661.
The auto-ci test suite also includes a test that uses parallel processing.
To do:
I suggest you will be better of with testing in your CI. I wrote an example for your package (https://github.com/drcandacemakedamoore/reviewmirp/blob/master/.github/workflows/on-commit.yml) if you like it we can make a PR from this fork; if you envision a different approach go for it; but I see no more normative and efficient way to handle testing that putting it in the CI.