Chin-Fu-Liu / Acute-stroke_Detection_Segmentation

GNU General Public License v3.0
19 stars 11 forks source link

Trying to unpickle estimator DecisionTreeClassifier from version 1.0.1 #4

Open agdamsbo opened 1 year ago

agdamsbo commented 1 year ago

I'm trying to get your v1.2 to run in a conda env on MacOS. I have had problems to get everything to install from the provided module_list.txt, and instead resorted to install the latest version of packages. I edited a few lines in the provided code to comply with the newest changes to python (v3.11).

Trying to run your code with basic settings in the "Configuration.xlsx", I get the following error when running the ------ Symmetric Diffeomorphic Mapping to MNI without skull for generating radiological report ------ step:

(/Users/au301842/ADSv1.2/adsenv) au301842@d45040 ~ % python ADSv1.2/codes/ADSv1Run.py -Configuration_path /Users/au301842/ADSv1.2/Configuration.xlsx
<IPython.core.display.HTML object>
Data Validation extension is not supported and will be removed
------ Loading DWI, b0 ------
------ Loading ADC ------
------ Loading JHU_SS_b0_padding template------
------ Mapping to MNI with skull for MaskNet------
Optimizing level 0 [max iter: 5]
Optimizing level 0 [max iter: 5]
Optimizing level 0 [max iter: 5]
------ Finished mapping to MNI with skull for MaskNet------
It takes 38.47 seconds
------ Loading MaskNet------
------ Inferencing brain mask------
------ Finished inferencing brain mask------
It takes 1.96 seconds
------ skull-stripping------
------ Loading JHU_SS_b0_ss_padding template------
------ Mapping to MNI without skull for lesion detection model ------
Optimizing level 0 [max iter: 15]
Optimizing level 0 [max iter: 15]
Optimizing level 0 [max iter: 15]
It takes 79.98 seconds
------ Normalizing dwi------
------ Calculating Prob. IS Map for CH3 ------
------ Loading detection model ------
------ Inferencing lesion prediction ------
It takes 38.58 seconds
------ Symmetric Diffeomorphic Mapping to MNI without skull for generating radiological report ------
It takes 139.66 seconds
------ Generating radiological report ------
------ Computing QFV ------
Trying to unpickle estimator DecisionTreeClassifier from version 1.0.1 when using version 1.3.0. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations
Traceback (most recent call last):
  File "/Users/au301842/ADSv1.2/codes/ADSv1Run.py", line 64, in <module>
    main(args)
  File "/Users/au301842/ADSv1.2/codes/ADSv1Run.py", line 54, in main
    ADSv1(Configuration_path=Configuration_path)
  File "/Users/au301842/ADSv1.2/codes/ADSv1Run.py", line 45, in ADSv1
    ADSv1_pipeline(SubjDir, 
    ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/au301842/ADSv1.2/codes/ADS_bin.py", line 403, in ADSv1_pipeline
    hydro_flag = gen_radiological_report(SubjDir, 
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/au301842/ADSv1.2/codes/ADS_RadiologicalReportGen.py", line 136, in gen_radiological_report
    AA_df_list = get_AA_report_list(AAModelsDir, QFV_list)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/au301842/ADSv1.2/codes/ADS_RadiologicalReportGen.py", line 79, in get_AA_report_list
    Vascular_predict_df = get_predict_df(QFV_list[0], AA_models_Dir, Model_list, 
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/au301842/ADSv1.2/codes/ADS_RadiologicalReport_bin.py", line 186, in get_predict_df
    clf = pickle.load(f)
          ^^^^^^^^^^^^^^
  File "sklearn/tree/_tree.pyx", line 714, in sklearn.tree._tree.Tree.__setstate__
  File "sklearn/tree/_tree.pyx", line 1418, in sklearn.tree._tree._check_node_ndarray
ValueError: node array from the pickle has an incompatible dtype:
- expected: {'names': ['left_child', 'right_child', 'feature', 'threshold', 'impurity', 'n_node_samples', 'weighted_n_node_samples', 'missing_go_to_left'], 'formats': ['<i8', '<i8', '<i8', '<f8', '<f8', '<i8', '<f8', 'u1'], 'offsets': [0, 8, 16, 24, 32, 40, 48, 56], 'itemsize': 64}
- got     : [('left_child', '<i8'), ('right_child', '<i8'), ('feature', '<i8'), ('threshold', '<f8'), ('impurity', '<f8'), ('n_node_samples', '<i8'), ('weighted_n_node_samples', '<f8')]

As far as I can gather, the problem comes down to the provided trained models being created in scikit-learn v1..0.1, which are not compatible with the latest v1.3.0. The referenced link suggests saving the trained models in a slightly different format for better cross platform compatibility . Is this something you have looked into?

I have no problems running your ADSbin.py.

Chin-Fu-Liu commented 1 year ago

Hello, Thanks for this suggeston. Since all the models were trained and saved under the old version scikit-learn. So we would suggest you create a virtual environment with the same version for all modules to avoid this kind of compartibility issue.

On the other hand, we will also consider if we can invest some time to update some modules' version and keep everything to update.

Thank you for letting us know this issue