search

LongxiZhou / DLPE-method

A comprehensive platform for analyzing pulmonary parenchyma lesions on chest CT.
BSD 3-Clause "New" or "Revised" License
15 stars 9 forks source link

AssertionError when running dcm_to_enhanced.py #3

Closed yufan122 closed 1 year ago

yufan122 commented 1 year ago

I was following instructions at https://github.com/LongxiZhou/DLPE-method/blob/master/colab.ipynb and trying to repeat the pipeline in Linux HPC cluster. I was not able to run environment.yml successfully, it always returns: "The environment is inconsistent, please check the package plan carefully The following packages are causing the inconsistency:

Traceback (most recent call last): File "/vf/users/DLPE-method-master/dcm_to_enhanced.py", line 27, in lung_mask = predictor.predict_lung_masks_rescaled_array(rescaled_ct_array, refine=False, batch_size=batch_size) File "/vf/users/DLPE-method-master/prediction/predict_rescaled.py", line 103, in predict_lung_masks_rescaled_array lung_mask = get_prediction_rescaled_array_binary(rescaled_array, check_point_directory, threshold=2., File "/vf/users/DLPE-method-master/prediction/predict_rescaled.py", line 47, in get_prediction_rescaled_array_binary assert len(os.listdir(check_point_dict)) == 3 AssertionError

LongxiZhou commented 1 year ago

The assertion checks whether you have placed the checkpoint files correctly. As the assertion is False, this means there are missing documents or other unrelated documents in the target directory. Best Longxi

On Sat, Jul 8, 2023 at 12:16 AM yufan122 @.***> wrote:

I was following instructions at https://github.com/LongxiZhou/DLPE-method/blob/master/colab.ipynb https://urldefense.com/v3/__https://github.com/LongxiZhou/DLPE-method/blob/master/colab.ipynb__;!!Nmw4Hv0!z4Hl-wD4ojr30bzqOncqJiCTIGZYj1sod0o_vM9rRb_rHFwywK7rR1hkQKP4YEuWGHHoblhEvQFeDjqoPgKe-CZdIMQQ_sM$ and trying to repeat the pipeline in Linux HPC cluster. I was not able to run environment.yml successfully, it always returns: "The environment is inconsistent, please check the package plan carefully The following packages are causing the inconsistency:

  • conda-forge/linux-64::cryptography==40.0.2=py310h34c0648_0
  • conda-forge/linux-64::conda==23.3.1=py310hff52083_0 ......." However, without running the .yml file, I manually installed all the missing softwares one by one following error messages each time after running python dcm_to_enhanced.py. Finally I was able to run it, but ended up with the following outputs and errors: (base) @.*** DLPE-method-master]$ python dcm_to_enhanced.py given wc_ww wc: -600 ww: 1600 number_dcm_files: 365 the resolution for x, y, z in mm: [0.75, 0.75, 1] the array corresponds to a volume of: 384.0 384.0 365.0 array has shape: (512, 512, 365) min-average-max: -1024.0 -678.2724 3071.0 std: 482.63736 stack complete! the original resolution is [0.75, 0.75, 1] tissue type: lung predicting lung masks

Traceback (most recent call last): File "/vf/users/DLPE-method-master/dcm_to_enhanced.py", line 27, in lung_mask = predictor.predict_lung_masks_rescaled_array(rescaled_ct_array, refine=False, batch_size=batch_size) File "/vf/users/DLPE-method-master/prediction/predict_rescaled.py", line 103, in predict_lung_masks_rescaled_array lung_mask = get_prediction_rescaled_array_binary(rescaled_array, check_point_directory, threshold=2., File "/vf/users/DLPE-method-master/prediction/predict_rescaled.py", line 47, in get_prediction_rescaled_array_binary assert len(os.listdir(check_point_dict)) == 3 AssertionError

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/LongxiZhou/DLPE-method/issues/3__;!!Nmw4Hv0!z4Hl-wD4ojr30bzqOncqJiCTIGZYj1sod0o_vM9rRb_rHFwywK7rR1hkQKP4YEuWGHHoblhEvQFeDjqoPgKe-CZdy2iMrtY$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AL6XWCRFDM5HWV6YEPDODTLXPB4BPANCNFSM6AAAAAA2CJJ46U__;!!Nmw4Hv0!z4Hl-wD4ojr30bzqOncqJiCTIGZYj1sod0o_vM9rRb_rHFwywK7rR1hkQKP4YEuWGHHoblhEvQFeDjqoPgKe-CZd8AuEyJs$ . You are receiving this because you are subscribed to this thread.Message ID: @.***>

--

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

yufan122 commented 1 year ago

Thank you so much for the prompt reply! Really appreciate it! May I ask where should I modify the path of checkpoint files? I'm pretty sure I've changed global parameters "trained_model_top_dict", "dcm_directory" and "enhance_array_output_directory" in dcm_to_enhanced.py correctly. Are there anywhere else I need to make a change? Thank you! Yu

LongxiZhou commented 1 year ago

The error is assertion error: assert len(os.listdir(check_point_dict)) == 3 is False, as I have said, the assertion checks whether you have placed the checkpoint files correctly. You are pretty sure you changed the global parameters, but this is totally irrelevant with the assertion error. The assertion checks files, which means there are missing files or other unrelated files in the check_point_dict.

It is impossible for me to answer all trivial questions. We have provided examples using Jupyter Notebook that you can run with Google Colab: https://github.com/LongxiZhou/DLPE-method/blob/master/colab.ipynb

Before continuing to ask me about errors happening in your computer, first run the example with Google Colab.

On Tue, Jul 11, 2023 at 12:20 AM yufan122 @.***> wrote:

Thank you so much for the prompt reply! Really appreciate it! May I ask where should I modify the path of checkpoint files? I'm pretty sure I've changed global parameters "trained_model_top_dict", "dcm_directory" and "enhance_array_output_directory" in dcm_to_enhanced.py correctly. Are there anywhere else I need to make a change? Thank you! Yu

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/LongxiZhou/DLPE-method/issues/3*issuecomment-1629747454__;Iw!!Nmw4Hv0!1Jc4PcjAclqAXMlDcL2Wvmo8ToK3obp5Ba0LMBTOh03TYzuJ5rql1r5WHPbfVecvY4IkytBZu1ov23F0Q2ZzvmpjacIjfjc$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AL6XWCUDXBY4P6ZOWWTXRUTXPRWZTANCNFSM6AAAAAA2CJJ46U__;!!Nmw4Hv0!1Jc4PcjAclqAXMlDcL2Wvmo8ToK3obp5Ba0LMBTOh03TYzuJ5rql1r5WHPbfVecvY4IkytBZu1ov23F0Q2ZzvmpjaoHO5u4$ . You are receiving this because you commented.Message ID: @.***>

--

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

LongxiZhou commented 1 year ago

My code has very detailed explainables. This is the code where you meet the error. [image: 5201689101516_.pic.jpg]

Please read my code, download checkpoint files, and place the checkpoint files in relevant directory. :param check_point_dict: where the model_guided saved, should in check_point_dict/direction/model_name.pth

On Tue, Jul 11, 2023 at 9:40 PM Longxi Zhou @.***> wrote:

The error is assertion error: assert len(os.listdir(check_point_dict)) == 3 is False, as I have said, the assertion checks whether you have placed the checkpoint files correctly. You are pretty sure you changed the global parameters, but this is totally irrelevant with the assertion error. The assertion checks files, which means there are missing files or other unrelated files in the check_point_dict.

It is impossible for me to answer all trivial questions. We have provided examples using Jupyter Notebook that you can run with Google Colab: https://github.com/LongxiZhou/DLPE-method/blob/master/colab.ipynb

Before continuing to ask me about errors happening in your computer, first run the example with Google Colab.

On Tue, Jul 11, 2023 at 12:20 AM yufan122 @.***> wrote:

Thank you so much for the prompt reply! Really appreciate it! May I ask where should I modify the path of checkpoint files? I'm pretty sure I've changed global parameters "trained_model_top_dict", "dcm_directory" and "enhance_array_output_directory" in dcm_to_enhanced.py correctly. Are there anywhere else I need to make a change? Thank you! Yu

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/LongxiZhou/DLPE-method/issues/3*issuecomment-1629747454__;Iw!!Nmw4Hv0!1Jc4PcjAclqAXMlDcL2Wvmo8ToK3obp5Ba0LMBTOh03TYzuJ5rql1r5WHPbfVecvY4IkytBZu1ov23F0Q2ZzvmpjacIjfjc$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AL6XWCUDXBY4P6ZOWWTXRUTXPRWZTANCNFSM6AAAAAA2CJJ46U__;!!Nmw4Hv0!1Jc4PcjAclqAXMlDcL2Wvmo8ToK3obp5Ba0LMBTOh03TYzuJ5rql1r5WHPbfVecvY4IkytBZu1ov23F0Q2ZzvmpjaoHO5u4$ . You are receiving this because you commented.Message ID: @.***>

--

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

yufan122 commented 1 year ago

Sorry for the confusion, and the attached image doesn't show up on my side. The check_point_dict is referring to subfolders of the downloaded trained_models, is that correct? Following the Traceback info, I checked predict_rescaled.py, the top_directory_check_point = '/trained_models/' which contains all models with X, Y, Z subfolders as I downloaded, when I checked check_point_dict, it was not defined, so where and how should I define the check_point_dict parameter?

*I tried running the Colab notebook directly, but the environment building step was unsuccessful, and I was planning to run DLPE on some other dataset in our HPC, that's why I tested the pipeline directly there. Thank you for your patience!

yufan122 commented 1 year ago

Figured it out. The '/trained_models/' folder downloaded from Google Drive contains series of hidden folders '.DS_Store', need to remove them before running python dcm_to_enhanced.py. [...trained_models]$ find . -name ".DS_Store" -print -delete

LongxiZhou commented 1 year ago

Awesome, thanks for the update!

yufan122 @.***>于2023年7月20日 周四下午6:48写道：

Closed #3 https://urldefense.com/v3/__https://github.com/LongxiZhou/DLPE-method/issues/3__;!!Nmw4Hv0!yejyogoGdAlvK6oNM2iqzAPEteHRCMXyNU0T38DPAqzRFf5gaV5ITD6_5zaoz3uhogTAz4Lo6yvwQt9IfGUfgCoQ0-MY0xM$ as completed.

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/LongxiZhou/DLPE-method/issues/3*event-9876287209__;Iw!!Nmw4Hv0!yejyogoGdAlvK6oNM2iqzAPEteHRCMXyNU0T38DPAqzRFf5gaV5ITD6_5zaoz3uhogTAz4Lo6yvwQt9IfGUfgCoQj2MgFn4$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AL6XWCV4KDF6SPRTKMUFTGDXRFHMPANCNFSM6AAAAAA2CJJ46U__;!!Nmw4Hv0!yejyogoGdAlvK6oNM2iqzAPEteHRCMXyNU0T38DPAqzRFf5gaV5ITD6_5zaoz3uhogTAz4Lo6yvwQt9IfGUfgCoQkzoM31U$ . You are receiving this because you commented.Message ID: @.***>

--

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.

LongxiZhou commented 1 year ago

This file is established by MacOS, not Google drive.

yufan122 @.***>于2023年7月20日 周四下午6:47写道：

Figured it out. The '/trained_models/' folder downloaded from Google Drive contains series of hidden folders '.DS_Store', need to remove them before running python dcm_to_enhanced.py. [...trained_models]$ find . -name ".DS_Store" -print -delete

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/LongxiZhou/DLPE-method/issues/3*issuecomment-1644167631__;Iw!!Nmw4Hv0!wm47TFIpZClrl4LFPTWJZpk2s5Tt2K71hu6KsqPYzhZIqlf9N8YXoPo32pUTTzFIsZQCR5RtgF9FCylRvMyAJb5XYcltcwo$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AL6XWCQ3DYWDTEFNC32RTFTXRFHKZANCNFSM6AAAAAA2CJJ46U__;!!Nmw4Hv0!wm47TFIpZClrl4LFPTWJZpk2s5Tt2K71hu6KsqPYzhZIqlf9N8YXoPo32pUTTzFIsZQCR5RtgF9FCylRvMyAJb5XtWL0o5M$ . You are receiving this because you commented.Message ID: @.***>

--

This message and its contents, including attachments are intended solely for the original recipient. If you are not the intended recipient or have received this message in error, please notify me immediately and delete this message from your computer system. Any unauthorized use or distribution is prohibited. Please consider the environment before printing this email.