search

Emory-HITI / Niffler

Niffler: A DICOM Framework for Machine Learning and Processing Pipelines.
https://emory-hiti.github.io/Niffler/
BSD 3-Clause "New" or "Revised" License
90 stars 53 forks source link

MemoryError while extracting metadata using PNG Extraction Module #267

Closed anbhimi closed 2 years ago

anbhimi commented 2 years ago

While extracting metadata using the PNG Extraction Module (CommonHeadersOnly - true), all the DICOM tags are collected stored in a CSV format using pandas.

The memory error occurs because - in some instances, the DICOM Images could have a large no. of tags than expected which creates a sparse CSV file resulting in a large data frame that cannot be saved using pandas.

The error can be fixed by transferring such DICOM files (which have a large no. of tags) into a new folder in failed-dicom folder in the same module.

anbhimi commented 2 years ago

Fixed the issue in PR - 271. But did not transfer the failing DICOMs to a new folder. I will submit a new PR to perform the operation.

Nitesh639 commented 2 years ago

Hy @anbhimi & @pradeeban, Please give me example of the DICOM Images who have a large no. of tags than expected.

pradeeban commented 2 years ago

@Nitesh639 Thanks for your interest.

This issue is mostly fixed by @anbhimi in PR-271 and he just needs to copy the failed DICOMs to a new folder. I believe he is on it and about to complete it. Best to focus on other issues.

For other bugs, you can use https://www.cancerimagingarchive.net/collections/ for your sample DICOM data.

Nitesh639 commented 2 years ago

@pradeeban ok

pradeeban commented 2 years ago

Fixed by @anbhimi

pradeeban commented 2 years ago

293