If there are large no. of mapping or metadata files - concatenating or merging them leads to a Memory Error.

Emory-HITI / Niffler

Niffler: A DICOM Framework for Machine Learning and Processing Pipelines.

https://emory-hiti.github.io/Niffler/

BSD 3-Clause "New" or "Revised" License

90 stars 53 forks source link

If there are large no. of mapping or metadata files - concatenating or merging them leads to a Memory Error. #307

Closed pradeeban closed 1 year ago

pradeeban commented 2 years ago

Spotted with DICOMs of 1.1TB and 1000 metadata files (small files).

anbhimi commented 1 year ago

The MemoryError pops up when we have a large no. of DICOMs in the provided cohort/iteration. Thus, increasing the size of metadata and mapping data frames leads to memory errors in pandas.

A simple fix is to reduce the no. of DICOMs in the cohort leading to smaller metadata and mapping data frames that pandas could handle to merge or concatenate. The information is posted in the troubleshooting section of PNG Extraction README.md.