Add RICORD data creation and include in train/test split

[andyzzhao]

Pull Request Template

Description

• Added create_ricord_dataset/create_ricord_dataset.ipynb to crop and convert RICORD data from DICOM to PNG.
• Added RICORD data processing to create_COVIDx.ipynb and create_COVIDx_binary.ipynb where 200 RICORD images are add to test in both notebooks

New workflow for creating COVIDx dataset:

1. Download required images and files
2. Run create_ricord_data/create_ricord_dataset.ipynb
3. Run create_COVIDx.ipynb and/or create_COVIDx_binary.ipynb

Context of change

Please add options that are relevant and mark any boxes that apply.

• [x] Software (software that runs on the PC)
• [ ] Library (library that runs on the PC)
• [ ] Tool (tool that assists coding development)
• [ ] Other

Type of change

Please mark any boxes that apply.

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration.

create_COVIDx.ipynb and create_COVIDx_binary.ipynb were ran to confirm 200 RICORD images were added to test and the rest were added to train.

Checklist:

Please mark any boxes that have been completed.

• [x] I have performed a self-review of my own code.
• [x] I have commented my code, particularly in hard-to-understand areas.
• [ ] I have made corresponding changes to the documentation.
• [x] My changes generate no new warnings.
• [ ] Any dependent changes have been merged and published in downstream modules.

[haydengunraj]

@andyzzhao this looks good, but we might hold off on merging until we're ready to release this along with the BIMCV data. We'll discuss at tomorrow's meeting.

[haydengunraj]

@mayaliliya in the v8 label files, have we fixed the issues related to inconsistent labels that we had in the v7 dataset (e.g., #126)?

[mayaliliya]

@mayaliliya in the v8 label files, have we fixed the issues related to inconsistent labels that we had in the v7 dataset (e.g., #126)?

I fixed it in the data.py script with the brute force pop method. I am thinking we just merge this and then I will do a separate PR to address this bug next week as well as thoroughly addressing the duplicate issue (i.e. seeing if we can scavenge more images rather than removing all images with the same url base).