Improve output, save validation results to file, add more date validations - Githubissues

CanDIG / clinical_ETL_code

A library to convert clinical data in csv format into json documents for CanDIG ingestion

GNU Lesser General Public License v3.0

0 stars 0 forks source link

Improve output, save validation results to file, add more date validations #61

Closed mshadbolt closed 5 months ago

mshadbolt commented 5 months ago

`requirements.txt`

Added tqdm to allow for progress bar

`CSVConvert.py`

Output to commandline improvements:
- Colouring for better parseability
- Less output to command line that does not need to be fixed if verbose mode not enabled
- Progress bar for packet creation
- Output warnings and errors to separate file, no longer saved into _map.json - Would this mess with ingest?
- Warnings and errors not shown on commandline if more than 20 found
- Returns packets and whether errors are present
- Improved messaging so user knows when a file is ingestable

`mohschema.py`

Validations to check dates make sense
- treatment, diagnosis, birth dates cannot be after death date
- treatment, diagnosis, birth dates cannot be before birth date
- death date cannot be before birth date
- At least one '0' diagnosis date must be present
- stop overriding type() method

`mappings.py`

addition of _info method to distinguish when information is printed rather than a warning that requires action

tests

Changes to make the test data pass with the new validations