Data Validation To Do

[seshafi]

Data validation checks:

1. [ ] Correct data file format
2. [x] Correct column names
3. [x] No empty observations
4. [x] Missingness not beyond expected threshold
5. [x] Correct data types in each column
6. [x] No duplicate observations
7. [ ] No outlier or anomalous values
8. [x] Correct category levels (i.e., no string mismatches or single values)
9. [ ] Target/response variable follows expected distribution
10. [ ] No anomalous correlations between target/response variable and features/explanatory variables
11. [ ] No anomalous correlations between features/explanatory variables

After completing checklist:

• [ ] go back to "missingness not beyond threshold" and try to add that to Pa.checks
• [ ] handle invalid data
• [ ] for thalessemia & fluoroscopy (# major vessels): add the data range check

[seshafi]

We are unclear on how to check:

• correct data file format
• target/response following expected distribution
• No anomalous correlations between target/response variable and features/explanatory variables
• No anomalous correlations between features/explanatory variables

Check with Daniel/Tiff tomorrow.

Other questions:

• How can we incorporate multiple pa checks for one column?
• How should we correct issues (with data type, for ex)
• Our null value limit code looks messy. Should we limit the columns we do it on or is there a way to make a function or smth?

[seshafi]

Invalid data issues:

1. Exercised-induced angina: should be bool but it is currently object
2. number of major vessels: should be int but it's currently a float
3. Diagnosis of heart disease: should be limited to 0 and 1