Implement data validation for datasets

[fstayco]

Our data directory houses crucial raw datasets. Help us ensure the integrity of this data by implementing validation checks. Whether it's consistency checks or automating the validation process, your efforts will ensure our models run on clean data.

[Bchass]

I'm interested in this problem. What would be the main checks that should happen?

Some that come to mind as I'm thinking about this:

• File encoding
• File extension
• File size
• Data duplication
• Data consistency (e.g., date format is the same across all rows)

.geojson and .jpg would be a bit different with their checks