Describe the bug
Very unfortunate bug that's not really our doing, but 2 of the datasets in statistics.gov.scot have incomplete date objects (only time - see screenshot), so when we pull it in, datetime convertors auto-fills today's date. Meaning the updated date constantly changes (even though we don't actually know the last date).
To Reproduce
See data/scotgov-datasets-sparkql.csv and example source. The api call comes from sparkql_statistics.py
Expected behavior
Add handling - if date cannot be determined then return as NULL
Describe the bug Very unfortunate bug that's not really our doing, but 2 of the datasets in statistics.gov.scot have incomplete date objects (only time - see screenshot), so when we pull it in, datetime convertors auto-fills today's date. Meaning the updated date constantly changes (even though we don't actually know the last date).
To Reproduce See
data/scotgov-datasets-sparkql.csv
and example source. The api call comes fromsparkql_statistics.py
Expected behavior Add handling - if date cannot be determined then return as NULL
Screenshots
Hardware and software used NA
Additional context None