OpenDataScotland / the_od_bods

Collating open data from across Scotland
MIT License
21 stars 18 forks source link

sparkql_statistics.py treat incomplete date as null #215

Open KarenJewell opened 1 year ago

KarenJewell commented 1 year ago

Describe the bug Very unfortunate bug that's not really our doing, but 2 of the datasets in statistics.gov.scot have incomplete date objects (only time - see screenshot), so when we pull it in, datetime convertors auto-fills today's date. Meaning the updated date constantly changes (even though we don't actually know the last date).

To Reproduce See data/scotgov-datasets-sparkql.csv and example source. The api call comes from sparkql_statistics.py

Expected behavior Add handling - if date cannot be determined then return as NULL

Screenshots Screenshot 2023-01-29 at 18 02 17

Hardware and software used NA

Additional context None

KarenJewell commented 1 year ago

Add the fix anyway, but there's an admin to-do for me