search

simonw / scrape-open-data

Scrape various open data directories to create an index of what's available out there
https://open-data.datasette.io
28 stars 2 forks source link

Deleted records still show up #6

Closed simonw closed 2 years ago

simonw commented 2 years ago

As illustrated by: https://open-data.datasette.io/socrata/resources?_sort=id&domain__isblank=1

image

It looks to me like deleted records stay listed in the database, maybe due to the code that restores their old stored statistics.

simonw commented 2 years ago

Relevant code: https://github.com/simonw/scrape-open-data/blob/6cbeb68506164b2244ec752f439c80a767f0a5d2/build_socrata_db.py#L22-L31

simonw commented 2 years ago

I'm going to look for link is null and delete those records at the end of the script: https://open-data.datasette.io/socrata/resources?_sort=id&link__isnull=1

simonw commented 2 years ago

https://open-data.datasette.io/socrata/resources now lists 10,484 records, none of which have a blank link field.