search

simonw / datasette-socrata

Import data from Socrata into Datasette
Apache License 2.0
0 stars 1 forks source link

Errors during import are silently ignored #12

Closed simonw closed 2 years ago

simonw commented 2 years ago

While trying to import https://data.sfgov.org/Geographic-Locations-and-Boundaries/Parcels-Active-and-Retired/acdm-wktn I got this error in the server logs (but invisible to the end user):

future: <Task finished name='Task-2144' coro=<import_socrata.<locals>.run_the_import() done, defined at /usr/local/lib/python3.9/site-packages/datasette_socrata/__init__.py:224> exception=Error('field larger than field limit (131072)')>

I can implement the same approach I used here:

See https://github.com/simonw/datasette-upload-csvs/commit/1166fce451ba32846c2812c9dd12aeb8bfcadd5a

simonw commented 2 years ago

I'm going to add a "error" column to the socrata_imports table to store these errors, and teach the polling code to notice if they happen.

simonw commented 2 years ago

I'll need this to implement disk space checks which return errors if there is not enough space left:

simonw commented 2 years ago

I also need the polling JavaScript to look out for and display errors, rather than polling forever.