simonw / geojson-to-sqlite

CLI tool for converting GeoJSON files to SQLite (with SpatiaLite)
Apache License 2.0
57 stars 6 forks source link

`--spatialite` fails if features have no properties #30

Open simonw opened 2 years ago

simonw commented 2 years ago

https://raw.githubusercontent.com/datanews/amtrak-geojson/master/amtrak-combined.geojson

Has a feature like this:

"properties": { }

This fails with an error if you use the --spatialite option:

geojson-to-sqlite /tmp/amtrak.db amtrak /tmp/amtrak-combined.geojson --spatialite                                           
Traceback (most recent call last):
  File "/Users/simon/.local/bin/geojson-to-sqlite", line 8, in <module>
    sys.exit(cli())
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1134, in __call__
    return self.main(*args, **kwargs)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1059, in main
    rv = self.invoke(ctx)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/click/core.py", line 1401, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/click/core.py", line 767, in invoke
    return __callback(*args, **kwargs)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/geojson_to_sqlite/cli.py", line 32, in cli
    utils.import_features(
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/geojson_to_sqlite/utils.py", line 69, in import_features
    db[table].create(column_types, pk=pk)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 863, in create
    self.db.create_table(
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 517, in create_table
    self.execute(sql)
  File "/Users/simon/.local/pipx/venvs/geojson-to-sqlite/lib/python3.9/site-packages/sqlite_utils/db.py", line 236, in execute
    return self.conn.execute(sql)
sqlite3.OperationalError: near ")": syntax error
simonw commented 2 years ago

Here's why:

https://github.com/simonw/geojson-to-sqlite/blob/17c2d0995bec989fa38ef3a593649908833b7b0a/geojson_to_sqlite/utils.py#L70-L75

Since geometry is added later the call to db[table].create(column_types, pk=pk) has an empty column_types dictionary.

simonw commented 2 years ago

Really hard to know what to do about this one - the GeoJSON object here really is just a geometry with nothing else. There's not even an ID that can be used as a primary key.

So we really want to create a SQLite table with just a rowid. Is that even possible?

simonw commented 2 years ago

https://stackoverflow.com/a/36926664/6083 suggests using this:

CREATE TABLE tablename (rowid INTEGER PRIMARY KEY) WITHOUT ROWID;

That makes me a little bit nervous, see https://www.sqlite.org/withoutrowid.html - it's not clear to me if this kind of "fake" rowid table will cause weird problems in the future, especially if I start using tricks like this one: https://til.simonwillison.net/sqlite/track-timestamped-changes-to-a-table

I'm tempted to create a id column that is primary key integer and hence will auto-increment.

simonw commented 2 years ago

I think the safest thing to do here would be to create the table with a _temp column, add the geometry column and then drop that _temp column.

simonw commented 2 years ago

Fixed it! Here's that GeoJSON imported and rendered using https://datasette.io/plugins/datasette-geojson-map

image
simonw commented 2 years ago

I think I spotted a bug: https://github.com/simonw/geojson-to-sqlite/blob/23bdb74f857b067fbddbc924fd27b64d3441a46c/geojson_to_sqlite/utils.py#L70-L77

remove_tmp_column is never switched to True.