rustprooflabs / pgosm-flex

PgOSM Flex provides high quality OpenStreetMap datasets in PostGIS (Postgres) using the osm2pgsql Flex output.
MIT License
100 stars 20 forks source link

Move `osm.pgosm_flex` record creation from Lua to Python #304

Closed rustprooflabs closed 1 year ago

rustprooflabs commented 1 year ago

Details

I think the creation of the osm.pgosm_flex record during import could be moved from Lua to Python. This would remove some of the awkwardness related to that table's record management, such as the UUID added in https://github.com/rustprooflabs/pgosm-flex/pull/303 to track errors in replication. Having the initial INSERT happening from Python will allow using RETURNING id and the UUID component can be removed.

Having this step in Lua was a side effect of the pre-Docker usage. It has been at least 6 months since I have even attempted the non-Docker approach, and I intend to remove all of the documentation about that. So from my perspective that isn't a big consideration, though if others are using this I'd love to hear about it. I'm not set on this idea (yet) but that's the direction I'm leaning.

Moving forward on this is one more step towards only supporting the Docker usage.

rustprooflabs commented 1 year ago

This change will likely include changing --data-only to --skip-qgis-style. The DDL components currently in pgosm-meta.lua and pgosm-meta.sql will move into a sqitch migration. That change will allow deploying the sqitch portion without requiring loading the QGIS style, and making it more clear at the same time.

E.g. planning to change this part in docker/db.py


    if not data_only:
        LOGGER.info('Loading extras via Sqitch plus QGIS styles.')
        run_sqitch_prep(db_path)
        qgis_styles.load_qgis_styles(db_path=db_path,
                                     db_name=pg_conn_parts()['pg_db'],
                                     schema_name=schema_name)
    else:
        LOGGER.info('Data only mode enabled, no Sqitch or QGIS styles.')
rustprooflabs commented 1 year ago

The dev branch is tagged with 0.8.0-dev.1 at commit dcd6454. Tested update with replication from 0.7.2, seems to go smoothly. See #312 for the summary of changes. Test from dev image.

docker pull rustprooflabs/pgosm-flex:dev
rustprooflabs commented 1 year ago

I've tested this a few ways, keeps going smoothly with no manual steps.