rustprooflabs
commented
1 month ago Thank you for reporting this @jhon100! I haven't done much with these options, seems like there's a bug lurking surrounding how I added --force into the mix. My thought is your 2nd command should work w/out --force and we just need to add an appropriate check for --update mode. Does that match your expectations?
That logic should fit in to ImportMode.okay_to_run() (here-ish) with a check like this (untested):
if self.update == 'append':
return TrueIt will likely be a few days before I can test this. If you want to make a PR in the meantime I'd be happy to review/merge that. If not, I'll get there before too long!
jhon100
Hi,
I installed the latest version of pgosm-flex on Docker Desktop yesterday, and it works correctly when importing data from one region, but it always fails when trying to add data from a second region.
The commands I used and part of the logs are below. As you can see, when executing command (2), it indicated that the --force argument was necessary. In command (3), when using the --force command, it warned that the command was unnecessary. But the bigger problem is that it's giving a foreign key error. I tested several times, with the database empty and also by reversing the order of the regions, and the problem always occurs with the second region.
(1) python3 docker/pgosm_flex.py --ram=2 --input-file=south_australia.osm.pbf --update create --skip-nested
Result: [...] ...:INFO:pgosm-flex:pgosm_flex:Skipping calculating nested polygons ...:INFO:pgosm-flex:pgosm_flex:Skipping pg_dump ...:INFO:pgosm-flex:pgosm_flex:PgOSM Flex complete!
(2) python3 docker/pgosm_flex.py --ram=2 --input-file=northern_territory.osm.pbf --update append --skip-nested
Result: [...] ...:ERROR:pgosm-flex:helpers:Prior data exists in the osm schema and --force was not used.
(3) python3 docker/pgosm_flex.py --ram=2 --input-file=northern_territory.osm.pbf --force --update append --skip-nested
Result: [...] ...:WARNING:pgosm-flex:pgosm_flex:Using --force with the built-in database is unnecessary. The pgosm database is always dropped and recreated when running on localhost (in Docker). ...:INFO:pgosm-flex:db:Database already existed. ...:INFO:pgosm-flex:db:Preparing database schema: osm [...] ...:INFO:pgosm-flex:helpers:2024-07-28 22:40:40 ERROR: DB copy thread failed: Database error: ERROR: update or delete on table "place_polygon" violates foreign key constraint "fk_place_polygon_nested" on table "place_polygon_nested" ...:INFO:pgosm-flex:helpers:DETAIL: Key (osm_id)=(-2316596) is still referenced from table "place_polygon_nested". [...]
LOG ...STATEMENT: DELETE FROM "osm"."place_polygon" WHERE osm_id IN (-3676,-3918,-3919,-8896,-9487,-153172,[...],-17848463)