search

omniscale / imposm3

Imposm imports OpenStreetMap data into PostGIS
http://imposm.org/docs/imposm3/latest/
Apache License 2.0
711 stars 156 forks source link

Post Replication SQL Hook #190

Closed thomersch closed 3 years ago

thomersch commented 5 years ago

As requested by @giggls a post replication hook is implemented: E.g. when materialised views need updating after an OSM update import has been completed. To reduce impact during catch-up, it is only applied on the most recent update.

A configuration option has been added to the JSON file, which can now carry an SQL query that is executed.

If you have any questions, let us know. We will be both at FOSSGIS in Dresden, in case there is a need for discussion IRL.

olt commented 5 years ago

Citing from https://groups.google.com/d/msg/imposm/B8hxfcPkSN0/s3EZTF61CgAJ

I think an option to imposm run to exit after one import is a nicer solution. This way you can integrate Imposm into your script, instead of adding a "process management" to Imposm. You can do this already with the undocumented IMPOSM3_SINGLE_DIFF environment (IMPOSM3_SINGLE_DIFF=1 imposm run).

Another solution would be to make Imposm to call arbitrary SQL commands after an import/update. This makes it more complex, but the commands can run in the same SQL transaction, giving you a clean all-or-nothing.

Your patch works outside the transaction, so there is no benefit from it compared to the IMPOSM3_SINGLE_DIFF hack (which could be improved, so that is becomes an official option).

thomersch commented 5 years ago

I see a benefit in so far as Sven has currently wrapper scripts that have to check from the outside whether imposm is still catching up with replication, which basically defies the purpose of the loop in imposm itself. I think we have found a general purpose solution in a central place which feels less like a hack than the SINGLE_DIFF solution.

As for the transaction I would be happy to change it to use the update tx.

The current patch does not break anything: When the user specifies an SQL command, it will be executed, otherwise nothing happens at all. I agree that running in the context of the transaction makes total sense, but already there is an operational improvement. We have tested it during the Hackweekend together, so the needed wrapper script could go away and no breakage was introduced.

olt commented 5 years ago

Too bad we couldn't discuss this IRL, but I had to cancel FOSSGIS this year :-/

The current patch does not break anything: When the user specifies an SQL command, it will be executed, otherwise nothing happens at all. I agree that running in the context of the transaction makes total sense, but already there is an operational improvement.

But adding support for transactions will then break your current patch...

This patch needs to support for transactions from the start. Please also update the documentation and tests.