mattyschell / geodatabase-replication-toiler

Creative Commons Zero v1.0 Universal
1 stars 0 forks source link

Tweak Subaddress With Periodic Commits #11

Closed mattyschell closed 1 year ago

mattyschell commented 1 year ago

The SQL generator here is technically correct but gets wacky attempting to divide the inserts into 2 chunks with a commit in the middle. Even with the wackiness we still get out of memory errors on load to PostgreSQL in Azure.

https://github.com/mattyschell/geodatabase-replication-toiler/blob/main/src/sql/data/oracle/subaddress_export2postgres.sql

The workaround for now is to drop 2-3 additional commits manually into the load.sql.

Better will be to refactor the SQL to write out the SQL plainly and then rewrite the file automatically, parameterizing the number of records per commit.

rewriteinserts.py subaddress_load2postgres.sql loadready.sql 250000