darold / ora2pg

Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL compatible schema. It connects your Oracle database, scan it automatically and extracts its structure or data, it then generates SQL scripts that you can load into PostgreSQL.
http://www.ora2pg.com/
GNU General Public License v3.0
978 stars 341 forks source link

Add support for oracle_fdw in combination with psql "\copy" and server-side COPY using BINARY stream #1783

Closed nasmart closed 1 month ago

nasmart commented 1 month ago

This pull request adds two new modes of operation for Ora2Pg when using oracle_fdw for data transport:

These modes are only available when using oracle_fdw (FDW_SERVER set) with COPY.

The specific mode of COPY is controlled via a newly introduced configuration (ORACLE_FDW_BINARY_COPY_MODE, which can be either "local" or "server", defaulting to "local").

local (psql-based) should work on any PostgreSQL-based system, including managed offerings such as Cloud SQL for PostgreSQL, AlloyDB for PostgreSQL and Amazon Aurora.

server is expected to only work in self-managed/unmanaged PostgreSQL environments.

The motivation for the new modes is purely performance: ~3x increase in throughput has been observed with "local" mode. "server" is expected to perform even better when comparing like-for-like PostgreSQL systems.

Notes:

  1. The pull request changes the behaviour of COPY when used in combination with oracle_fdw (FDW_SERVER set). The prior behaviour was that INSERT and COPY both use INSERT statements. The changes in this pull request result in COPY now using a form of PostgreSQL "COPY".

  2. The pull request also includes significant changes to doc/ora2pg.3 caused by the change from Pod::Man 4.14 to Pod::Man 5.01 for documentation generation.

darold commented 1 month ago

Thanks for the patch. I have made on change in commit 1e9effc to use $ENV{PGPASSWORD} = $self->{dbpwd}; to set the environment variable without exposing the password in the psql command and to avoid setting a .pgpass file.