search

darold / ora2pg

Ora2Pg is a free tool used to migrate an Oracle database to a PostgreSQL compatible schema. It connects your Oracle database, scan it automatically and extracts its structure or data, it then generates SQL scripts that you can load into PostgreSQL.
http://www.ora2pg.com/
GNU General Public License v3.0
991 stars 342 forks source link

Progress bar shows wrong totals and percentage when using WHERE #1631

Closed alamp1360 closed 1 year ago

alamp1360 commented 1 year ago

Hi!

I am using Ora2Pg v23.2 running inside a docker container. Image is docker.io/georgmoser/ora2pg:latest. Ora2Pg is used for migrating data from a oracle table with about 27 million entries. Due to some difficulties with dates and times, i need to do the migration in steps of one month at a time, so i am using WHERE clauses. The export/import runs like a charm, however, the progess bar does not show the "real" progress, but always shows percentages from the total number of rows in the table. This is the output when the export/import job has finished:

[========================>] 1/1 tables (100.0%) end of scanning. [> ] 1068511/27485030 rows (3.9%) Table LOGENTRY (455 recs/sec) [> ] 1068511/27485030 total rows (3.9%) - (2347 sec., avg: 455 recs/sec). [========================>] 27485030/27485030 rows (100.0%) on total estimated data (2347 sec., avg: 11710 recs/sec)

As you can see, there were just about 1 Million records processed, but the progress bar did not run up to 100%.

I know it is just a minor issue, but it would be nice to see the actual numbers to estimate how long the process will run.

Greetings

darold commented 1 year ago

This is expected, Ora2Pg base the row count on Oracle statistic during data export and of course it has no idea of the WHERE clause cardinality. Running a real row count is too much time expensive.