Feature Request: improved Online DDL row estimation

[shlomi-noach]

Feature Description

In Online DDL ALTER TABLE migrations, Vitess estimates the number of table rows via SHOW TABLE STATUS. This number can be very skewed. As the migration runs, Vitess indicates its progress in _vt.schema_migrations as a [0..100] value. This column is also exposed in SHOW VITESS_MIGRATIONS output.

The problem is that table_rows can be so skewed that:

• At some stage rows_copied is actually higher than table_rows. The migration shows 99% or 100% for hours until it actually completes.
• Or, the migration suddenly terminates at 75%.

This happens because the estimated number of rows can be very much higher or very much lower than the actual number of rows in the table. The user experience is poor as the user is never sure what the actual status is.

Potential approaches:

1. Run ANALYZE TABLE before/during the migration to get better row estimate. This is an easy solution. The row estimate improves, but still remains imprecise (typically pushing down the error margin to 5%)
2. Run SELECT COUNT(*) before the migration, and update count with ongoing processed events from the binary log. Problem is that this query may actually run for hours, sometimes even longer than the migration itself. Also, it had better run on a replica.

Use Case(s)

Any long running Online DDL (schema change/migration)

[shlomi-noach]

Addressed by https://github.com/vitessio/vitess/pull/13352

[shlomi-noach]

I'm closing this issue as https://github.com/vitessio/vitess/pull/13352 offers a reasonable solution.