CrunchyData / pg_parquet

Copy to/from Parquet in S3 from within PostgreSQL
Other
355 stars 11 forks source link

COPY progress reporting is not supported #40

Open onderkalaci opened 1 month ago

onderkalaci commented 1 month ago

When you normally do COPY FROM/TO on heap tables, they show up in pg_stat_progress_copy view.

With pg_parquet, when I do COPY .. TO .. (parquet), the view is not populated:

select * from pg_stat_progress_copy;
(0 rows)

When I do COPY .. FROM (parquet), bytes_total is not populated, which is making the progress reporting less useful than it could be:

select * from pg_stat_progress_copy;
-[ RECORD 1 ]----+----------
pid              | 60146
datid            | 5
datname          | postgres
relid            | 24770
command          | COPY FROM
type             | CALLBACK
bytes_processed  | 389524432
bytes_total      | 0
tuples_processed | 22978000
tuples_excluded  | 0
tuples_skipped   | 0
aykut-bozkurt commented 3 weeks ago

Our custom parquet dest receiver api does not update copy progress stats table. 2 solutions: