search

yugabyte / yb-voyager

Data migration Engine for YugabyteDB database
38 stars 10 forks source link

[Voyager] export-data errors out if background metadata queries (count*) are still running when pg_dump completes. #1920

Open makalaaneesh opened 6 days ago

makalaaneesh commented 6 days ago

Jira Link: DB-14122

Description

Sample logs: 

2024-11-16 18:16:47.240719 ERROR logging.go:30 Failed to query "select count(*) from public.\"table_1\"" for row count of "[CurrentName=(table_1) SourceName=(table_1) TargetName=()]": sql: database is closed

2024-11-16 18:16:47.240989 ERROR logging.go:30 Failed to query "select count(*) from public.\"table_2\"" for row count of "[CurrentName=(table_2) SourceName=(table_2) TargetName=()]": sql: database is closed

This happens when :

  1. goroutine A is executing count(*) on table x
  2. goroutines B,C,D,... are waiting to get access to the connection in order to execute count(*) on tables y,z,a,...
  3. pg_dump completes, therefore as part of clean up in the main goroutine, we call source.DB().Disconnect() . which closes the connections that goroutines A,B,C,D are using/waiting to use.
  4. goroutines A,B,C,D encounter error and exit the application.

Fix:

  1. Even though there are multiple tables being exported at the same time, there is only one connection in the pool for goroutines A,B,C,D. This is the reason there is a backlog of goroutines waiting to execute count(*). The pool size should be equal to parallel-jobs
  2. The goroutine running count(*) on table X should be shut down after export of table X is complete. This does not happen as of now.
  3. When goroutines A,B,C,D encounter an error, they should not bring down the process, they should silently log the error and continue, as this is not a critical process.

Issue Type

kind/bug

Warning: Please confirm that this issue does not contain any sensitive information