search

yugabyte / yb-voyager

Data migration Engine for YugabyteDB database
38 stars 10 forks source link

Partial fix for [export-data errors out if background metadata queries (count*) are still running when pg_dump completes.] #1932

Closed makalaaneesh closed 1 day ago

makalaaneesh commented 2 days ago

Implements fixes 1,3 in https://github.com/yugabyte/yb-voyager/issues/1920 : When goroutines A,B,C,D encounter an error, they should not bring down the process, they should silently log the error and continue, as this is not a critical process.

makalaaneesh commented 2 days ago

@makalaaneesh can we do this change also

SetMaxIdleConns(paralleljobs)

To me this seems harmless and should fix the issue for large databases to a go extend, what do you think.

Yeaah, I was thinking about it. For the most part, no concerns. Just about total connections to source db being parallel-jobs x 2 (pg_dump/ora2pg using parallel-jobs and voyager layer using parallel-jobs). But since it's capped, I don't think it should be a big problem. Let me go ahead and do it. cc: @hbhanawat

makalaaneesh commented 1 day ago

https://jenkins.dev.yugabyte.com/job/users/job/yb-voyager-testing/job/yb-voyager-testing-pipeline/3998/