search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.01k stars 4.11k forks source link

Scaling Airbyte Instances with PGBouncer - Connection Issues #44447

Open durvalcarvalho opened 2 months ago

durvalcarvalho commented 2 months ago

Platform Version

0.50.29

What step the error happened?

On Deploy

Relevant information

I’m facing an issue when trying to run Airbyte/Temporal with PgBouncer.

At my workplace, we have some instances of Airbyte, all sharing a single database server, with each instance using its own exclusive set of databases.

As the number of Airbyte instances has grown, we noticed that the number of open connections was increasing rapidly, exhausting the database server’s resources. I started encountering the error:

FATAL: remaining connection slots are reserved for non-replication super-users

$ SELECT usename, count(*) FROM pg_stat_activity group by usename;
+------------+-------+
|  usename   | count |
+------------+-------+
| instance_1 |    48 |
| instance_2 |    56 |
| postgres   |    10 |
| instance_3 |    17 |
| rdsadmin   |     3 |
| instance_4 |    53 |
+------------+-------+

By observing these connections, it became clear that they are opened and not closed, with almost all of them remaining in an idle state.

We first attempted to use the SQL_MAX_IDLE_CONNS setting as described here, but this didn’t work; setting this variable didn’t make any difference.

Our idea was to use PgBouncer in transaction pool mode to avoid this large number of open connections, but the airbyte/temporal services didn’t behave well when running behind PgBouncer.

The error I’m seeing is:

pq: bind message supplies X parameters, but prepared statement \"\" requires Y

After researching this issue further, I discovered that this is happening because of prepared statements, which by default do not work with this type of DB connection pooling. One potential solution is to use SQL_CONNECT_ATTRIBUTES=binary_parameters=yes, but nothing changed; it seems that airbyte/temporal is not using this connection attribute.

The full log can be found here.

Some questions I have:

  1. Is this large number of database connections expected?
  2. Should Temporal work when running behind PgBouncer, or is this a pattern that should be avoided?

Links to identical and similar issues:

Relevant log output

Attaching to airbyte-temporal
airbyte-temporal  | Start init
airbyte-temporal  | Done init
airbyte-temporal  | PostgreSQL started.
airbyte-temporal  | Starting to update the temporal DB
airbyte-temporal  | Update the temporal DB is done
airbyte-temporal  | Starting to update the temporal visibility DB
airbyte-temporal  | Update the temporal visibility DB is done
airbyte-temporal  | starting temporal server
airbyte-temporal  | Temporal CLI address: 192.168.48.3:7233.
airbyte-temporal  | Waiting for Temporal server to start...
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:42.801Z","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: pq: bind message supplies 2 parameters, but prepared statement \"\" requires 8","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\t/home/builder/temporal/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).Start\n\t/home/builder/temporal/common/namespace/registry.go:212\ngo.temporal.io/server/common/namespace.RegistryLifetimeHooks.func1\n\t/home/builder/temporal/common/namespace/fx.go:44\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:130\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:95\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:679\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:784"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:42.801Z","msg":"Operation failed with internal error.","error":"UpsertClusterMembership operation failed. Error: pq: bind message supplies 8 parameters, but prepared statement \"\" requires 2","operation":"UpsertClusterMembership","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).UpsertClusterMembership.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1115\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).UpsertClusterMembership\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1117\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).UpsertClusterMembership.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:948\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).UpsertClusterMembership\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:951\ngo.temporal.io/server/common/membership.(*ringpopMonitor).upsertMyMembership\n\t/home/builder/temporal/common/membership/rpMonitor.go:190\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeat\n\t/home/builder/temporal/common/membership/rpMonitor.go:253\ngo.temporal.io/server/common/membership.(*ringpopMonitor).Start\n\t/home/builder/temporal/common/membership/rpMonitor.go:133\ngo.temporal.io/server/common/resource.MembershipMonitorProvider.func1\n\t/home/builder/temporal/common/resource/fx.go:287\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:130\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:95\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:679\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:784"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:42.868Z","msg":"Operation failed with internal error.","error":"GetClusterMembers operation failed. Error: pq: bind message supplies 4 parameters, but prepared statement \"\" requires 2","operation":"GetClusterMembers","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).GetClusterMembers.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1103\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).GetClusterMembers\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1105\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).GetClusterMembers.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:935\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).GetClusterMembers\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:939\ngo.temporal.io/server/common/membership.(*ringpopMonitor).fetchCurrentBootstrapHostports\n\t/home/builder/temporal/common/membership/rpMonitor.go:273\ngo.temporal.io/server/common/membership.(*ringpopMonitor).Start.func1\n\t/home/builder/temporal/common/membership/rpMonitor.go:139\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap.func1\n\t/home/builder/temporal/common/membership/ringpop.go:95\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:175\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:176\ngo.temporal.io/server/common/membership.(*RingPop).bootstrap\n\t/home/builder/temporal/common/membership/ringpop.go:114\ngo.temporal.io/server/common/membership.(*RingPop).Start\n\t/home/builder/temporal/common/membership/ringpop.go:84\ngo.temporal.io/server/common/membership.(*ringpopMonitor).Start\n\t/home/builder/temporal/common/membership/rpMonitor.go:138\ngo.temporal.io/server/common/resource.MembershipMonitorProvider.func1\n\t/home/builder/temporal/common/resource/fx.go:287\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:130\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:95\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:679\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:784"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:42.869Z","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: pq: bind message has 6 result formats but query has 3 columns","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\t/home/builder/temporal/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).Start\n\t/home/builder/temporal/common/namespace/registry.go:212\ngo.temporal.io/server/common/namespace.RegistryLifetimeHooks.func1\n\t/home/builder/temporal/common/namespace/fx.go:44\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:130\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:95\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:679\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:784"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:42.919Z","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: pq: bind message has 6 result formats but query has 3 columns","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\t/home/builder/temporal/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).Start\n\t/home/builder/temporal/common/namespace/registry.go:212\ngo.temporal.io/server/common/namespace.RegistryLifetimeHooks.func1\n\t/home/builder/temporal/common/namespace/fx.go:44\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).runStartHook\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:130\ngo.uber.org/fx/internal/lifecycle.(*Lifecycle).Start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/internal/lifecycle/lifecycle.go:95\ngo.uber.org/fx.(*App).start\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:679\ngo.uber.org/fx.withTimeout.func1\n\t/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:784"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:43.022Z","msg":"Operation failed with internal error.","error":"GetOrCreateShard: failed to get ShardID 4. Error: pq: bind message supplies 1 parameters, but prepared statement \"\" requires 2","operation":"GetOrCreateShard","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).GetOrCreateShard.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:163\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).GetOrCreateShard\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:165\ngo.temporal.io/server/common/persistence.(*shardRetryablePersistenceClient).GetOrCreateShard.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:168\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*shardRetryablePersistenceClient).GetOrCreateShard\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:172\ngo.temporal.io/server/service/history/shard.(*ContextImpl).loadShardMetadata\n\t/home/builder/temporal/service/history/shard/context_impl.go:1737\ngo.temporal.io/server/service/history/shard.(*ContextImpl).acquireShard.func1\n\t/home/builder/temporal/service/history/shard/context_impl.go:1894\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:175\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:176\ngo.temporal.io/server/service/history/shard.(*ContextImpl).acquireShard\n\t/home/builder/temporal/service/history/shard/context_impl.go:1962"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:43.025Z","msg":"Operation failed with internal error.","error":"GetOrCreateShard: failed to get ShardID 2. Error: pq: bind message supplies 1 parameters, but prepared statement \"\" requires 2","operation":"GetOrCreateShard","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).GetOrCreateShard.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:163\ngo.temporal.io/server/common/persistence.(*shardPersistenceClient).GetOrCreateShard\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:165\ngo.temporal.io/server/common/persistence.(*shardRetryablePersistenceClient).GetOrCreateShard.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:168\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*shardRetryablePersistenceClient).GetOrCreateShard\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:172\ngo.temporal.io/server/service/history/shard.(*ContextImpl).loadShardMetadata\n\t/home/builder/temporal/service/history/shard/context_impl.go:1737\ngo.temporal.io/server/service/history/shard.(*ContextImpl).acquireShard.func1\n\t/home/builder/temporal/service/history/shard/context_impl.go:1894\ngo.temporal.io/server/common/backoff.ThrottleRetry.func1\n\t/home/builder/temporal/common/backoff/retry.go:175\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/backoff.ThrottleRetry\n\t/home/builder/temporal/common/backoff/retry.go:176\ngo.temporal.io/server/service/history/shard.(*ContextImpl).acquireShard\n\t/home/builder/temporal/service/history/shard/context_impl.go:1962"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:43.160Z","msg":"Operation failed with internal error.","error":"CreateTaskQueue operation failed. Failed to make task queue 26084adf4a3f:33e53e45-7fad-43dd-bc90-061f34ecdb12 of type Workflow. Error: pq: bind message supplies 5 parameters, but prepared statement \"\" requires 6","operation":"CreateTaskQueue","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).CreateTaskQueue.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:543\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).CreateTaskQueue\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:545\ngo.temporal.io/server/service/matching.(*taskQueueDB).takeOverTaskQueueLocked\n\t/home/builder/temporal/service/matching/db.go:147\ngo.temporal.io/server/service/matching.(*taskQueueDB).RenewLease\n\t/home/builder/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry.func1\n\t/home/builder/temporal/service/matching/taskWriter.go:302\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry\n\t/home/builder/temporal/service/matching/taskWriter.go:306\ngo.temporal.io/server/service/matching.(*taskWriter).initReadWriteState\n\t/home/builder/temporal/service/matching/taskWriter.go:131\ngo.temporal.io/server/service/matching.(*taskWriter).taskWriterLoop\n\t/home/builder/temporal/service/matching/taskWriter.go:221\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:43.124Z","msg":"Operation failed with internal error.","error":"GetHistoryTasks operation failed. Select failed. CategoryID: 5. Error: pq: bind message supplies 7 parameters, but prepared statement \"\" requires 4","operation":"GetArchivalTasks","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*executionPersistenceClient).GetHistoryTasks.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:371\ngo.temporal.io/server/common/persistence.(*executionPersistenceClient).GetHistoryTasks\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:373\ngo.temporal.io/server/common/persistence.(*executionRetryablePersistenceClient).GetHistoryTasks.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:366\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*executionRetryablePersistenceClient).GetHistoryTasks\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:370\ngo.temporal.io/server/service/history/queues.NewScheduledQueue.func1.1\n\t/home/builder/temporal/service/history/queues/queue_scheduled.go:91\ngo.temporal.io/server/common/collection.(*PagingIteratorImpl[...]).getNextPage\n\t/home/builder/temporal/common/collection/pagingIterator.go:116\ngo.temporal.io/server/common/collection.NewPagingIterator[...]\n\t/home/builder/temporal/common/collection/pagingIterator.go:52\ngo.temporal.io/server/service/history/queues.(*IteratorImpl).HasNext\n\t/home/builder/temporal/service/history/queues/iterator.go:71\ngo.temporal.io/server/service/history/queues.(*SliceImpl).SelectTasks\n\t/home/builder/temporal/service/history/queues/slice.go:375\ngo.temporal.io/server/service/history/queues.(*ReaderImpl).loadAndSubmitTasks\n\t/home/builder/temporal/service/history/queues/reader.go:434\ngo.temporal.io/server/service/history/queues.(*ReaderImpl).eventLoop\n\t/home/builder/temporal/service/history/queues/reader.go:403"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:43.217Z","msg":"Operation failed with internal error.","error":"CreateTaskQueue operation failed. Failed to make task queue 26084adf4a3f:916143c9-c032-4ca2-af39-b9a81fa1bf34 of type Workflow. Error: pq: bind message supplies 5 parameters, but prepared statement \"\" requires 2","operation":"CreateTaskQueue","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).CreateTaskQueue.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:543\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).CreateTaskQueue\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:545\ngo.temporal.io/server/service/matching.(*taskQueueDB).takeOverTaskQueueLocked\n\t/home/builder/temporal/service/matching/db.go:147\ngo.temporal.io/server/service/matching.(*taskQueueDB).RenewLease\n\t/home/builder/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry.func1\n\t/home/builder/temporal/service/matching/taskWriter.go:302\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry\n\t/home/builder/temporal/service/matching/taskWriter.go:306\ngo.temporal.io/server/service/matching.(*taskWriter).initReadWriteState\n\t/home/builder/temporal/service/matching/taskWriter.go:131\ngo.temporal.io/server/service/matching.(*taskWriter).taskWriterLoop\n\t/home/builder/temporal/service/matching/taskWriter.go:221\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:43.381Z","msg":"Operation failed with internal error.","error":"CreateTaskQueue operation failed. Failed to make task queue 26084adf4a3f:8d59639d-cedc-4706-91a3-b0f11485a7e5 of type Workflow. Error: pq: bind message supplies 5 parameters, but prepared statement \"\" requires 2","operation":"CreateTaskQueue","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).CreateTaskQueue.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:543\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).CreateTaskQueue\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:545\ngo.temporal.io/server/service/matching.(*taskQueueDB).takeOverTaskQueueLocked\n\t/home/builder/temporal/service/matching/db.go:147\ngo.temporal.io/server/service/matching.(*taskQueueDB).RenewLease\n\t/home/builder/temporal/service/matching/db.go:109\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry.func1\n\t/home/builder/temporal/service/matching/taskWriter.go:302\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/service/matching.(*taskWriter).renewLeaseWithRetry\n\t/home/builder/temporal/service/matching/taskWriter.go:306\ngo.temporal.io/server/service/matching.(*taskWriter).initReadWriteState\n\t/home/builder/temporal/service/matching/taskWriter.go:131\ngo.temporal.io/server/service/matching.(*taskWriter).taskWriterLoop\n\t/home/builder/temporal/service/matching/taskWriter.go:221\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:43.389Z","msg":"Operation failed with internal error.","error":"GetNamespace operation failed. Error pq: bind message supplies 2 parameters, but prepared statement \"\" requires 5","operation":"GetNamespace","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).GetNamespace.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:623\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).GetNamespace\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:625\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).GetNamespace.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:828\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).GetNamespace\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:832\ngo.temporal.io/server/service/frontend.(*namespaceHandlerImpl).DescribeNamespace\n\t/home/builder/temporal/service/frontend/namespace_handler.go:374\ngo.temporal.io/server/service/frontend.(*WorkflowHandler).DescribeNamespace\n\t/home/builder/temporal/service/frontend/workflow_handler.go:278\ngo.temporal.io/api/workflowservice/v1._WorkflowService_DescribeNamespace_Handler.func1\n\t/go/pkg/mod/go.temporal.io/api@v1.18.2-0.20230324225508-f2c7ab685b44/workflowservice/v1/service.pb.go:1390\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:63\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:67\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*CallerInfoInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/caller_info.go:80\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*SDKVersionInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/sdk_version.go:69\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/rate_limit.go:86\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceRateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_rate_limit.go:91\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceCountLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_count_limit.go:111\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).StateValidationIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:194\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/authorization.(*interceptor).Interceptor\n\t/home/builder/temporal/common/authorization/interceptor.go:153\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:157\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).handleLocalAPIInvocation\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:203\ngo.temporal.io/server/service/frontend.(*RedirectionInterceptor).Intercept\n\t/home/builder/temporal/service/frontend/redirection_interceptor.go:177\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceLogInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_logger.go:84\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*NamespaceValidatorInterceptor).NamespaceValidateIntercept\n\t/home/builder/temporal/common/rpc/interceptor/namespace_validator.go:111\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:137\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1155\ngo.temporal.io/api/workflowservice/v1._WorkflowService_DescribeNamespace_Handler\n\t/go/pkg/mod/go.temporal.io/api@v1.18.2-0.20230324225508-f2c7ab685b44/workflowservice/v1/service.pb.go:1392\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"}
airbyte-temporal  | temporal.api.workflowservice.v1.WorkflowService: SERVING
airbyte-temporal  | Temporal server started.
airbyte-temporal  | Registering default namespace: default.
airbyte-temporal  | Name: default
airbyte-temporal  | Id: c3a5f943-0cce-441d-b5c0-3cf3aa96a363
airbyte-temporal  | Description: Default namespace for Temporal Server.
airbyte-temporal  | OwnerEmail: 
airbyte-temporal  | NamespaceData: map[string]string(nil)
airbyte-temporal  | State: Registered
airbyte-temporal  | Retention: 720h0m0s
airbyte-temporal  | ActiveClusterName: active
airbyte-temporal  | Clusters: active
airbyte-temporal  | HistoryArchivalState: Disabled
airbyte-temporal  | IsGlobalNamespace: false
airbyte-temporal  | FailoverVersion: 0
airbyte-temporal  | FailoverHistory: []
airbyte-temporal  | VisibilityArchivalState: Disabled
airbyte-temporal  | Bad binaries to reset:
airbyte-temporal  | +-----------------+----------+------------+--------+
airbyte-temporal  | | BINARY CHECKSUM | OPERATOR | START TIME | REASON |
airbyte-temporal  | +-----------------+----------+------------+--------+
airbyte-temporal  | +-----------------+----------+------------+--------+
airbyte-temporal  | Default namespace default already registered.
airbyte-temporal  | Adding Custom*Field search attributes.
airbyte-temporal  | Search attributes already exist.
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:44.154Z","msg":"Operation failed with internal error.","error":"GetTasks operation failed. Failed to get rows. Error: pq: bind message supplies 5 parameters, but prepared statement \"\" requires 2","operation":"GetTasks","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTasks.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:507\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTasks\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:509\ngo.temporal.io/server/service/matching.(*taskQueueDB).GetTasks\n\t/home/builder/temporal/service/matching/db.go:222\ngo.temporal.io/server/service/matching.(*taskReader).getTaskBatchWithRange\n\t/home/builder/temporal/service/matching/taskReader.go:228\ngo.temporal.io/server/service/matching.(*taskReader).getTaskBatch\n\t/home/builder/temporal/service/matching/taskReader.go:249\ngo.temporal.io/server/service/matching.(*taskReader).getTasksPump\n\t/home/builder/temporal/service/matching/taskReader.go:181\ngo.temporal.io/server/internal/goro.(*Group).Go.func1\n\t/home/builder/temporal/internal/goro/group.go:58"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:43:42.868Z","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: pq: bind message has 6 result formats but query has 3 columns","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\t/home/builder/temporal/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).refreshLoop\n\t/home/builder/temporal/common/namespace/registry.go:357\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:39:44.009Z","msg":"Operation failed with internal error.","error":"GetTaskQueue operation failed. Failed to check if task queue temporal-sys-tq-scanner-taskqueue-0 of type Workflow existed. Error: pq: bind message supplies 2 parameters, but prepared statement \"\" requires 5","operation":"GetTaskQueue","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTaskQueue.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:567\ngo.temporal.io/server/common/persistence.(*taskPersistenceClient).GetTaskQueue\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:569\ngo.temporal.io/server/service/matching.(*taskQueueDB).getVersioningDataLocked\n\t/home/builder/temporal/service/matching/db.go:306\ngo.temporal.io/server/service/matching.(*taskQueueDB).GetVersioningData\n\t/home/builder/temporal/service/matching/db.go:291\ngo.temporal.io/server/service/matching.(*taskQueueManagerImpl).GetVersioningData\n\t/home/builder/temporal/service/matching/taskQueueManager.go:471\ngo.temporal.io/server/service/matching.(*matchingEngineImpl).GetTaskQueueMetadata\n\t/home/builder/temporal/service/matching/matchingEngine.go:803\ngo.temporal.io/server/service/matching.(*Handler).GetTaskQueueMetadata\n\t/home/builder/temporal/service/matching/handler.go:403\ngo.temporal.io/server/api/matchingservice/v1._MatchingService_GetTaskQueueMetadata_Handler.func1\n\t/home/builder/temporal/api/matchingservice/v1/service.pb.go:593\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept.func1\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:63\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/rpc/interceptor.(*RetryableInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/retry.go:67\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*RateLimitInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/rate_limit.go:86\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc/interceptor.(*TelemetryInterceptor).Intercept\n\t/home/builder/temporal/common/rpc/interceptor/telemetry.go:157\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/metrics.NewServerMetricsTrailerPropagatorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:113\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/metrics.NewServerMetricsContextInjectorInterceptor.func1\n\t/home/builder/temporal/common/metrics/grpc.go:66\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.UnaryServerInterceptor.func1\n\t/go/pkg/mod/go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v0.36.4/interceptor.go:341\ngoogle.golang.org/grpc.getChainUnaryHandler.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1164\ngo.temporal.io/server/common/rpc.ServiceErrorInterceptor\n\t/home/builder/temporal/common/rpc/grpc.go:137\ngoogle.golang.org/grpc.chainUnaryInterceptors.func1\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1155\ngo.temporal.io/server/api/matchingservice/v1._MatchingService_GetTaskQueueMetadata_Handler\n\t/home/builder/temporal/api/matchingservice/v1/service.pb.go:595\ngoogle.golang.org/grpc.(*Server).processUnaryRPC\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1345\ngoogle.golang.org/grpc.(*Server).handleStream\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:1722\ngoogle.golang.org/grpc.(*Server).serveStreams.func1.2\n\t/go/pkg/mod/google.golang.org/grpc@v1.54.0/server.go:966"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:50:42.848Z","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: pq: bind message has 6 result formats but query has 3 columns","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\t/home/builder/temporal/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).refreshLoop\n\t/home/builder/temporal/common/namespace/registry.go:357\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:56:42.869Z","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: pq: bind message has 6 result formats but query has 3 columns","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\t/home/builder/temporal/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).refreshLoop\n\t/home/builder/temporal/common/namespace/registry.go:357\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:57:42.871Z","msg":"Operation failed with internal error.","error":"ListNamespaces operation failed. Failed to get namespace rows. Error: pq: bind message has 6 result formats but query has 3 columns","operation":"ListNamespaces","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:683\ngo.temporal.io/server/common/persistence.(*metadataPersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:685\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:887\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*metadataRetryablePersistenceClient).ListNamespaces\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:891\ngo.temporal.io/server/common/namespace.(*registry).refreshNamespaces\n\t/home/builder/temporal/common/namespace/registry.go:386\ngo.temporal.io/server/common/namespace.(*registry).refreshLoop\n\t/home/builder/temporal/common/namespace/registry.go:357\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T13:58:43.325Z","msg":"Operation failed with internal error.","error":"UpsertClusterMembership operation failed. Error: pq: unnamed prepared statement does not exist","operation":"UpsertClusterMembership","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).UpsertClusterMembership.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1115\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).UpsertClusterMembership\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1117\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).UpsertClusterMembership.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:948\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).UpsertClusterMembership\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:951\ngo.temporal.io/server/common/membership.(*ringpopMonitor).upsertMyMembership\n\t/home/builder/temporal/common/membership/rpMonitor.go:190\ngo.temporal.io/server/common/membership.(*ringpopMonitor).startHeartbeatUpsertLoop.func1\n\t/home/builder/temporal/common/membership/rpMonitor.go:307"}
airbyte-temporal  | {"level":"error","ts":"2024-08-20T14:01:42.801Z","msg":"Operation failed with internal error.","error":"ListClusterMetadata operation failed. Failed to get cluster metadata rows. Error: pq: unnamed prepared statement does not exist","operation":"ListClusterMetadata","logging-call-at":"persistenceMetricClients.go:1171","stacktrace":"go.temporal.io/server/common/log.(*zapLogger).Error\n\t/home/builder/temporal/common/log/zap_logger.go:150\ngo.temporal.io/server/common/persistence.updateErrorMetric\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1171\ngo.temporal.io/server/common/persistence.(*metricEmitter).recordRequestMetrics\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1148\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).ListClusterMetadata.func1\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1040\ngo.temporal.io/server/common/persistence.(*clusterMetadataPersistenceClient).ListClusterMetadata\n\t/home/builder/temporal/common/persistence/persistenceMetricClients.go:1042\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).ListClusterMetadata.func1\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:972\ngo.temporal.io/server/common/backoff.ThrottleRetryContext\n\t/home/builder/temporal/common/backoff/retry.go:199\ngo.temporal.io/server/common/persistence.(*clusterMetadataRetryablePersistenceClient).ListClusterMetadata\n\t/home/builder/temporal/common/persistence/persistenceRetryableClients.go:976\ngo.temporal.io/server/common/cluster.(*metadataImpl).listAllClusterMetadataFromDB.func1\n\t/home/builder/temporal/common/cluster/metadata.go:558\ngo.temporal.io/server/common/collection.(*PagingIteratorImpl[...]).getNextPage\n\t/home/builder/temporal/common/collection/pagingIterator.go:116\ngo.temporal.io/server/common/collection.NewPagingIterator[...]\n\t/home/builder/temporal/common/collection/pagingIterator.go:52\ngo.temporal.io/server/common/cluster.(*metadataImpl).listAllClusterMetadataFromDB\n\t/home/builder/temporal/common/cluster/metadata.go:575\ngo.temporal.io/server/common/cluster.(*metadataImpl).refreshClusterMetadata\n\t/home/builder/temporal/common/cluster/metadata.go:442\ngo.temporal.io/server/common/cluster.(*metadataImpl).refreshLoop\n\t/home/builder/temporal/common/cluster/metadata.go:429\ngo.temporal.io/server/internal/goro.(*Handle).Go.func1\n\t/home/builder/temporal/internal/goro/goro.go:64"}
marcosmarxm commented 2 months ago

@benmoriceau can you help here?