Temporal v1.23.0 fails to start with error: Unable to create SQL database, but v1.20.1 works fine.

ichait-bgl commented 4 months ago

Helm Chart Version

0.60.1

What step the error happened?

On deploy

Relevant information

The issue is with the new temporal image temporalio/auto-setup:1.23.0: when deploying to EKS (with AWS RDS as host), temporal fails to connect with that error seen above. Changing the image on the fly of the pod to the old image 1.20.1, temporal connects to the DB properly and does its job fine. => it seems that either the new image has an issue, or the helm chart is missing something to account for changes in the new image ?

Relevant log output

ERROR   Unable to create SQL database.  {"error": "unable to connect to DB, tried default DB names: postgres,defaultdb, errors: [pq: no pg_hba.conf entry for host \"10.111.68.17\", user \"airbyte\", database \"postgres\", no encryption pq: no pg_hba.conf entry for host \"10.111.68.17\", user \"airbyte\", database \"defaultdb\", no encryption]", "logging-call-at": "handler.go:94"}

marcosmarxm commented 4 months ago

Can you provide steps of what are you trying to run? Are you trying to connect your deployment to an external database or are you running the default helm install airbyte?

ichait-bgl commented 4 months ago

Thanks @marcosmarxm!

We receive the error as part of installing/upgrading the airbyte helm chart. We are connecting to an external AWS RDS Postgres instance. It should be noted that only Temporal has the issue connecting to the db. The other airbyte modules connect fine.

As an aside, another user also confirmed that downgrading Temporal to previous version fixed the issue ... Airbyte-slack

Please let me know if you need any further info.

marcosmarxm commented 4 months ago

Thanks for the info @ichait-bgl I added the issue to the platform team backlog. Let's see if this impact more users to get a signal is something wrong.

jkleinkauff commented 2 months ago

@marcosmarxm we are facing the same issue after trying the upgrade from 0.54.15 to 0.293.4 (chart version). Temporal section in values.yaml was left untouched:

temporal:
  replicaCount: 1
  extraEnv:
    - name: "SQL_TLS" 
      value: "true"
    - name: "SQL_TLS_DISABLE_HOST_VERIFICATION"
      value: "true"
    - name: "SQL_TLS_ENABLED"
      value: "true"
    - name: "SQL_HOST_VERIFICATION" 
      value: "false"
    - name: SKIP_DB_CREATE
      value: "true"
    - name: DBNAME
      value: temporal_db
    - name: VISIBILITY_DBNAME
      value: temporal_visibility_db

│ 2024-07-17T12:48:32.390Z    ERROR    Unable to create SQL database.    {"error": "unable to connect to DB, tried default DB names: postgres,defaultdb, errors: [pq: no pg_hba.conf entry for host \"191.xxx.9 │
│ 2024/07/17 12:48:32 Loading config; env=docker,zone=,configDir=config                                                                                                                                         │
│ 2024/07/17 12:48:32 Loading config files=[config/docker.yaml]

Tail:

│ [Fx] Error returned: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider                                                                                              │
│     /home/builder/temporal/temporal/fx.go:180:                                                                                                                                                                │
│ sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal_db". Expected version: 1.11 cannot be greater than Actual version: 1.9                                       │
│ [Fx] ERROR        Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2                                                           │
│     /go/pkg/mod/go.uber.org/fx@v1.20.0/module.go:251:                                                                                                                                                         │
│ failed to build fxevent.Logger:                                                                                                                                                                               │
│ could not build arguments for function "go.temporal.io/server/temporal".init.func8

marcosmarxm commented 2 months ago

@airbytehq/platform-deployments is there any change in the temporal value chart part?

grozwalker commented 2 months ago

Try this extra envs

temporal:
  extraEnv:
    - name: SQL_TLS_ENABLED
      value: "true"
    - name: POSTGRES_TLS_ENABLED
      value: "true"
    - name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
      value: "true"

it solve my problem and maybe it helps you (maybe you need extra envs like DBNAME for your case)

XIAZY commented 1 month ago

Try this extra envs
temporal:
  extraEnv:
    - name: SQL_TLS_ENABLED
      value: "true"
    - name: POSTGRES_TLS_ENABLED
      value: "true"
    - name: POSTGRES_TLS_DISABLE_HOST_VERIFICATION
      value: "true"
it solve my problem and maybe it helps you (maybe you need extra envs like DBNAME for your case)

Thanks @grozwalker , can confirm it solved the problem for me as well. Just a quick question @marcosmarxm if you don't mind, are users supposed to use these extraEnvs? Disabling host verification sounds a bit unsafe to me

airbytehq / airbyte