search

temporalio / temporal

Temporal service
https://docs.temporal.io
MIT License
12.02k stars 850 forks source link

Temporal fails to connect to Google Cloud SQL Postgres when password contains certain characters #5729

Open NAjustin opened 7 months ago

NAjustin commented 7 months ago

Expected Behavior

Temporal should connect correctly regardless of what characters are contained in the password.

Actual Behavior

Temporal actually has mixed results when certain characters are present:

  1. The temporal-sql-tool appears to run correctly. For example, it creates the temporal and temporal_visibility databases, as well as creates/updates the schemas. This also proves that the password is, in fact, valid.
  2. But that's where success ends. Once the creation/update is handled, connections begin to error: sql schema version compatibility check failed: pq: password authentication failed for user "example-user"

Here's the full section of logs containing errors:

[Fx] ERROR  Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*App).constructCustomLogger.func2
/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:414:
failed to build fxevent.Logger:
could not build arguments for function "go.temporal.io/server/temporal".glob..func8
/home/builder/temporal/temporal/fx.go:1025:
failed to build log.Logger:
received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider
/home/builder/temporal/temporal/fx.go:159:
sql schema version compatibility check failed: pq: password authentication failed for user "example-user"
Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*App).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/fx@v1.18.2/app.go:414): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/home/builder/temporal/temporal/fx.go:1025): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:159): sql schema version compatibility check failed: pq: password authentication failed for user "example-user".

Steps to Reproduce the Problem

  1. Use special characters in your password. Verified (failed) example: y{0zzu2p3\t*az<g
  2. Deploy Temporal (I used Helm)
  3. Enjoy the errors

Changing the password to not include the above characters makes the identical setup work correctly (and also fixes the surrounding errors). I didn't have the time to hunt down the offending character sequence, but my money is on the \t getting turned into a tab somewhere in the execution (e.g. before or after the password gets turned into a DSN)—but because the temporal-sql-tool scripts run in the same deployment, I think it's safe to rule out environmental factors. It may just be that net/url.QueryEscape isn't the right tool for the job to always generate a valid DSN, or it may be somewhere else in that code flow.

(It doesn't appear that this code flow has changed in more recent versions than what's being supplied in my upstream.)

Specifications

jaylin-sf commented 4 months ago

I have the same issue with 1.24.2, the databases in PostgreSQL (Azure Flexible Server) have been created successfully. But it failed to build log.Logger for some reason.

Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/home/runner/go/pkg/mod/go.uber.org/fx@v1.21.1/module.go:292): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:1009): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/runner/work/docker-builds/docker-builds/temporal/temporal/fx.go:183): sql schema version compatibility check failed: pq: no pg_hba.conf entry for host "20.227.4.75", user "temporal_pg_test", database "temporal", no encryption.

BTW: I passed in CA file in environment variables with the latest docker image (temporalio/auto-setup): POSTGRES_TLS_ENABLED = true POSTGRES_TLS_CA_FILE = {path from container volume}

DB = postgres12

raymondregrello commented 2 months ago

Ran into the issue as well, I believe it'll get fixed by https://github.com/temporalio/helm-charts/pull/551