[Bug] Several services crashing when deploying v.0.24.0

[vishwa-trulioo]

What are you really trying to do?

I attempted updating the Temporal deployment from Helm chart version 0.22.1 to 0.24.0. Several of the services starts crashing.

Details: Current Helm chart version: v0.22.1 Expected version: v0.24.0 Storage backend: MySQL/Aurora

Error summary:

sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal". Expected version: 1.10 cannot be greater than Actual version: 1.9.

Kubernetes env details:

% kubectl --namespace=temporal get pods -l "app.kubernetes.io/instance=temporal"
NAME                                  READY   STATUS             RESTARTS        AGE
temporal-admintools-c5b5554f8-6hdm2   1/1     Running            0               11m
temporal-frontend-5dd4b6cd49-qrtkj    1/1     Running            0               17m
temporal-frontend-7895bcd6c6-fb7mn    0/1     CrashLoopBackOff   7 (4s ago)      11m
temporal-frontend-7895bcd6c6-npqsj    0/1     CrashLoopBackOff   7 (20s ago)     11m
temporal-history-774b8bf6b9-hfw9c     0/1     CrashLoopBackOff   2 (13s ago)     35s
temporal-history-774b8bf6b9-r6zqw     0/1     CrashLoopBackOff   7 (35s ago)     11m
temporal-history-fb6bb49cd-kkdvb      1/1     Running            0               17m
temporal-matching-65b59c5868-wtw9z    1/1     Running            0               17m
temporal-matching-758f4c7865-76ztd    0/1     CrashLoopBackOff   7 (25s ago)     11m
temporal-matching-758f4c7865-hmrt5    0/1     CrashLoopBackOff   5 (2m19s ago)   5m29s
temporal-web-956f445dd-tx6mg          1/1     Running            0               11m
temporal-worker-5d8dc58949-9clhc      1/1     Running            0               17m
temporal-worker-5d8dc58949-dhnvn      1/1     Running            0               17m
temporal-worker-d849c84bb-jqxbg       0/1     CrashLoopBackOff   7 (14s ago)     11m

Error details:

• All crashing pods show the an error similar to this,

  kubectl -n temporal logs temporal-frontend-7895bcd6c6-58mfg
  TEMPORAL_ADDRESS is not set, setting it to 10.96.36.97:7233
  2023/07/06 18:17:47 Loading config; env=docker,zone=,configDir=config
  2023/07/06 18:17:47 Loading config files=[config/docker.yaml]
  {"level":"info","ts":"2023-07-06T18:17:47.005Z","msg":"Build info.","git-time":"2023-07-01T00:28:10.000Z","git-revision":"2f4b63623bd895eeea98df72115cdb26b4939e39","git-modified":true,"go-arch":"amd64","go-os":"linux","go-version":"go1.20.5","cgo-enabled":false,"server-version":"1.21.1","debug-mode":false,"logging-call-at":"main.go:148"}
  {"level":"info","ts":"2023-07-06T18:17:47.006Z","msg":"Updated dynamic config","logging-call-at":"file_based_client.go:195"}
  {"level":"warn","ts":"2023-07-06T18:17:47.006Z","msg":"Not using any authorizer and flag `--allow-no-auth` not detected. Future versions will require using the flag `--allow-no-auth` if you do not want to set an authorizer.","logging-call-at":"main.go:178"}
  [Fx] PROVIDE    *pprof.PProfInitializerImpl <= go.temporal.io/server/common/pprof.NewInitializer()
  [Fx] PROVIDE    *temporal.ServerImpl <= go.temporal.io/server/temporal.NewServerFxImpl()
  [Fx] PROVIDE    *temporal.serverOptions <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    chan interface {} <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    temporal.synchronizationModeParams <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    *config.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    *config.PProf <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    log.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    resource.ServiceNames <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    resource.NamespaceLogger <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    resolver.ServiceResolver <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    client.AbstractDataStoreFactory <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    searchattribute.Mapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    []grpc.UnaryServerInterceptor <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    authorization.Authorizer <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    authorization.ClaimMapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    authorization.JWTAudienceMapper <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    log.Logger <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    client.FactoryProvider <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    dynamicconfig.Client <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    encryption.TLSConfigProvider <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    *client.Config <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    client.Client <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    metrics.Handler <= go.temporal.io/server/temporal.ServerOptionsProvider()
  [Fx] PROVIDE    []trace.SpanExporter <= go.temporal.io/server/temporal.glob..func2()
  [Fx] PROVIDE    client.FactoryProviderFn <= go.temporal.io/server/temporal.PersistenceFactoryProvider()
  [Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.HistoryServiceProvider()
  [Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.MatchingServiceProvider()
  [Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.FrontendServiceProvider()
  [Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.InternalFrontendServiceProvider()
  [Fx] PROVIDE    *temporal.ServicesMetadata[group = "services"] <= go.temporal.io/server/temporal.WorkerServiceProvider()
  [Fx] PROVIDE    *cluster.Config <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()
  [Fx] PROVIDE    config.Persistence <= go.temporal.io/server/temporal.ApplyClusterMetadataConfigProvider()
  [Fx] SUPPLY []temporal.ServerOption
  [Fx] PROVIDE    fx.Lifecycle <= go.uber.org/fx.New.func1()
  [Fx] PROVIDE    fx.Shutdowner <= go.uber.org/fx.(*App).shutdowner-fm()
  [Fx] PROVIDE    fx.DotGraph <= go.uber.org/fx.(*App).dotGraph-fm()
  [Fx] ERROR      Failed to initialize custom logger: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2
  /go/pkg/mod/go.uber.org/fx@v1.19.1/module.go:218:
  failed to build fxevent.Logger:
  could not build arguments for function "go.temporal.io/server/temporal".glob..func8
  /home/builder/temporal/temporal/fx.go:1004:
  failed to build log.Logger:
  received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider
  /home/builder/temporal/temporal/fx.go:166:
  sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal". Expected version: 1.10 cannot be greater than Actual version: 1.9
  Unable to create server. Error: could not build arguments for function "go.uber.org/fx".(*module).constructCustomLogger.func2 (/go/pkg/mod/go.uber.org/fx@v1.19.1/module.go:218): failed to build fxevent.Logger: could not build arguments for function "go.temporal.io/server/temporal".glob..func8 (/home/builder/temporal/temporal/fx.go:1004): failed to build log.Logger: received non-nil error from function "go.temporal.io/server/temporal".ServerOptionsProvider (/home/builder/temporal/temporal/fx.go:166): sql schema version compatibility check failed: version mismatch for keyspace/database: "temporal". Expected version: 1.10 cannot be greater than Actual version: 1.9.

Notes:

• I did verify that schema is updated to fit the required temporal version (v1.21.1)
• I even did a clean new install. It still produces the same errors.

Minimal Reproduction

Environment/Versions

• OS and processor: Linux
• Temporal Version: 1.21.1
• Hosted environment: Kubernetes/EKS

Additional context

• I did verify that schema is updated to fit the required temporal version (v1.21.1)
• I even did a clean new install. It still produces the same errors.

[vishwa-trulioo]

This may be related to the bug described in release 1.21.1. It says,

Do not upgrade to this version if you using advanced visibility with MySQL as storage. The release of v1.21.0 included an script to upgrade the schema, but it contains a bug. Please wait for the release of v1.21.2 which will fix it.

I will keep my eye on it to see if it fixes with 1.21.2 release and confirm if it automatically fixes.

[robholland]

Assuming this is fixed, please re-open if not.