MySQL Domain cadence-system does not exist

[BrendanBall]

I'm trying to run a clean cadence setup with mysql using docker compose and cadence fails with the error Domain cadence-system does not exist. Some logs:

cadence_1      | {"level":"info","ts":"2019-11-05T05:38:43.585Z","msg":"none","service":"cadence-history","component":"shard-controller","lifecycle":"Started","address":"172.19.0.4:7934","logging-call-at":"shardController.go:159"}
cadence_1      | {"level":"info","ts":"2019-11-05T05:38:43.585Z","msg":"started","service":"cadence-history","service":"cadence-history","logging-call-at":"service.go:419"}
cadence_1      | {"level":"info","ts":"2019-11-05T05:38:43.585Z","msg":"Get dynamic config","name":"history.acquireShardInterval","value":"1m0s","default-value":"1m0s","logging-call-at":"config.go:58"}
cadence_1      | {"level":"fatal","ts":"2019-11-05T05:38:54.469Z","msg":"failed to verify if cadence system domain exists","service":"cadence-worker","error":"EntityNotExistsError{Message: Domain cadence-system does not exist.}","logging-call-at":"service.go:349","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Fatal\n\t/cadence/common/log/loggerimpl/logger.go:140\ngithub.com/uber/cadence/service/worker.(*Service).ensureSystemDomainExists\n\t/cadence/service/worker/service.go:349\ngithub.com/uber/cadence/service/worker.(*Service).Start\n\t/cadence/service/worker/service.go:158\nmain.execute\n\t/cadence/cmd/server/server.go:226"}

docker-compose.yaml

I made sure to pull latest ubercadence/server:master-auto-setup

[wxing1292]

thanks for reporting this. fix #2770 will solve this

[wxing1292]

@BrendanBall merged to master, plz take another try. if works, plz close the issue

[BrendanBall]

@wxing1292 I pulled latest ubercadence/server:master-auto-setup but it's still giving the same error. Is it working when you run it on your machine?

[wxing1292]

@BrendanBall yes i tested before sending out the PR. can you try removing the docker image locally and pull again?

[wxing1292]

@BrendanBall actually, the merging to master did not correctly triggered the build & push to docker hub, give me a sec

[wxing1292]

@BrendanBall plz take another try

[BrendanBall]

Thanks, that seems to be working again now. It starts up and I tested creating a test workflow and that seems to be working. The only thing I noticed is getting this log a lot while cadence is starting up, but it's either normal or a different issue:

{"level":"error","ts":"2019-11-08T13:57:09.653Z","msg":"Internal service error","service":"cadence-frontend","error":"InternalServiceError{Message: Not enough hosts to serve the request}","logging-call-at":"workflowHandler.go:3073","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(*loggerImpl).Error\n\t/cadence/common/log/loggerimpl/logger.go:134\ngithub.com/uber/cadence/service/frontend.(*WorkflowHandler).error\n\t/cadence/service/frontend/workflowHandler.go:3073\ngithub.com/uber/cadence/service/frontend.(*WorkflowHandler).StartWorkflowExecution\n\t/cadence/service/frontend/workflowHandler.go:1697\ngithub.com/uber/cadence/service/frontend.(*DCRedirectionHandlerImpl).StartWorkflowExecution.func2\n\t/cadence/service/frontend/dcRedirectionHandler.go:1103\ngithub.com/uber/cadence/service/frontend.(*NoopRedirectionPolicy).WithDomainNameRedirect\n\t/cadence/service/frontend/dcRedirectionPolicy.go:112\ngithub.com/uber/cadence/service/frontend.(*DCRedirectionHandlerImpl).StartWorkflowExecution\n\t/cadence/service/frontend/dcRedirectionHandler.go:1099\ngithub.com/uber/cadence/.gen/go/cadence/workflowserviceserver.handler.StartWorkflowExecution\n\t/cadence/.gen/go/cadence/workflowserviceserver/server.go:1374\ngo.uber.org/yarpc/encoding/thrift.thriftUnaryHandler.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/encoding/thrift/inbound.go:61\ngo.uber.org/yarpc/internal/observability.(*Middleware).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/internal/observability/middleware.go:141\ngo.uber.org/yarpc/api/middleware.unaryHandlerWithMiddleware.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/tchannel.handler.callHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/transport/tchannel/handler.go:215\ngo.uber.org/yarpc/transport/tchannel.handler.handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/transport/tchannel/handler.go:118\ngo.uber.org/yarpc/transport/tchannel.handler.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/transport/tchannel/handler.go:107\ngithub.com/uber/tchannel-go.channelHandler.Handle\n\t/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/handlers.go:126\ngithub.com/uber/tchannel-go.(*Connection).dispatchInbound\n\t/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/inbound.go:203"}

[wxing1292]

Not enough hosts to serve the request is pretty common during service startup, since ringpop (the DHT lib) cannot find the corresponding host to serve API calls, plz just ignore