search

uber / cadence

Cadence is a distributed, scalable, durable, and highly available orchestration engine to execute asynchronous long-running business logic in a scalable and resilient way.
https://cadenceworkflow.io
MIT License
8.18k stars 790 forks source link

cadence service with external cassandra throwing error-Not enough hosts to serve the request #2999

Closed amitesh176 closed 4 years ago

amitesh176 commented 4 years ago

I am facing below issue while starting cadence service with external cassandra.

Below are the commands i fired to start Cadence server ,Cadence-wen and statsd server. Cassandra is running on localhost(Windows 10),I am using Windows docker desktop.

docker run --name cadence --network app-tier -p 7933:7933 -p 7934:7934 -p 7935:7935 -p 7939:7939 -e CASSANDRA_SEEDS=host.docker.internal -e KEYSPACE=cadence -e VISIBILITY_KEYSPACE=cadence_visibility -e SKIP_SCHEMA_SETUP=TRUE -e RINGPOP_SEEDS=host.docker.internal -e STATSD_ENDPOINT=host.docker.internal:8125 -e NUM_HISTORY_SHARDS=1024 -e LOG_LEVEL=info ubercadence/server:master-auto-setup

docker run --name cadence-web -p 8088:8088 -d --link cadence -e CADENCE_TCHANNEL_SERVICE=cadence-frontend -e CADENCE_WEB_PORT=8088 -e CADENCE_TCHANNEL_PEERS=192.168.65.2:7933 ubercadence/web:3.4.1

docker run --name statsd --network app-tier -p 8080:80 -p 2003:2003 -p 8125:8125 -p 8126:8126 -d graphiteapp/graphite-statsd

Cadence server is Up and running ,Cadence-web UI is accessible. BUT while running any workflow i am getting exception saying-

16:36:19.137 [main] WARN c.u.cadence.internal.common.Retryer - Retrying after failure org.apache.thrift.TException: Rpc error:<ErrorResponse id=1 errorType=Timeout message=Request timeout after 1014ms> at com.uber.cadence.serviceclient.WorkflowServiceTChannel.throwOnRpcError(WorkflowServiceTChannel.java:513) at com.uber.cadence.serviceclient.WorkflowServiceTChannel.doRemoteCall(WorkflowServiceTChannel.java:488) at com.uber.cadence.serviceclient.WorkflowServiceTChannel.startWorkflowExecution(WorkflowServiceTChannel.java:773) at com.uber.cadence.serviceclient.WorkflowServiceTChannel.lambda$StartWorkflowExecution$7(WorkflowServiceTChannel.java:761) at com.uber.cadence.serviceclient.WorkflowServiceTChannel.measureRemoteCall(WorkflowServiceTChannel.java:533) at com.uber.cadence.serviceclient.WorkflowServiceTChannel.StartWorkflowExecution(WorkflowServiceTChannel.java:760) at com.uber.cadence.internal.external.GenericWorkflowClientExternalImpl.lambda$startWorkflowInternal$0(GenericWorkflowClientExternalImpl.java:134) at com.uber.cadence.internal.common.Retryer.retryWithResult(Retryer.java:122) at com.uber.cadence.internal.external.GenericWorkflowClientExternalImpl.startWorkflowInternal(GenericWorkflowClientExternalImpl.java:132) at com.uber.cadence.internal.external.GenericWorkflowClientExternalImpl.startWorkflow(GenericWorkflowClientExternalImpl.java:77) at com.uber.cadence.internal.sync.WorkflowStubImpl.startWithOptions(WorkflowStubImpl.java:116) at com.uber.cadence.internal.sync.WorkflowStubImpl.start(WorkflowStubImpl.java:180) at com.uber.cadence.internal.sync.WorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:175) at com.uber.cadence.internal.sync.WorkflowInvocationHandler.access$400(WorkflowInvocationHandler.java:46) at com.uber.cadence.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:286) at com.uber.cadence.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:246) at com.uber.cadence.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:162) at com.sun.proxy.$Proxy3.getGreeting(Unknown Source) at com.uber.cadence.samples.hello.HelloActivity.main(HelloActivity.java:92)

Below error i am getting while starting cadence-server.

{"level":"error","ts":"2020-01-24T10:56:17.380Z","msg":"Internal service error","service":"cadence-frontend","error":"InternalServiceError{Message: Not enough hosts to serve the request} ","logging-call-at":"workflowHandler.go:3143","stacktrace":"github.com/uber/cadence/common/log/loggerimpl.(loggerImpl).Error\n\t/cadence/common/log/loggerimpl/logger.go:134\ngithub.com/ uber/cadence/service/frontend.(WorkflowHandler).error\n\t/cadence/service/frontend/workflowHandler.go:3143\ngithub.com/uber/cadence/service/frontend.(WorkflowHandler).PollForActivityTa sk\n\t/cadence/service/frontend/workflowHandler.go:429\ngithub.com/uber/cadence/service/frontend.(DCRedirectionHandlerImpl).PollForActivityTask.func2\n\t/cadence/service/frontend/dcRedi rectionHandler.go:489\ngithub.com/uber/cadence/service/frontend.(NoopRedirectionPolicy).WithDomainNameRedirect\n\t/cadence/service/frontend/dcRedirectionPolicy.go:112\ngithub.com/uber/c adence/service/frontend.(DCRedirectionHandlerImpl).PollForActivityTask\n\t/cadence/service/frontend/dcRedirectionHandler.go:485\ngithub.com/uber/cadence/service/frontend.(AccessControl ledWorkflowHandler).PollForActivityTask\n\t/cadence/service/frontend/accessControlledHandler.go:347\ngithub.com/uber/cadence/.gen/go/cadence/workflowserviceserver.handler.PollForActivity Task\n\t/cadence/.gen/go/cadence/workflowserviceserver/server.go:938\ngo.uber.org/yarpc/encoding/thrift.thriftUnaryHandler.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/encoding/thrift /inbound.go:61\ngo.uber.org/yarpc/internal/observability.(Middleware).Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/internal/observability/middleware.go:141\ngo.uber.org/yarpc/api/mid dleware.unaryHandlerWithMiddleware.Handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/api/middleware/inbound.go:71\ngo.uber.org/yarpc/api/transport.InvokeUnaryHandler\n\t/go/pkg/mod/go.uber .org/yarpc@v1.42.0/api/transport/handler_invoker.go:70\ngo.uber.org/yarpc/transport/tchannel.handler.callHandler\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/transport/tchannel/handler.go:21 5\ngo.uber.org/yarpc/transport/tchannel.handler.handle\n\t/go/pkg/mod/go.uber.org/yarpc@v1.42.0/transport/tchannel/handler.go:118\ngo.uber.org/yarpc/transport/tchannel.handler.Handle\n\t /go/pkg/mod/go.uber.org/yarpc@v1.42.0/transport/tchannel/handler.go:107\ngithub.com/uber/tchannel-go.channelHandler.Handle\n\t/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/handlers.go: 126\ngithub.com/uber/tchannel-go.(*Connection).dispatchInbound\n\t/go/pkg/mod/github.com/uber/tchannel-go@v1.16.0/inbound.go:203"} {

meiliang86 commented 4 years ago

This sounds like some setup problem rather than code issue. Please seek help in the Slack chat room.