search

Hydrospheredata / hydro-serving

MLOps Platform
http://docs.hydrosphere.io
Apache License 2.0
271 stars 42 forks source link

getting this error when trying to run application status: 500, error: InternalUncaught, message: UNAVAILABLE: HTTP status code 503 invalid content-type: null trailers: #236

Closed albertnanda closed 5 years ago

albertnanda commented 5 years ago

Hi,

I am getting this error when I am trying to run an application with a custom runtime.

status: 500, error: InternalUncaught, message: UNAVAILABLE: HTTP status code 503 invalid content-type: null trailers:

albertnanda commented 5 years ago

This is the only log i could find out from the container - hydrosphere/serving-gateway:latest

ERROR[scala-execution-context-global-151] 2019-01-22 07:01:00,269 ApplicationExecutionServiceImpl.$anonfun$serve$3.212 Request to model failed io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 503 invalid content-type: null trailers: Metadata(:status=503,date=Tue, 22 Jan 2019 07:01:00 GMT,server=envoy) at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:467) ~[grpc-stub-1.11.0.jar:1.11.0] at io.hydrosphere.serving.grpc.HeaderClientServerInterceptor$ContextualizedClientCallListener.onClose(HeaderClientServerInterceptor.scala:57) ~[serving-grpc-scala_2.12-0.1.22.jar:0.1.22] at io.hydrosphere.serving.grpc.HeaderClientServerInterceptor$ContextualizedClientCallListener.onClose(HeaderClientServerInterceptor.scala:57) ~[serving-grpc-scala_2.12-0.1.22.jar:0.1.22] at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:37) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:37) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:391) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:471) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:553) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:474) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:591) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.11.0.jar:1.11.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_151] ERROR[hydroserving-gateway-akka.actor.default-dispatcher-13] 2019-01-22 07:01:00,271 HttpApi.applyOrElse.30 UNAVAILABLE: HTTP status code 503 invalid content-type: null trailers: Metadata(:status=503,date=Tue, 22 Jan 2019 07:01:00 GMT,server=envoy) io.grpc.StatusRuntimeException: UNAVAILABLE: HTTP status code 503 invalid content-type: null trailers: Metadata(:status=503,date=Tue, 22 Jan 2019 07:01:00 GMT,server=envoy) at io.grpc.Status.asRuntimeException(Status.java:526) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:467) ~[grpc-stub-1.11.0.jar:1.11.0] at io.hydrosphere.serving.grpc.HeaderClientServerInterceptor$ContextualizedClientCallListener.onClose(HeaderClientServerInterceptor.scala:57) ~[serving-grpc-scala_2.12-0.1.22.jar:0.1.22] at io.hydrosphere.serving.grpc.HeaderClientServerInterceptor$ContextualizedClientCallListener.onClose(HeaderClientServerInterceptor.scala:57) ~[serving-grpc-scala_2.12-0.1.22.jar:0.1.22] at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:37) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:37) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:391) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:471) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:553) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:474) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:591) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[grpc-core-1.11.0.jar:1.11.0] at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) ~[grpc-core-1.11.0.jar:1.11.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_151] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_151] at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_151]

KineticCookie commented 5 years ago

Hi. Could you please attach results of:

  1. curl localhost:8082/routes
  2. curl localhost:8082/clusters
albertnanda commented 5 years ago

Hi,

Here are the log files: routes.txt clusters.txt

KineticCookie commented 5 years ago

Looks like the manager service failed to update routing table for application. Could you please send manager logs (container with hydrosphere/serving-manager:latest image), and provide some additional info:

  1. Do you use docker-compose or k8s?
  2. How did you deploy the hydro-serving? Do you use our templates, or you made custom one?
ravimadhusudhan8 commented 5 years ago

And it happens with all models/services.

  1. We use docker-compose /integrations
  2. With both your templates and runtimes we created
albertnanda commented 5 years ago

@KineticCookie : Facing the same issue while testing with a large number of requests. A few services are giving this error.

KineticCookie commented 5 years ago

Issue is not actual since we removed Envoy in the newest release.