Hydrospheredata / hydro-serving

MLOps Platform
http://docs.hydrosphere.io
Apache License 2.0
271 stars 42 forks source link

GRPC call error: #246

Closed albertnanda closed 5 years ago

albertnanda commented 5 years ago

Hi,

I am getting the following error while making a GRPC call: status = StatusCode.UNAVAILABLE details = "Trying to connect an http1.x server"

The application is working, I have tested it with hydroservingUI web method.

Thanks, AN

tidylobster commented 5 years ago

Hello, @albertnanda

Can you provide a code you're using to establish connection with gRPC endpoint?

albertnanda commented 5 years ago

@tidylobster Here is the code

import grpc import hydro_serving_grpc as hs import numpy as np

channel = grpc.insecure_channel("localhost:80") stub = hs.PredictionServiceStub(channel) model_spec = hs.ModelSpec(name="tokenize", signature_name="tokenize") text = "The quick brown fox jumps over the lazy dog" tensor_shape = hs.TensorShapeProto(dim=[hs.TensorShapeProto.Dim(size=-1)]) text = text.encode('utf-8') tensor = hs.TensorProto(dtype=hs.DT_STRING, tensor_shape=tensor_shape, string_val=[text]) request = hs.PredictRequest(model_spec=model_spec, inputs={"text": tensor}) result = stub.Predict(request) print(result)

albertnanda commented 5 years ago

This is the complete error message: Traceback (most recent call last): File "client.py", line 15, in result = stub.Predict(request) File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 550, in call return _end_unary_response_blocking(state, call, False, None) File "/home/ubuntu/anaconda3/lib/python3.6/site-packages/grpc/_channel.py", line 467, in _end_unary_response_blocking raise _Rendezvous(state, None, None, deadline) grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with: status = StatusCode.UNAVAILABLE details = "Trying to connect an http1.x server" debug_error_string = "{"created":"@1550587709.114299527","description":"Error received from peer","file":"src/core/lib/surface/call.cc","file_line":1036,"grpc_message":"Trying to connect an http1.x server","grpc_status":14}"

KineticCookie commented 5 years ago

Hi What service do you have on port 80? In our demo deployments envoy sidecar listens 80 port and redirects all GRPC traffic to the corresponding services. Looks like you have other service which listens this port. Or there is possibility that envoy encountered errors during initialization.

Could you please check that and provide the content of localhost:8082/clusters and localhost:8082/routes

albertnanda commented 5 years ago

@KineticCookie clusters.txt routes.txt

KineticCookie commented 5 years ago

Well, gateway::172.16.0.11:9091::rq_total::0 shows that GRPC requests land somewhere else. Could you please check your 80 port?

albertnanda commented 5 years ago

@KineticCookie sudo lsof -i :80 docker-pr 3565 root 4u IPv6 35035 0t0 TCP :http (LISTEN) sudo netstat -lnp | grep ':80 ' tcp6 0 0 :::80 ::: LISTEN 3565/docker-proxy

albertnanda commented 5 years ago

Deleted all the networks and config files, rebooted, seems to be working now