search

Hydrospheredata / hydro-serving

MLOps Platform
http://docs.hydrosphere.io
Apache License 2.0
271 stars 42 forks source link

gRPC high workload handling #285

Closed tidylobster closed 3 years ago

tidylobster commented 5 years ago

With the recent changes on the manager component applications seem to handle high traffic workload on gRPC interface poorly. A connection gets closed periodically, but then re-establishes very quickly. See error log below: 


  File "simulation.py", line 81, in <module>
    simulate_production_traffic(request_delay=0.5)
  File "simulation.py", line 66, in simulate_production_traffic
    result = stub.Predict(request)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 562, in __call__
    return _end_unary_response_blocking(state, call, False, None)
  File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/grpc/_channel.py", line 466, in _end_unary_response_blocking
    raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "Socket closed"
        debug_error_string = "{"created":"@1563195018.063527000","description":"Error received from peer ipv4:18.184.72.114:443","file":"src/core/lib/surface/call.cc","file_line":1041,"grpc_message":"Socket closed","grpc_status":14}"```
stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.