search

Hydrospheredata / hydro-serving

MLOps Platform
http://docs.hydrosphere.io
Apache License 2.0
271 stars 42 forks source link

Model version number stuck on `2`. #161

Closed IceKhan13 closed 6 years ago

IceKhan13 commented 6 years ago

Model version number stuck on 2.

dopeshot

Infrastructure data

Manager information

  1. Manager version, or commit SHA.
  2. Manager logs. Second upload of model 
    [INFO ] i.h.s.m.s.m.b.InfoProgressHandler$ i.h.s.m.s.m.b.InfoProgressHandler$.handle.37 ProgressMessage(null,null, ---> 5608e9099050
,null,null,None)
[INFO ] i.h.s.m.s.m.b.InfoProgressHandler$ i.h.s.m.s.m.b.InfoProgressHandler$.handle.37 ProgressMessage(null,null,null,null,null,None)
[INFO ] i.h.s.m.s.m.b.InfoProgressHandler$ i.h.s.m.s.m.b.InfoProgressHandler$.handle.37 ProgressMessage(null,null,Successfully built 5608e9099050
,null,null,None)
[INFO ] i.h.s.m.s.m.b.InfoProgressHandler$ i.h.s.m.s.m.b.InfoProgressHandler$.handle.37 ProgressMessage(null,null,Successfully tagged ks-test:2
,null,null,None)

    Third upload

    [INFO ] i.h.s.m.s.m.b.InfoProgressHandler$ i.h.s.m.s.m.b.InfoProgressHandler$.handle.37 ProgressMessage(null,null,null,null,null,None)
[INFO ] i.h.s.m.s.m.b.InfoProgressHandler$ i.h.s.m.s.m.b.InfoProgressHandler$.handle.37 ProgressMessage(null,null,Successfully built fcfe8947eb60
,null,null,None)
[INFO ] i.h.s.m.s.m.b.InfoProgressHandler$ i.h.s.m.s.m.b.InfoProgressHandler$.handle.37 ProgressMessage(null,null,Successfully tagged ks-test:2
,null,null,None)
  3. Manager deployment command. hs upload --host localhost --port 9090
  4. Manager configuration. default

Sidecar information

  1. Sidecar version or commit SHA. hydrosphere/serving-sidecar:latest
  2. Sidecar logs. 
    
[2018-06-08 08:40:55.761][1][info][main] source/server/drain_manager_impl.cc:63] shutting down parent after drain
[2018-06-08 08:42:18.424][1][warning][upstream] source/common/config/grpc_mux_impl.cc:205] gRPC config stream closed: 13, 
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.RouteConfiguration failed
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.Listener failed
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment failed
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment failed
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment failed
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment failed
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.ClusterLoadAssignment failed
[2018-06-08 08:42:18.424][1][warning][config] bazel-out/k8-opt/bin/source/common/config/_virtual_includes/grpc_mux_subscription_lib/common/config/grpc_mux_subscription_impl.h:66] gRPC update for type.googleapis.com/envoy.api.v2.Cluster failed
[2018-06-08 08:48:12.934][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:388] add/update cluster r3m4e0 starting warming
[2018-06-08 08:48:12.936][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:395] warming cluster r3m4e0 complete
[2018-06-08 08:50:51.626][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:437] removing cluster r3m4e0
[2018-06-08 08:51:08.379][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:388] add/update cluster r3m5e0 starting warming
[2018-06-08 08:51:08.382][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:395] warming cluster r3m5e0 complete
[2018-06-08 08:56:18.631][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:437] removing cluster r3m5e0
[2018-06-08 08:56:31.216][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:388] add/update cluster r3m6e0 starting warming
[2018-06-08 08:56:31.218][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:395] warming cluster r3m6e0 complete
[2018-06-08 08:57:12.003][1][info][upstream] source/common/upstream/cluster_manager_impl.cc:437] removing cluster r3m6e0
3. Sidecar deployment command.
4. Sidecar configuration.

docker run -e MANAGER_HOST=$HOST_IP -e HOST_PRIVATE_IP=$HOST_IP \ -e MANAGER_PORT=9091 \ -e SERVICE_ID=-20 \ -e SERVICE_NAME="manager" \ -p 8080:8080 -p 8081:8081 -p 8082:8082 \ hydrosphere/serving-sidecar:latest


## ML data
### Runtime
1. Runtime image (if you got it from public sources)

{ "name": "hydrosphere/serving-runtime-python", "version": "3.6-latest", "modelTypes": [ "string" ], "tags": [ "string" ], "configParams": {} }

2. Repository with commit SHA or release tag (if you built it yourself)
3. Runtime logs.

INFO:PythonRuntimeService:Received inference request: model_spec { name: "KblK" signature_name: "ks_test" } inputs { key: "distribution" value { dtype: DT_STRING string_val: "normal" } } inputs { key: "distributionSample" value { dtype: DT_DOUBLE tensor_shape { dim { size: -1 } } double_val: 1.0 } } inputs { key: "sample" value { dtype: DT_DOUBLE tensor_shape { dim { size: -1 } } double_val: 1.0 } }

INFO:PythonRuntimeService:Answer: outputs { key: "ksStatistics" value { dtype: DT_DOUBLE double_val: 0.895 } } outputs { key: "pValue" value { dtype: DT_DOUBLE double_val: 0.17860426969764479 } } outputs { key: "rejectionLevel" value { dtype: DT_DOUBLE double_val: 1.3633957605919127 } }

KineticCookie commented 6 years ago

Note: Not reproducible on current test models