kubeflow / examples

A repository to host extended examples and tutorials
Apache License 2.0
1.39k stars 751 forks source link

tensorflow/serving:1.15.0 saved_model_warmup.cc:105] No warmup data file found at s3 #1002

Open jacklu2016 opened 1 year ago

jacklu2016 commented 1 year ago

I have followed process in mnist_vanilla_k8s.ipynb , everything is ok. But when deploy serving model, i got errors

`2022-10-13 05:01:41.524629: I tensorflow_serving/servables/tensorflow/saved_model_warmup.cc:105] No warmup data file found at s3://lo-harbor-mnist/mnist/export/1665634166/assets.extra/tf_serving_warmup_requests
2022-10-13 05:01:41.524693: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:54] Found secret key
2022-10-13 05:01:41.524872: I external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:54] Connection has been released. Continuing.
2022-10-13 05:01:41.525581: E external/org_tensorflow/tensorflow/core/platform/s3/aws_logging.cc:60] HTTP response code: 404
Exception name: 
Error message: No response body.
8 response headers:
accept-ranges : bytes
content-length : 0
content-security-policy : block-all-mixed-content
date : Thu, 13 Oct 2022 05:01:41 GMT
server : MinIO/RELEASE.2019-08-14T20-37-41Z
vary : Origin
x-amz-request-id : 171D884D020B07AE
x-xss-protection : 1; mode=block`

I check s3 bucket, there is not exist warmup file mention above s3://lo-harbor-mnist/mnist/export/1665634166/assets.extra/tf_serving_warmup_requests

Environment: Kubenetes version:v1.21.14 kubeflow version:1.5.1 jupyter notebook version:public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:v1.5.0 mnist image version: tensorflow/tensorflow:1.15.2-py3 modle serving version:tensorflow/serving:1.15.0