search

aws / amazon-sagemaker-examples

Example 📓 Jupyter notebooks that demonstrate how to build, train, and deploy machine learning models using 🧠 Amazon SageMaker.
https://sagemaker-examples.readthedocs.io
Apache License 2.0
10.1k stars 6.76k forks source link

[Bug Report] Edge Manager notebook errors loading darknet model #2179

Open mwunderl opened 3 years ago

mwunderl commented 3 years ago

Link to the notebook https://github.com/aws/amazon-sagemaker-examples/blob/master/sagemaker_edge_manager/sagemaker_edge_example/sagemaker_edge_example.ipynb

Describe the bug Edge agent fails to load provided darknet model (yolov3-tiny). I tried on both the AMI used in the notebook (ami-0bc87a16c757a7f07) and the latest version (ami-0f9543706892e0363). The notebook has steps for two models. The keras model works fine. But if you load the darknet model there is an unclear error. After encountering this error, further errors appear if try to list models or use the keras model that was previously working. This appears to be due to the agent trying to get a list of models before doing other operations.

[libprotobuf ERROR /codebuild/output/src790744299/src/NeoAgentSmith/third-party/grpc/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when serializing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.

To Reproduce

  1. Compile and package models per the instructions in the notebook.
  2. Create certificates, roles, etc.
  3. Prepare a directory of all of the demo files and upload to s3.
  4. Launch an EC2 instance as instructed and connect with SSH.

  5. Download the files and run the agent

    aws s3 sync s3://sagemaker-artifacts-XXXXXXX/em-demo/ demo/
cd demo
chmod 755 bin/*
chmod 400 certificates/us-west-2.pem
./bin/sagemaker_edge_agent_binary -a /tmp/sagemaker_edge_agent_example.sock -c sagemaker_edge_config.json

  6. Connect with another SSH session and run the client commands with keras.

    cd demo
./bin/sagemaker_edge_agent_client_example LoadModel keras_model demo-keras
Model demo-keras located at /home/ubuntu/demo/keras_model loaded
LoadModel succeeded

./bin/sagemaker_edge_agent_client_example Predict demo-keras keras.bmp input_1 224 224 3
Done reading the image
Predict succeeded
Flattened RAW Output Tensor:1
9.15062e-05 ...

./bin/sagemaker_edge_agent_client_example PredictAndCapture demo-keras keras.bmp input_1 224 224 3
Done reading the image
Predict succeeded
Capture Data succeeded
Flattened RAW Output Tensor:1
9.15062e-05 ...

./bin/sagemaker_edge_agent_client_example ListModels
There are 1 models
Model 0  demo-keras
ListModels succeeded

./bin/sagemaker_edge_agent_client_example UnloadModel demo-keras
Model demo-keras has been unloaded
UnLoadModel succeeded

  7. Try with darknet

    ./bin/sagemaker_edge_agent_client_example LoadModel darknet_model demo-darknet
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
LoadModel failed

  8. Try previously working commands

    ./bin/sagemaker_edge_agent_client_example LoadModel keras_model demo-keras
Model demo-keras located at /home/ubuntu/demo/keras_model loaded
LoadModel succeeded

./bin/sagemaker_edge_agent_client_example ListModels
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
ListModels failed

./bin/sagemaker_edge_agent_client_example Predict demo-keras keras.bmp input_1 224 224 3
[libprotobuf ERROR _deps/grpc-src/third_party/protobuf/src/google/protobuf/wire_format_lite.cc:577] String field 'AWS.SageMaker.Edge.TensorMetadata.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
Model demo-keras has not been loaded

20210430-edgeagent.txt

ngluna commented 3 years ago

What's the date of release binary artifact that you downloaded?

mwunderl commented 3 years ago

I used 1.20210305.a4bc999

ngluna commented 3 years ago

1.20201218.81f481f works just fine, but I also had issues loading models using the agent in 1.20210305.a4bc999.

In my case, I couldn't load either darknet or keras model.