Set up object detection ML server externally.

[danellecline]

This task is to setup an object detection server externally.

This is relatively simple to do in AWS.

Reference:

yolov5 inference

IMPORTANT. This was done in our new AWS sub account by setting up credentials through:

https://mbari.awsapps.com/start

using an mbari username, e.g. duane and an mbari password.

Keys can be setup either through environment variables, e.g. AWS_REGION, AWS_SECRET_ACCESS_KEY, etc. or in a ~/.aws/credentials file.

[danellecline]

Step 1. Export your model to tensorflow saved model format.

@lonnylundsten you can do this in your Colab notebook with

python export.py --weights best.pt --include saved_model --conf-thres 0.25 --iou-thres 0.45 --nms

This can also be done with the ultralytics/yolov5 docker image.

Saved Model format encapsulates the whole network topology and weights. There is no need to have any python code sitting along side the model weights.

[danellecline]

Step 2. Package that model as required for serving through an endpoint and upload it to a bucket

mkdir -p export/Servo
mv best_saved_model export/Servo/1
tar -czvf model.tar.gz export/
aws s3 cp model.tar.gz s3://902005-models/20221028T054534Z-model.tar.gz

[lonnylundsten]

I'm assuming that these are instructions and that this isn't a task assigned to me? It's not clear but I guess I'm wondering if this is needed that ya'll can take care of it, right?

Hi @lonnylundsten - that's right, no action needed on your part. I'm handling this task. Just including you for reference as I'm unclear if people who are tagged in the instructions and not assigned get a notification.

[lonnylundsten]

OK. Perfect. I wasn't certain if tagging me == assigning task or not. Thanks!

[danellecline]

Step 3. Setup the endpoint with a few lines of python code. Let's call this mbari315k, and put it is on a ml.m5.xlarge instance with runs 0.24 cents/hour for evaluation. This is not a GPU, but it could be moved to a GPU instance if there is a need. @duane-edgington may have some thoughts on better instances. I'm following the blog instructions here.

pip install sagemaker

import os
import tensorflow as tf
from tensorflow.keras import backend
from sagemaker.tensorflow import TensorFlowModel

model_data = 's3://902005-models/20221028T054534Z-model.tar.gz'
role = 'arn:aws:iam::168552337187:role/DeepSeaAI'

model = TensorFlowModel(model_data=model_data, framework_version='2.8', role=role)

INSTANCE_TYPE = 'ml.m5.xlarge'
ENDPOINT_NAME = 'mbari315k'

predictor = model.deploy(initial_instance_count=1,
                            instance_type=INSTANCE_TYPE,
                            endpoint_name=ENDPOINT_NAME)

After a few minutes, the endpoint should now be visible 🥳 💯. Check out the endpoint through the console at 902005_vaa endpoints

[danellecline]

Step 4. Test the endpoint with a empty image

import numpy as np
import json
import boto3
import json

ENDPOINT_NAME = 'mbari315k'

model_height, model_width = 640, 640
blank_image = np.zeros((model_height, model_width,3), np.uint8)
data = np.array(blank_image.astype(np.float32)/255.)
payload = json.dumps([data.tolist()])

client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                   ContentType='application/json',
                                   Body=payload)

result = json.loads(response['Body'].read().decode())
print('Results: ', result)

which should output something like

Results:  {'predictions': [{'output_0': [[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0,
0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0,
0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]], 'output_1': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'output_2': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'output_3': 0}]}

[danellecline]

To test with a PNG image, similar to what we capture and download via VARS:

import numpy as np
import json
import boto3
import json
from PIL import Image
from numpy import asarray

ENDPOINT_NAME = 'mbari315k'
model_height, model_width = 640, 640

test_image = '/Volumes/Tempbox/Lonny/benthic_test_images/20191104T235504Z--9faf7c89-2fef-4a13-bf8b-dc79ceb3f981.png'

rgba_image = Image.open(test_image)
rgb_image = rgba_image.convert('RGB').resize((model_width,model_height))
data = np.array(np.array(rgb_image).astype(np.float32))

payload = json.dumps([data.tolist()])

client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                   ContentType='application/json',
                                   Body=payload)

result = json.loads(response['Body'].read().decode())
print('Results: ', result)

which returns something like

Results:  {'predictions': [{'output_0': [[0.995500684, 0.869658589, 1.00374675, 0.888075113], [0.999413252, 0.87126, 1.00516915, 0.885706604], [0.995500684, 0.869658589, 1.00374675, 0.888075113], [0.0133666955, 0.67006743, 0.0212151259, 0.68677783], [0.377415061, 0.990918756, 0.377846777, 0.99359262],...], 'output_2': [15.0, 15.0, 383.0, 15.0, 15.0...], 'output_3': 100}]}

[danellecline]

If we need to revisit this for performance, e.g. scaling for large data, many users, etc. it will be here for reference. Closing now.

[danellecline]

Transferring this issue to the deepsea-ai repository for implementation

[danellecline]

@duane-edgington I am adding these simple steps into a new command to simplify deployment and tear-down for the i2MAP model. Stay tuned for a request to test this.

[danellecline]

Relevant work

• https://aws.amazon.com/blogs/machine-learning/creating-a-machine-learning-powered-rest-api-with-amazon-api-gateway-mapping-templates-and-amazon-sagemaker/
• https://betterprogramming.pub/machine-learning-model-api-using-yolov5-with-fast-api-192f1290a982

[danellecline]

A FastAPI design was designed to do this. Related repo. Closing this.