mbari-org / deepsea-ai

DeepSea-AI is a Python package to simplify processing deep sea video in AWS https://aws.amazon.com from a command line.
http://docs.mbari.org/deepsea-ai/
Apache License 2.0
3 stars 1 forks source link

Set up object detection ML server externally. #20

Closed danellecline closed 1 year ago

danellecline commented 2 years ago

This task is to setup an object detection server externally.

This is relatively simple to do in AWS.

Reference:

yolov5 inference

IMPORTANT. This was done in our new AWS sub account by setting up credentials through:

https://mbari.awsapps.com/start

using an mbari username, e.g. duane and an mbari password.

Keys can be setup either through environment variables, e.g. AWS_REGION, AWS_SECRET_ACCESS_KEY, etc. or in a ~/.aws/credentials file.

danellecline commented 2 years ago

Step 1. Export your model to tensorflow saved model format.

@lonnylundsten you can do this in your Colab notebook with

python export.py --weights best.pt --include saved_model --conf-thres 0.25 --iou-thres 0.45 --nms

This can also be done with the ultralytics/yolov5 docker image.

Saved Model format encapsulates the whole network topology and weights. There is no need to have any python code sitting along side the model weights.

danellecline commented 2 years ago

Step 2. Package that model as required for serving through an endpoint and upload it to a bucket

mkdir -p export/Servo
mv best_saved_model export/Servo/1
tar -czvf model.tar.gz export/
aws s3 cp model.tar.gz s3://902005-models/20221028T054534Z-model.tar.gz
lonnylundsten commented 2 years ago

I'm assuming that these are instructions and that this isn't a task assigned to me? It's not clear but I guess I'm wondering if this is needed that ya'll can take care of it, right?

Hi @lonnylundsten - that's right, no action needed on your part. I'm handling this task. Just including you for reference as I'm unclear if people who are tagged in the instructions and not assigned get a notification.

lonnylundsten commented 2 years ago

OK. Perfect. I wasn't certain if tagging me == assigning task or not. Thanks!

danellecline commented 2 years ago

Step 3. Setup the endpoint with a few lines of python code. Let's call this mbari315k, and put it is on a ml.m5.xlarge instance with runs 0.24 cents/hour for evaluation. This is not a GPU, but it could be moved to a GPU instance if there is a need. @duane-edgington may have some thoughts on better instances. I'm following the blog instructions here.

pip install sagemaker

import os
import tensorflow as tf
from tensorflow.keras import backend
from sagemaker.tensorflow import TensorFlowModel

model_data = 's3://902005-models/20221028T054534Z-model.tar.gz'
role = 'arn:aws:iam::168552337187:role/DeepSeaAI'

model = TensorFlowModel(model_data=model_data, framework_version='2.8', role=role)

INSTANCE_TYPE = 'ml.m5.xlarge'
ENDPOINT_NAME = 'mbari315k'

predictor = model.deploy(initial_instance_count=1,
                            instance_type=INSTANCE_TYPE,
                            endpoint_name=ENDPOINT_NAME)

After a few minutes, the endpoint should now be visible 🥳 💯. Check out the endpoint through the console at 902005_vaa endpoints

endpoints-aws-console
danellecline commented 2 years ago

Step 4. Test the endpoint with a empty image

import numpy as np
import json
import boto3
import json

ENDPOINT_NAME = 'mbari315k'

model_height, model_width = 640, 640
blank_image = np.zeros((model_height, model_width,3), np.uint8)
data = np.array(blank_image.astype(np.float32)/255.)
payload = json.dumps([data.tolist()])

client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                   ContentType='application/json',
                                   Body=payload)

result = json.loads(response['Body'].read().decode())
print('Results: ', result)

which should output something like

Results:  {'predictions': [{'output_0': [[0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0,
0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0,
0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0], [0.0, 0.0, 0.0, 0.0]], 'output_1': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'output_2': [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0,
0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0], 'output_3': 0}]}
danellecline commented 2 years ago

To test with a PNG image, similar to what we capture and download via VARS:

import numpy as np
import json
import boto3
import json
from PIL import Image
from numpy import asarray

ENDPOINT_NAME = 'mbari315k'
model_height, model_width = 640, 640

test_image = '/Volumes/Tempbox/Lonny/benthic_test_images/20191104T235504Z--9faf7c89-2fef-4a13-bf8b-dc79ceb3f981.png'

rgba_image = Image.open(test_image)
rgb_image = rgba_image.convert('RGB').resize((model_width,model_height))
data = np.array(np.array(rgb_image).astype(np.float32))

payload = json.dumps([data.tolist()])

client = boto3.client('sagemaker-runtime')
response = client.invoke_endpoint(EndpointName=ENDPOINT_NAME,
                                   ContentType='application/json',
                                   Body=payload)

result = json.loads(response['Body'].read().decode())
print('Results: ', result)

which returns something like

Results:  {'predictions': [{'output_0': [[0.995500684, 0.869658589, 1.00374675, 0.888075113], [0.999413252, 0.87126, 1.00516915, 0.885706604], [0.995500684, 0.869658589, 1.00374675, 0.888075113], [0.0133666955, 0.67006743, 0.0212151259, 0.68677783], [0.377415061, 0.990918756, 0.377846777, 0.99359262],...], 'output_2': [15.0, 15.0, 383.0, 15.0, 15.0...], 'output_3': 100}]}
danellecline commented 1 year ago

If we need to revisit this for performance, e.g. scaling for large data, many users, etc. it will be here for reference. Closing now.

danellecline commented 1 year ago

Transferring this issue to the deepsea-ai repository for implementation

danellecline commented 1 year ago

@duane-edgington I am adding these simple steps into a new command to simplify deployment and tear-down for the i2MAP model. Stay tuned for a request to test this.

danellecline commented 1 year ago

Relevant work

danellecline commented 1 year ago

A FastAPI design was designed to do this. Related repo. Closing this.