This is the cloudformation package to demonstrate Machine Learning model deployment at scale with AWS Greengrass on NVIDIA Jetson Devices. This demonstrates consists four parts:
If you have not done so, configure your AWS CLI with a profile linked to your account.
aws configure --profile ANY_NAME_YOU_LIKENext, please clone this repo onto any PATH onto your machine.
git clone https://github.com/rvanderwerf/aws-iot-greengrass-deploy-nvidia-deepstream-on-edge.git
cd aws-iot-greengrass-deploy-nvidia-deepstream-on-edgeOnce you successfully cloned this repo, please edit deploy.env file to replace
source deploy.envThen, we are going to use the contents in provisioning_cf_script folder.
cd provisioning_cf_scriptExamine the deploy.sh file. This is the starting point of the cloud formation. This bash script first creates a S3 bucket to store all of the relevant assets created in this repo. Then, it calls provision-greengrass-cfn.yml CloudFormation script with AWS SAM CLI toolset.
After CloudFormation runs successfully, we are going to generate several commands with S3 pre-signed links the provisioned certificates to be downloaded to Jetson devices (which only last for one day) into install_greengrass.sh file, which we are going to use later. The pre-signed links last for a week. If those links expire, you will have to manually run the last 5 lines in deploy.sh again and generate new links.
This is a CloudFormation script run by deploy.sh. This script internally calls a one-time lambda function to create two things:
Run the provisioning script:
. deploy.shAfter this, an output.txt file should be generated in this folder. Just leave it there. You can also navigate to AWS console, and see things just created in Manage -> Things.
We are going to simulate a package to be deployed from AWS cloud to Jetson device by using example applications and models from DeepStream SDK. We can first download DeepStream SDK for Jetson onto this machine from https://developer.nvidia.com/deepstream-download, and download the .tar version. After successful download, you can untar the package:
mkdir $GG_DEPLOYMENT_HOME/deepstream-source
tar -xpvf <DOWNLOAD_PATH>/deepstream_sdk_<VERSION>_jetson.tbz2 -C $GG_DEPLOYMENT_HOME/deepstream-sourceThen, we are going to upload this model into the S3 bucket we created in Part 1.
cd $GG_DEPLOYMENT_HOME/deepstream-source/opt/nvidia/deepstream/deepstream-5.0/samples/models/Primary_Detector
zip model.zip *
aws s3 cp ./model.zip s3://$S3_BUCKET/model/model.zip --profile $AWS_PROFILEIf you do not have DeepStream installed, such as running a Jetson Nano SD JetPack 4.4 Image, copy the DeepStream SDK to your device:
scp <DOWNLOAD_PATH>/deepstream_sdk_<VERSION>_jetson.tbz2 <YOUR_JETSON_USERNAME>@<YOUR_JETSON_IP>:/home/<USER_HOME>
Now ssh do your Jetson device and install the SDK
cd /
sudo tar xpvf <DOWNLOAD_PATH>/deepstream_sdk_<VERSION>_jetson.tbz2Once we upload the model, we also need to prepare a DeepStream package to be deployed with Greengrass. We are going to use DeepStream sample app on your Jetson device for this demonstration:
scp <YOUR_JETSON_USERNAME>@<YOUR_JETSON_IP>:<ABSOLUTE_DEEPSTREAM_PATH>/sources/apps/sample_apps/deepstream-app/* $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app/Then we need to copy the configuration files to this sample application, and modify it with PLACEHOLDERS so we can locate the ML model correctly when the DeepStream app starts to run in Greengrass:
cp $GG_DEPLOYMENT_HOME/deepstream-source/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/config_infer_primary_nano.txt $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app/
cp $GG_DEPLOYMENT_HOME/deepstream-source/opt/nvidia/deepstream/deepstream-5.0/samples/configs/deepstream-app/source1_usb_dec_infer_resnet_int8.txt $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app/
cd $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_app
### On MacOS/BSD
sed -i '.tmp' -e 's|model-engine-file|#model-engine-file|g' config_infer_primary_nano.txt
sed -i '.tmp' -e 's|../../models/Primary_Detector_Nano|/resnet_10_model|g' config_infer_primary_nano.txt
sed -i '.tmp' -e 's|model-engine-file|#model-engine-file|g' source1_usb_dec_infer_resnet_int8.txt
### On Ubuntu/Debian
sed -i -e 's|model-engine-file|#model-engine-file|g' config_infer_primary_nano.txt
sed -i -e 's|../../models/Primary_Detector_Nano|/resnet_10_model|g' config_infer_primary_nano.txt
sed -i -e 's|model-engine-file|#model-engine-file|g' source1_usb_dec_infer_resnet_int8.txt
mv config_infer_primary_nano.txt config_infer_primary.txt
## Only needed on MacOS/BSD
rm *.tmpOpen the "source1_usb_dec_infer_resnet_int8.txt" you just created in any text editor you prefer. We need to modify the source and sink to the known source. In this demo, we are going to use a RTSP camera as the source. You may also choose to use other sources. So we modify the source to be:
[source0]
enable=1
#Type - 1=CameraV4L2 2=URI 3=MultiURI 4=RTSP
type=4
uri=<MY_RTSP_STREAM>
num-sources=1
#drop-frame-interval=2
gpu-id=0
# (0): memtype_device - Memory type Device
# (1): memtype_pinned - Memory type Host Pinned
# (2): memtype_unified - Memory type Unified
cudadec-memtype=0Then we disable any sink except the RTSP stream sink:
[sink2]
enable=1
#Type - 1=FakeSink 2=EglSink 3=File 4=RTSPStreaming 5=Overlay
type=4
#1=h264 2=h265
codec=1
sync=0
bitrate=4000000
# set below properties in case of RTSPStreaming
rtsp-port=8554
udp-port=5400If you have other source preference, feel free to use those as well. As long as you verify that your Jetson device have access to these sources or sinks.
If you are experiencing error, it is likely that you have switched your terminal and lost environmental variables. In this case, please run locate output.txt in this folder, and copy the last line, which looks similar to: greengrass-deepstream-\<YOUR-AWS-ACCOUNT-NUMBER>-test-assets. And re-export these environmental variables.
source ./deploy.envAnd then pick up from where you experienced error in the previous commands and try again.
The major function for this script is to create a greengrass group. And to this greengrass group, we attach 6 modules:
This link posts more detailed information on this set-up: https://aws.amazon.com/blogs/iot/automating-aws-iot-greengrass-setup-with-aws-cloudformation/
In order to use Greengrass lambda function successfully, you also need to git clone the Greengrass SDK in your local Lambda function.
cd $GG_DEPLOYMENT_HOME
git clone https://github.com/aws/aws-greengrass-core-sdk-python.git
cp -r aws-greengrass-core-sdk-python/greengrasssdk $GG_DEPLOYMENT_HOME/formation_cf_script/lambda_deepstream_appRun the scripts:
cd $GG_DEPLOYMENT_HOME/formation_cf_script
. deploy.shAfter script runs successfully, you should be able to observe on AWS console a Greengrass group ready to be deployed. And you could click from AWS console to do a deployment on the right upper corner, but don't click deploy yet until Greengrass is set up.
If the deployment status shows and yellow label and says "pending", that means the Greengrass group has been successfully created. Otherwise, please click on the deployment failure entry and read the error log.
Take out your Jetson device, and install Greengrass by running on your Jetson device install_greengrass.sh script we have prepared for you. In order to copy this script to your Jetson device first, you can either use scp to copy install_greengrass.sh to your Jetson device or upload it somewhere to be downloaded from your Jetson device. And then run
scp $GG_DEPLOYMENT_HOME/formation_cf_script/install_greengrass.sh <YOUR_JETSON_USERNAME>@<YOUR_JETSON_IP>:~/Downloads
ssh <YOUR_JETSON_USERNAME>@<YOUR_JETSON_IP>
cd ~/Downloads
sudo sh install_greengrass.shThis script will automatically download and install Greengrass from the following page: https://docs.aws.amazon.com/greengrass/latest/developerguide/what-is-gg.html#gg-core-download-tab It will also automatically retrieve the certificates and Greengrass configurations you have created in Part 1 from S3 bucket, and download them on your Jetson device.
You should now have all the resources needed to run DeepStream and Greengrass, so we can start Greengrass. You can now run the following script, and observe the inference result on the sink we just configured.
cd /greengrass/ggc/core
sudo ./greengrassd startIf your bucket is not public, you will have noticed some errors during the GG setup. If that's the case, from your PC, copy the bucket contents to your local machine:
cd $GG_DEPLOYMENT_HOME mkdir s3bucket aws s3 cp --recursive s3://greengrass-deepstream-XXXX-test-assets . Now copy the items in the install_greengrass.sh script from your local folder to the correct places on your Jetson device.
If your inference results are not outputted correctly, you can read Greengrass runtime logs to figure out why. Please navigate, on your Jetson device, to the following directory:
sudo su
cd /greengrass/ggc/var/log/For Greengrass system logs, you can read log fils in the system folder. For example, "/greengrass/ggc/var/log/system/GGConnManager.log" contains all the messsages Greengrass had received or sent as MQTT broker. "/greengrass/ggc/var/log/system/runtime.log" shows generic Greengrass operation logs.
For Lambda function specific logs, you can locate them in
/greengrass/ggc/var/log/user/us-west-2/xxxxxxxxxxwhere xxxxxxxxxx is you AWS account number. All of the DeepStream operations logs should also be in this folder.
In this application, we also created a separate set of certificates and a corresponding thing for DeepStream app to authenticate with both AWS IoT Core and this Greengrass group. In order to use AWS specific msg-broker in DeepStream, you need to follow this GitHub (but skip provisioning process, because certificates have already been created for you): https://github.com/awslabs/aws-iot-core-integration-with-nvidia-deepstream
Please use the links in output.txt in this folder to download the certificates to the right folders.
This library is licensed under the MIT-0 License. See the LICENSE file.