fxgsell / GG-Edge-Inference

Using AWS Greengrass with the Nvidia Jetson TX2 to run ML models prepared with Amazon SageMaker.
MIT License
17 stars 7 forks source link

2-face-detection - Deployment failing within make #7

Closed shirkeyaws closed 6 years ago

shirkeyaws commented 6 years ago

1) Deployment failing 2) Deployment failing silently

make output

ec2-user:~/environment/GG-Edge-Inference/2-face-detection (master) $ make
echo "Zipping..."
Zipping...
rm -f package.zip
zip -rq package.zip greengrass_common  greengrass_ipc_python_sdk  greengrasssdk  *.py
echo "Uploading to Lambda"
Uploading to Lambda
aws --profile default --region us-east-1 lambda update-function-code --function-name ml-edge-workshop-lab-1 --zip-file fileb://`pwd`/package.zip
{
    "TracingConfig": {
        "Mode": "PassThrough"
    }, 
    "CodeSha256": "x4otKs1etleHIqP+H5tMfeRe9D5DUhGK44wQ4svy1tk=", 
    "FunctionName": "ml-edge-workshop-lab-1", 
    "CodeSize": 21229, 
    "MemorySize": 128, 
    "FunctionArn": "arn:aws:lambda:us-east-1:306280812807:function:ml-edge-workshop-lab-1", 
    "Version": "$LATEST", 
    "Role": "arn:aws:iam::306280812807:role/ml-edge-workshop-lab-1-role", 
    "Timeout": 3, 
    "LastModified": "2018-06-09T12:14:53.463+0000", 
    "Handler": "lambda_function.lambda_handler", 
    "Runtime": "python2.7", 
    "Description": ""
}
aws --profile default --region us-east-1 lambda publish-version --function-name ml-edge-workshop-lab-1 --query Version --output text  > LAMBDA_VERSION
aws --profile default --region us-east-1 lambda update-alias --name latest --function-name ml-edge-workshop-lab-1 --function-version `cat LAMBDA_VERSION`
{
    "AliasArn": "arn:aws:lambda:us-east-1:306280812807:function:ml-edge-workshop-lab-1:latest", 
    "FunctionVersion": "1", 
    "Name": "latest", 
    "Description": ""
}
rm LAMBDA_VERSION
echo "Deploying to GG"
Deploying to GG
aws --profile default --region us-east-1 greengrass list-groups --query "Groups[?Name=='ml-edge-workshop'].Id" --output text > GROUP_ID
aws --profile default --region us-east-1 greengrass list-groups --query "Groups[?Name=='ml-edge-workshop'].LatestVersion" --output text > GROUP_VERSION
aws --profile default --region us-east-1 greengrass create-deployment --group-id `cat GROUP_ID` --deployment-type NewDeployment --group-version-id `cat GROUP_VERSION` --query "DeploymentId" --output text > GROUP_DEPLOYMENT
aws --profile default --region us-east-1 greengrass get-deployment-status --group-id `cat GROUP_ID` --deployment-id `cat GROUP_DEPLOYMENT` --query "DeploymentStatus" --output text  > STATUS
Build in progress: Building
rm GROUP_ID GROUP_VERSION GROUP_DEPLOYMENT STATUS

manual steps to inspect output of deployment steps (ie. STATUS file before it is deleted)

ec2-user:~/environment/GG-Edge-Inference/2-face-detection (master) $ aws --profile default --region us-east-1 greengrass list-groups --query "Groups[?Name=='ml-edge-workshop'].Id" --output text > GROUP_ID                                 
ec2-user:~/environment/GG-Edge-Inference/2-face-detection (master) $ aws --profile default --region us-east-1 greengrass list-groups --query "Groups[?Name=='ml-edge-workshop'].LatestVersion" --output text > GROUP_VERSION
ec2-user:~/environment/GG-Edge-Inference/2-face-detection (master) $ aws --profile default --region us-east-1 greengrass create-deployment --group-id `cat GROUP_ID` --deployment-type NewDeployment --group-version-id `cat GROUP_VERSION` --query "DeploymentId" --output text > GROUP_DEPLOYMENT
ec2-user:~/environment/GG-Edge-Inference/2-face-detection (master) $ aws --profile default --region us-east-1 greengrass get-deployment-status --group-id `cat GROUP_ID` --deployment-id `cat GROUP_DEPLOYMENT` --query "DeploymentStatus" --output text  > STATUS
ec2-user:~/environment/GG-Edge-Inference/2-face-detection (master) $ cat ./STATUS 
Failure
shirkeyaws commented 6 years ago

Coudl the following issue be mitigated via aws greengrass associate-service-role-to-account

Actual error from console:

screen shot 2018-06-09 at 8 34 35 pm
shirkeyaws commented 6 years ago

performed manual deploy from AWS console and was given a permission request as shown, approved this and reran make/deploy

gg_deploy_failure_2

but still in an In Progress state after ~25 mins

gg_deploy_failure_4
shirkeyaws commented 6 years ago

Now trying to force a reset of the deployments, and Deploy again

screen shot 2018-06-09 at 9 10 57 pm screen shot 2018-06-09 at 9 11 16 pm screen shot 2018-06-09 at 9 12 12 pm

... and still in progress ...

screen shot 2018-06-09 at 9 13 29 pm
shirkeyaws commented 6 years ago

Outstanding/remaining issue:

fxgsell commented 6 years ago

Resolved by 666f118d12a6498d119982ade2e2352bdcda9708