Deployment error!

[vinodjsr]

Hi,

I am trying to install the basic people counter app in Panorama device as per the aws-panorama-samples/people_counter_tutorial.ipynb

I am getting the following set of errors at the deployment stage.

time 0: ERROR fetchFromS3(76) Error: GetObject: : curlCode: 28, Timeout was reached

time 1: ERROR downloadArtifactsS3(76) download file failed: {accountid}/payloads/device-*/deployments/xxxabc.json

time 2:ERROR downloadApplicationGraph(102) Downloading application payload for application: applicationInstance-* failed!

time 3: ERROR installApplicationTask(264) Failed to install application: applicationInstance- : EXCEPTION THROWN: at <time 3> ERROR loadApplicationGraph(301) Failed to install application:Unable to open application graph file: /data/cloud/graphs/applicationInstance-/graph.json:Please redeploy the application or reset your device. If issue persists, ask Panorama team for assistance

The device is online the whole time and all other pre-requisites are satisfied. I have tried many times but still get the same error. Wanted some help in this regard from AWS Panorama team.

Thanks.

[shimomut]

Hi @vinodjsr , thank you for your post.

I would like you to check 2 things

1. Could you check if you see following log in the occ_log stream? "Unable to fetch device credentials" or "Unable to update device credentials" or "Failed to fetch device credential"

2. Could you check if your network (where your Panorama device is located) is configured to allow access to S3 endpoints? Any firewall setting restricting access to S3? https://docs.aws.amazon.com/panorama/latest/dev/appliance-network.html#security-infrastructure-internet

[vinodjsr]

Hi,

1) Could not traceback any of these errors. Although its a bit puzzling that the containers compiled by panorama samples and upload to {myaccountid} s3 bucket (?) are not being fetched by the panorama device on {myaccountid} even though the object in s3 are public and block is disabled. Also access points to {myaccountid} s3 are internet enabled.

2) There is no firewall anywhere.

What I indeed would like to know are: 1) Once the containers are formed in the test utility, where are they uploaded? Are they uploaded to my s3 bucket with my access points or are they hosted on some AWS public s3 bucket and served by public AWS ECR?

2) I am getting regular curl timeout 28 <GetObject: : curlCode: 28, Timeout was reached> error, even though the device is online consistently. Its only when the artifacts are downloaded do i get the timeout error. Is there any network setting in Panorama service where this error can be avoided?

3) The locations of downloads are not clear enough for a newbie to start debugging- e.g the location <download file failed: 030722939722/nodePackages/abstract_rtsp_media_source/manifests/1.0/1f1bbff6b79e32b91dd51376f50e78ad333199515078d749710921f76498bdb7.json> is not clear. Here 030722939722 is not even my AWS account id!

Any pointers to comprehensive documentation on the panorama code flow starting from model building in EC2 hosted jupyter notebook to containerization of the application to its upload (where/what/how is the application being uploaded? own s3 or public s3/ECR?) to location from where panorama accesses these containers ( s3/ECR access point configuration) and how to check if the device is actually downloading the application and at which stage the device fails or is interrupted (?) WILL BE HIGHLY APPRECIATED.

PS: The dev and api documentation do not answer any of the above questions and their structure is very vague. The documentation fails to address any workflow level troubleshooting and is not useful in general pipeline debug setting.

Thanks, vinod

[shimomut]

@vinodjsr, the s3 buckets are owned by Panorama service, they are configured to be accessed only from specific customer account (in this case, yours) and IAM role. Just to be clear, it is not public.

To troubleshoot your asset download failures, I would like to gather full logs from all log streams. Could we please start an email discussion? Please send an email to me : shimomut{at}amazon.com

Then we would ask to export log streams around the failure using this tool. https://github.com/aws-samples/aws-panorama-samples/tree/main/tools/export_logs

[vinodjsr]

@shimomut,

I was able to fix the deployment error by resolving network latency. Turns out the panorama device requires very low network latency. Thanks for all the help though.

Thanks, vinod

[vinodjsr]

Issue resolved. Thanks @shimomut for prompt response and pointing to debug tools.