Dunfell Greengrass v2 2.4.0 runtime failure during Greengrass deployments on raspberrypi 64bit

[jcheng2021]

I built a image with meta-aws with rpi64. the greengrass Nucleus was launched successfully. I tried to deploy a simple hello-world docker container, the deployment failed due to following error.

2021-12-21T15:24:45.730Z [INFO] (pool-2-thread-15) com.aws.greengrass.componentmanager.ComponentManager: No running component satisfies the requirement. Searching in the local component store.. {} 2021-12-21T15:24:45.735Z [INFO] (pool-2-thread-15) com.aws.greengrass.componentmanager.ComponentManager: Can't find a local candidate that satisfies the requirement.. {} 2021-12-21T15:24:45.736Z [INFO] (pool-2-thread-15) com.aws.greengrass.componentmanager.ComponentManager: negotiate-version-with-cloud-start. Negotiating version with cloud. {} 2021-12-21T15:24:48.322Z [ERROR] (pool-2-thread-15) com.aws.greengrass.componentmanager.ComponentManager: Failed to negotiate version with cloud and no local version to fall back to. {componentName=61 bcbb9009c585469bb49ffc, versionRequirement={thing/61b2c9436580054670f2d51e==1.0.0}} software.amazon.awssdk.services.greengrassv2data.model.GreengrassV2DataException: null (Service: GreengrassV2Data, Status Code: 403, Request ID: f4d374a5-15d7-9529-a73b-ee81da68c197, Extended Request ID: null) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.greengrassv2data.DefaultGreengrassV2DataClient.resolveComponentCandidates(DefaultGreengrassV2DataClient.java:500) at com.aws.greengrass.componentmanager.ComponentServiceHelper.resolveComponentVersion(ComponentServiceHelper.java:68) at com.aws.greengrass.componentmanager.ComponentManager.lambda$negotiateVersionWithCloud$0(ComponentManager.java:198) at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49) at com.aws.greengrass.componentmanager.ComponentManager.negotiateVersionWithCloud(ComponentManager.java:197) at com.aws.greengrass.componentmanager.ComponentManager.resolveComponentVersion(ComponentManager.java:154) at com.aws.greengrass.componentmanager.DependencyResolver.lambda$resolveDependencies$2(DependencyResolver.java:127) at com.aws.greengrass.componentmanager.DependencyResolver.resolveComponentDependencies(DependencyResolver.java:225) at com.aws.greengrass.componentmanager.DependencyResolver.resolveDependencies(DependencyResolver.java:125) at com.aws.greengrass.deployment.DefaultDeploymentTask.lambda$call$2(DefaultDeploymentTask.java:105) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

2021-12-21T15:24:48.329Z [ERROR] (pool-2-thread-14) com.aws.greengrass.deployment.DeploymentService: Error occurred while processing deployment. {deploymentId=44130225-f11c-4b1b-818f-500ceabf07a1, ser viceName=DeploymentService, currentState=RUNNING} java.util.concurrent.ExecutionException: com.aws.greengrass.componentmanager.exceptions.NoAvailableComponentVersionException: Failed to negotiate component '61bcbb9009c585469bb49ffc' version with clou d and no local applicable version satisfying requirement '{thing/61b2c9436580054670f2d51e==1.0.0}'. at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:107) at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:38) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.aws.greengrass.componentmanager.exceptions.NoAvailableComponentVersionException: Failed to negotiate component '61bcbb9009c585469bb49ffc' version with cloud and no local applicable vers ion satisfying requirement '{thing/61b2c9436580054670f2d51e==1.0.0}'. at com.aws.greengrass.componentmanager.ComponentManager.negotiateVersionWithCloud(ComponentManager.java:210) at com.aws.greengrass.componentmanager.ComponentManager.resolveComponentVersion(ComponentManager.java:154) at com.aws.greengrass.componentmanager.DependencyResolver.lambda$resolveDependencies$2(DependencyResolver.java:127) at com.aws.greengrass.componentmanager.DependencyResolver.resolveComponentDependencies(DependencyResolver.java:225) at com.aws.greengrass.componentmanager.DependencyResolver.resolveDependencies(DependencyResolver.java:125) at com.aws.greengrass.deployment.DefaultDeploymentTask.lambda$call$2(DefaultDeploymentTask.java:105) ... 4 more Caused by: software.amazon.awssdk.services.greengrassv2data.model.GreengrassV2DataException: null (Service: GreengrassV2Data, Status Code: 403, Request ID: f4d374a5-15d7-9529-a73b-ee81da68c197, Extend ed Request ID: null) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleErrorResponse(CombinedResponseHandler.java:123) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handleResponse(CombinedResponseHandler.java:79) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:59) at software.amazon.awssdk.core.internal.http.CombinedResponseHandler.handle(CombinedResponseHandler.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:40) at software.amazon.awssdk.core.internal.http.pipeline.stages.HandleResponseStage.execute(HandleResponseStage.java:30) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:73) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptTimeoutTrackingStage.execute(ApiCallAttemptTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:77) at software.amazon.awssdk.core.internal.http.pipeline.stages.TimeoutExceptionHandlingStage.execute(TimeoutExceptionHandlingStage.java:39) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:50) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallAttemptMetricCollectionStage.execute(ApiCallAttemptMetricCollectionStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:64) at software.amazon.awssdk.core.internal.http.pipeline.stages.RetryableStage.execute(RetryableStage.java:34) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:56) at software.amazon.awssdk.core.internal.http.StreamManagingStage.execute(StreamManagingStage.java:36) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.executeWithTimer(ApiCallTimeoutTrackingStage.java:80) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:60) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallTimeoutTrackingStage.execute(ApiCallTimeoutTrackingStage.java:42) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:48) at software.amazon.awssdk.core.internal.http.pipeline.stages.ApiCallMetricCollectionStage.execute(ApiCallMetricCollectionStage.java:31) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:37) at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:193) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:133) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:159) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:112) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:167) at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:94) at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:55) at software.amazon.awssdk.services.greengrassv2data.DefaultGreengrassV2DataClient.resolveComponentCandidates(DefaultGreengrassV2DataClient.java:500) at com.aws.greengrass.componentmanager.ComponentServiceHelper.resolveComponentVersion(ComponentServiceHelper.java:68) at com.aws.greengrass.componentmanager.ComponentManager.lambda$negotiateVersionWithCloud$0(ComponentManager.java:198) at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:49) at com.aws.greengrass.componentmanager.ComponentManager.negotiateVersionWithCloud(ComponentManager.java:197) ... 9 more

2021-12-21T15:25:00.678Z [INFO] (pool-2-thread-11) com.aws.greengrass.deployment.DeploymentService: Current deployment finished. {DeploymentId=arn:aws:greengrass:us-east-1:118134108323:configuration:t hing/61b2c9436580054670f2d51e:11, serviceName=DeploymentService, currentState=RUNNING} 2021-12-21T15:25:00.758Z [INFO] (pool-2-thread-11) com.aws.greengrass.deployment.DeploymentStatusKeeper: Stored deployment status. {DeploymentId=arn:aws:greengrass:us-east-1:118134108323:configuration :thing/61b2c9436580054670f2d51e:11, DeploymentStatus=FAILED} 2021-12-21T15:25:00.762Z [INFO] (pool-2-thread-11) com.aws.greengrass.deployment.ShadowDeploymentListener: Updated reported state for deployment. {CONFIGURATION_ARN=arn:aws:greengrass:us-east-1:118134 108323:configuration:thing/61b2c9436580054670f2d51e:11, status=FAILED} 2021-12-21T15:25:00.803Z [INFO] (pool-2-thread-11) com.aws.greengrass.status.FleetStatusService: fss-status-update-published. Status update published to FSS. {serviceName=FleetStatusService, currentSt ate=RUNNING} 2021-12-21T15:25:00.807Z [INFO] (pool-2-thread-11) com.aws.greengrass.deployment.DeploymentDirectoryManager: Persist link to last deployment. {link=/greengrass/v2/deployments/previous-failure} 2021-12-21T15:25:00.809Z [INFO] (pool-2-thread-11) com.aws.greengrass.deployment.DeploymentDirectoryManager: Clean up link to earlier deployment. {link=/greengrass/v2/deployments/previous-failure}

Could it be the recipe issue?

[rpcme]

Hello @jcheng2021 sorry for the late response, can you please give us the repo branches and hashes you're using? This is displayed at the start of the yocto build. Also, providing local.conf will be useful. For us, it's working, but that doesn't mean that there could be issue under specific configurations.

[jcheng2021]

Thanks for contacting me. the repo branch I'm using is dunfell.

Here are the conf snipplet: AWS_IOT_GG = "greengrass-bin \ greengrass-bin-demo \ ntp \ python3-awscli \ docker \ python3-docker-compose \ "

RPI4_WIFI = "crda \ iw \ linux-firmware-rpidistro-bcm43455 \ wpa-supplicant \ "

RPI4_BT = "bluez-firmware-rpidistro-bcm4345c0-hcd \ bluez5 \ bluez5-dev \ bluez5-noinst-tools \ bluez5-obex \ bluez5-testtools \ python3-pybluez \ "

RPI4_NET = "i2c-tools \ python3-smbus \ bridge-utils \ hostapd \ dhcp-server \ iptables \ "

IMAGE_INSTALL_append = " ${AWS_IOT_GG}" IMAGE_INSTALL_append = " ${RPI4_WIFI}" IMAGE_INSTALL_append = " ${RPI4_BT}" IMAGE_INSTALL_append = " ${RPI4_NET}"

taken from meta-raspberrypi

IMAGE_INSTALL_append = " kernel-modules" IMAGE_INSTALL_append = " strace"

mosquitto and dependencies

IMAGE_INSTALL_append = " mosquitto-dev" IMAGE_INSTALL_append = " mosquitto-clients" IMAGE_FEATURES += "ssh-server-dropbear"

[rpcme]

Let me notify internal team to assist.

[mangelino]

HI @jcheng2021 , can you share the deployment configuration you are using? IoT Console > Greengrass (menu) > Deployments (menu) > your deployment (link) > IoT Job (link) > Job Document (tab)

[MikeDombo]

The error is a http 403 which means access denied. Therefore, your greengrass device is missing the proper permissions in the IoT policy. Please verify the IoT policy. See https://docs.aws.amazon.com/greengrass/v2/developerguide/device-auth.html#greengrass-core-minimal-iot-policy

[jcheng2021]

I can't find the IoT Job and Job Document any where

[rpcme]

@jcheng2021 can you please verify permissions as @MikeDombo suggests? Mike is one of our senior Greengrass engineers on the service team - I hope his intuition here helps.

[rpcme]

Assuming this issue is resolved.