greengrass deployments are broken

[tleyden]

sudo /greengrass/v2/bin/greengrass-cli deployment status -i d539cffa-14a1-4326-a05d-9d669d0c87f0
Jan 01, 2021 8:45:28 AM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onConnectionSetup
INFO: Socket connection /greengrass/v2/ipc.socket:8033 to server result [AWS_ERROR_SUCCESS]
Jan 01, 2021 8:45:28 AM software.amazon.awssdk.eventstreamrpc.EventStreamRPCConnection$1 onProtocolMessage
INFO: Connection established with event stream RPC server
d539cffa-14a1-4326-a05d-9d669d0c87f0: IN_PROGRESS

Yet in the logs I see:

2021-01-01T16:28:44.484Z [INFO] (Serialized listener processor) com.aws.greengrass.lifecyclemanager.KernelLifecycle: executor-service-shutdown-initiated. {}
2021-01-01T16:28:44.483Z [WARN] (FleetStatusService-lifecycle) com.aws.greengrass.status.FleetStatusService: service-state-transition-interrupted. Service lifecycle thread interrupted. Thread will exit now. {serviceName=FleetStatusService, currentState=FINISHED}
2021-01-01T16:28:44.485Z [INFO] (Thread-1) com.aws.greengrass.lifecyclemanager.KernelLifecycle: Waiting for executors to shutdown. {}
2021-01-01T16:28:44.486Z [ERROR] (pool-2-thread-21) com.aws.greengrass.deployment.activator.DeploymentActivator: merge-config. Deployment failed. {deploymentId=d539cffa-14a1-4326-a05d-9d669d0c87f0}
java.util.concurrent.ExecutionException: java.lang.InterruptedException
    at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
    at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895)
    at com.aws.greengrass.deployment.DeploymentConfigMerger$AggregateServicesChangeManager.removeObsoleteServices(DeploymentConfigMerger.java:286)
    at com.aws.greengrass.deployment.activator.DefaultActivator.activate(DefaultActivator.java:85)
    at com.aws.greengrass.deployment.DeploymentConfigMerger.updateActionForDeployment(DeploymentConfigMerger.java:128)
    at com.aws.greengrass.deployment.DeploymentConfigMerger.mergeInNewConfig(DeploymentConfigMerger.java:96)
    at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:112)
    at com.aws.greengrass.deployment.DefaultDeploymentTask.call(DefaultDeploymentTask.java:36)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.InterruptedException
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
    at java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at com.aws.greengrass.lifecyclemanager.GreengrassService.lambda$close$2(GreengrassService.java:355)
    ... 3 more

2021-01-01T16:28:44.496Z [INFO] (pool-2-thread-21) com.aws.greengrass.deployment.DeploymentService: deployment-task-execution. Finished deployment task. {deploymentId=d539cffa-14a1-4326-a05d-9d669d0c87f0, serviceName=DeploymentService, currentState=FINISHED}
2021-01-01T16:28:44.501Z [INFO] (pool-2-thread-21) com.aws.greengrass.componentmanager.ComponentStore: delete-component-start. {componentIdentifier=com.example.MyDockerComponent-v1.0.0}
2021-01-01T16:28:44.531Z [INFO] (pool-2-thread-21) com.aws.greengrass.componentmanager.ComponentStore: delete-component-finish. {componentIdentifier=com.example.MyDockerComponent-v1.0.0}

[tleyden]

This docker error might be worth noting:

2021-01-01T16:00:30.106Z [WARN] (Copier) com.example.MyDockerComponent: stderr. Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.24/images/load?quiet=1: dial unix /var/run/docker.sock: connect: permission denied. {scriptName=services.com.example.MyDockerComponent.lifecycle.Install.Script, serviceName=com.example.MyDockerComponent, currentState=NEW}

Also, how to wipe it clean: https://forums.aws.amazon.com/thread.jspa?threadID=332859&tstart=0

[tleyden]

Filed bug: https://github.com/aws-greengrass/aws-greengrass-nucleus/issues/807

[tleyden]

Reinstall greengrass:

backup old one:

root@raspberrypi:/# systemctl stop greengrass.service
root@raspberrypi:/# systemctl disable greengrass.service
Removed /etc/systemd/system/multi-user.target.wants/greengrass.service.
root@raspberrypi:/# mv greengrass/ greengrass-broken-issue-807

install new one:

pi@raspberrypi:~ $ sudo -E java -Dlog.store=FILE -jar ./GreengrassCore/lib/Greengrass.jar --aws-region "us-west-2" --root "/greengrass/v2" --thing-name "EyePiGreengrassCore" --thing-group-name "EyePiGreengrassCoreGroup" --tes-role-name "EyePiGreengrassV2TokenExchangeRole" --tes-role-alias-name "EyePiGreengrassCoreTokenExchangeRoleAlias" --component-default-user "ggc_user:ggc_group" --provision true --setup-system-service true --deploy-dev-tools true
Added ggc_user to ggc_group
Provisioning AWS IoT resources for the device with IoT Thing Name: [EyePiGreengrassCore]...
Found IoT policy "GreengrassV2IoTThingPolicy", reusing it
Creating keys and certificate...
Attaching policy to certificate...
Creating IoT Thing "EyePiGreengrassCore"...
Attaching certificate to IoT thing...
Successfully provisioned AWS IoT resources for the device with IoT Thing Name: [EyePiGreengrassCore]!
Adding IoT Thing [EyePiGreengrassCore] into Thing Group: [EyePiGreengrassCoreGroup]...
IoT Thing Group "EyePiGreengrassCoreGroup" already existed, reusing it
Successfully added Thing into Thing Group: [EyePiGreengrassCoreGroup]
Setting up resources for aws.greengrass.TokenExchangeService ...
Attaching TES role policy to IoT thing...
IAM policy named "EyePiGreengrassV2TokenExchangeRoleAccess" already exists. Please attach it to the IAM role if not already
Configuring Nucleus with provisioned resource details...
Downloading Root CA from "https://www.amazontrust.com/repository/AmazonRootCA1.pem"
Created device configuration
Successfully configured Nucleus with provisioned resource details!
Thing group exists, it could have existing deployment and devices, hence NOT creating deployment for Greengrass first party dev tools, please manually create a deployment if you wish to
Successfully set up Nucleus as a system service

[tleyden]

No greengrass cli, and everything still unhealthy in aws console. Going to try to nuke again, and nuke stuff in aws console.

[tleyden]

That worked, now seems to be in a healthy state again