search

awslabs / aws-greengrass-labs-certificate-rotator

Greengrass component and companion cloud backend for rotating the core device certificate and private key
Apache License 2.0
11 stars 2 forks source link

Old cert hadn't been cleaned up from AWS console after cert rotation had been completed #10

Closed JyothiAkula92 closed 11 months ago

JyothiAkula92 commented 11 months ago

We could see component logs as they are supposed to be. Component was receiving messages at topics and able to publish messages at topics. Old cert was removed from device and replace it with new cert. device is able to communicate with AWS IoT and components are running with out any issues using new cert. But, caveat that we were seeing is, old cert wasn't removed from AWS console and it was still attached to the IoT Thing. On further investigation, we found that JobExecutionTerminal rule had not even triggered job-execution-terminal lambda to clean the old cert.

gregbreen commented 11 months ago

The backend CDK stack should have enabled this. It includes a custom resource to do it: https://github.com/awslabs/aws-greengrass-labs-certificate-rotator/blob/main/backend/lib/certificate-rotator-stack.ts#L378-L406

In CloudWatch there should be a log group /aws/lambda/AWSLabsCertificateRotatorCustomResourceJobExecutionEvents. It should have a log stream recording that this was enabled.

If there's no evidence it failed, could it be that someone disabled the job executions events after you deployed the CDK stack?

JyothiAkula92 commented 11 months ago

Response messages from the AWS IoT Jobs service don't pass through the message broker and they can't be subscribed to by other clients or rules. To be notified of jobs updates, enable these jobs events by using the AWS Management Console, or by using the API or CLI. So, We ended up enabling events for job execution

aws iot update-event-configurations \ --event-configurations "{\"JOB_EXECUTION\":{\"Enabled\":true}}"

from there on wards, our JobExecutionTerminal IoT rule started triggering job-execution-terminal lambda to clean the old cert