search

aws-greengrass / aws-greengrass-shadow-manager

A GreengrassV2 Component that provides offline device shadow documents and optional synchronization to the IoT device shadow service.
Apache License 2.0
9 stars 6 forks source link

fix: synchronize access to sync information #166

Closed jcosentino11 closed 1 year ago

jcosentino11 commented 1 year ago

Issue #, if available:

Description of changes:

Shadow manager maintains sync information in its database, to help make decisions on when a shadow should be synced to the cloud. Currently, this table is read and (sometimes) written by multiple threads: 1) MqttClient thread when receiving updates from IoT Core 2) IPC thread when receiving local shadow update, 3) Sync thread, when a cloud or local update sync request is executing.

This change ensures that read/write access to sync information is synchronized.

Why is this change necessary:

A customer has experienced a race condition in the local --> cloud sync flow where updates made in quick succession result in certain sync requests being dropped, leaving the cloud shadow in the wrong state. Logs show that the IPC thread that receives the local update wrongfully determines that no update is needed, based on outdated information from the sync table. 

2023-01-26T09:08:35.278Z [INFO] (IPC THREAD) com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl: Updating sync info. {thing name=thing, shadow name=shadow, cloud-version=1, local-version=4}
2023-01-26T09:08:35.357Z [INFO] (SYNC THREAD) com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl: Updating sync info. {thing name=thing, shadow name=shadow, cloud-version=2, local-version=3}

How was this change tested:

Any additional information or context required to review the change:

Checklist:

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

github-actions[bot] commented 1 year ago

Unit Tests Coverage Report

File Coverage Lines Branches
All files 85% 91% 80% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 84% 89% 80% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.Direction 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 99% 98% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 84% 98% 71% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 80% 78% 83% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 98% 95% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 83% 93% 73% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 98% 95% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 74% 77% 71% :white_check_mark:
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 0% 0% 0% :x:
com.aws.greengrass.shadowmanager.ShadowManager 78% 83% 72% :white_check_mark:
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.PubSubIntegrator 89% 93% 86% :white_check_mark:
com.aws.greengrass.shadowmanager.ShadowManager$1 43% 43% 0% :x:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 67% 67% 0% :white_check_mark:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 67% 67% 0% :white_check_mark:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 67% 67% 0% :white_check_mark:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 67% 67% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.util.JsonMerger 94% 100% 89% :white_check_mark:
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 91% 98% 84% :white_check_mark:
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.util.Validator 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.util.JsonUtil 95% 100% 90% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 89% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 99% 99% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 92% 95% 89% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 63% 77% 50% :x:
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 87% 90% 85% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 93% 86% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 55% 85% 25% :x:
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 96% 99% 93% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.SyncHandler 71% 89% 53% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 91% 91% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.RequestMerger 78% 76% 81% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.CloudDataClient 70% 69% 72% :white_check_mark:
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 87% 93% 82% :white_check_mark:
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75% :white_check_mark:
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 86% 92% 80% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowDocument 81% 89% 73% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 93% 98% 89% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ErrorMessage 75% 100% 50% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowRequest 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowState 84% 87% 81% :white_check_mark:
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0% :white_check_mark:

Minimum allowed coverage is 65%

Generated by :monkey: cobertura-action against 199c5228d5acd6370d0f1040421856c207b88c80

github-actions[bot] commented 1 year ago

Integration Tests Coverage Report

File Coverage Lines Branches
All files 66% 72% 60% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest 74% 78% 70% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.Direction 83% 91% 75% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.LocalDeleteSyncRequest 55% 59% 50% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.FullShadowSyncRequest 49% 56% 41% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.CloudDeleteSyncRequest 53% 55% 50% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.OverwriteCloudShadowRequest 76% 76% 75% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.BaseSyncRequest 45% 50% 41% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.model.OverwriteLocalShadowRequest 33% 33% 0% :x:
com.aws.greengrass.shadowmanager.sync.model.CloudUpdateSyncRequest 63% 61% 64% :white_check_mark:
com.aws.greengrass.shadowmanager.ShadowManagerDatabase 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ShadowManager 91% 92% 89% :white_check_mark:
com.aws.greengrass.shadowmanager.ShadowManagerDAOImpl 84% 90% 77% :white_check_mark:
com.aws.greengrass.shadowmanager.AuthorizationHandlerWrapper 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.PubSubIntegrator 22% 22% 21% :x:
com.aws.greengrass.shadowmanager.ShadowManager$1 14% 14% 0% :x:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractDeleteThingShadowOperationHandler 100% 100% 0% :white_check_mark:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractListNamedShadowsForThingOperationHandler 100% 100% 0% :white_check_mark:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractUpdateThingShadowOperationHandler 100% 100% 0% :white_check_mark:
software.amazon.awssdk.aws.greengrass.GeneratedAbstractGetThingShadowOperationHandler 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.util.JsonMerger 34% 38% 31% :x:
com.aws.greengrass.shadowmanager.util.SyncNodeMerger 0% 0% 0% :x:
com.aws.greengrass.shadowmanager.util.DataOwner 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.util.ShadowWriteSynchronizeHelper 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.util.Validator 67% 68% 65% :white_check_mark:
com.aws.greengrass.shadowmanager.util.JsonUtil 80% 87% 73% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler 77% 77% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.GetThingShadowRequestHandler 87% 74% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.PubSubClientWrapper 82% 82% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.IpcRateLimiter 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowIPCHandler 55% 55% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter 94% 100% 88% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowRequestHandler 86% 72% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.InboundRateLimiter$1 75% 100% 50% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler 74% 76% 72% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.DeleteThingShadowIPCHandler 53% 53% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.GetThingShadowIPCHandler 95% 95% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.ListNamedShadowsForThingIPCHandler 61% 55% 67% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.model.Operation 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.ipc.model.PubSubRequest 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.PeriodicSyncStrategy 71% 92% 50% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.SyncStrategyFactory 100% 100% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy 87% 86% 88% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy 83% 90% 75% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientFactory 78% 91% 64% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.RequestBlockingQueue 61% 70% 52% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.SyncHandler 82% 94% 71% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.IotDataPlaneClientWrapper 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.RequestMerger 31% 41% 21% :x:
com.aws.greengrass.shadowmanager.sync.CloudDataClient 47% 43% 50% :white_check_mark:
com.aws.greengrass.shadowmanager.model.configuration.ShadowSyncConfiguration 71% 81% 61% :white_check_mark:
com.aws.greengrass.shadowmanager.model.configuration.ThingShadowSyncConfiguration 76% 78% 75% :white_check_mark:
com.aws.greengrass.shadowmanager.configuration.ShadowDocSizeConfiguration 61% 73% 50% :white_check_mark:
com.aws.greengrass.shadowmanager.configuration.RateLimitsConfiguration 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.configuration.ComponentConfiguration 100% 100% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.model.StrategyType 55% 69% 40% :white_check_mark:
com.aws.greengrass.shadowmanager.sync.strategy.model.Strategy 90% 100% 80% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ResponseMessageBuilder 91% 91% 0% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowDocument 81% 89% 73% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowStateMetadata 81% 86% 77% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ErrorMessage 97% 94% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowRequest 88% 77% 100% :white_check_mark:
com.aws.greengrass.shadowmanager.model.ShadowState 94% 98% 91% :white_check_mark:
com.aws.greengrass.shadowmanager.model.LogEvents 100% 100% 0% :white_check_mark:

Minimum allowed coverage is 45%

Generated by :monkey: cobertura-action against 199c5228d5acd6370d0f1040421856c207b88c80

nikkhilmuthye commented 1 year ago

This looks fine to me - I had to go through the code again to see that the other form of isUpdateNecessary is called when the sync queue is executing the request.

This check was added in #106 . Will it cause a regression for that case and cause unnecessary full syncs? That was for a local update case, but I want to be certain that it won't cause an issue

Ah yes. That would cause a full sync. When a local update happens, we update the cloud shadow. Once the cloud shadow updates, we get an update on the MQTT topic that the device is subscribed to. So, the cloud update requests will get added to the queue. If the device updates the shadow during that time, these two requests will merge into a full sync request which is unnecessary since we only need to update the cloud shadow.