(shadowmanager): Reports payload exceeds maximum size allowed when it doesnt

[crispinboylan-int]

Describe the bug We have 30k shadows, and shadow manager is updated with the maxShadowDocumentSizeBytes (30720).

When we have a large shadow close to the 30k limit we receive this error in shadow manager, and we cant find a way to get the shadow successfully synced again even if we remove data because it cant process that version.

If the cloud shadow api allows the update to the shadow, why does shadow manager not?

To Reproduce

Have a shadow close to the limit size.

Expected behavior Accepted the update, as per the ap.

Actual behavior

Receive the following log message:

2022-09-29T15:40:39.487Z [INFO] (pool-2-thread-34) com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy: sync. Executing sync request. {Type=LocalUpdateSyncRequest, thing name=home-OUnWKRu7Pn1B, shadow name=RoomConfig} 2022-09-29T15:40:39.505Z [WARN] (pool-2-thread-34) com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler: handle-update-thing-shadow. {thing name=home-OUnWKRu7Pn1B, shadow name=RoomConfig} com.aws.greengrass.shadowmanager.exception.InvalidRequestParametersException: The payload exceeds the maximum size allowed at com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler.lambda$handleRequest$2(UpdateThingShadowRequestHandler.java:137) at com.aws.greengrass.ipc.common.ExceptionUtil.translateExceptions(ExceptionUtil.java:33) at com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler.handleRequest(UpdateThingShadowRequestHandler.java:98) at com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest.execute(LocalUpdateSyncRequest.java:121) at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:113) at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:50) at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:111) at com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy.lambda$new$0(BaseSyncStrategy.java:150) at com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy.syncLoop(BaseSyncStrategy.java:376) at com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy.syncLoop(RealTimeSyncStrategy.java:72) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833)

2022-09-29T15:40:39.507Z [ERROR] (pool-2-thread-34) com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy: sync. Skipping sync request. {thing name=home-OUnWKRu7Pn1B, shadow name=RoomConfig} com.aws.greengrass.shadowmanager.exception.SkipSyncRequestException: software.amazon.awssdk.aws.greengrass.model.InvalidArgumentsError: The payload exceeds the maximum size allowed at com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest.execute(LocalUpdateSyncRequest.java:151) at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$0(SyncHandler.java:113) at com.aws.greengrass.util.RetryUtils.runWithRetry(RetryUtils.java:50) at com.aws.greengrass.shadowmanager.sync.SyncHandler.lambda$new$1(SyncHandler.java:111) at com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy.lambda$new$0(BaseSyncStrategy.java:150) at com.aws.greengrass.shadowmanager.sync.strategy.BaseSyncStrategy.syncLoop(BaseSyncStrategy.java:376) at com.aws.greengrass.shadowmanager.sync.strategy.RealTimeSyncStrategy.syncLoop(RealTimeSyncStrategy.java:72) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:833) Caused by: software.amazon.awssdk.aws.greengrass.model.InvalidArgumentsError: The payload exceeds the maximum size allowed at com.aws.greengrass.shadowmanager.ipc.BaseRequestHandler.throwInvalidArgumentsError(BaseRequestHandler.java:89) at com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler.lambda$handleRequest$2(UpdateThingShadowRequestHandler.java:174) at com.aws.greengrass.ipc.common.ExceptionUtil.translateExceptions(ExceptionUtil.java:33) at com.aws.greengrass.shadowmanager.ipc.UpdateThingShadowRequestHandler.handleRequest(UpdateThingShadowRequestHandler.java:98) at com.aws.greengrass.shadowmanager.sync.model.LocalUpdateSyncRequest.execute(LocalUpdateSyncRequest.java:121) ... 11 more

Environment

• OS: Ubuntu 22.04
• JDK version: 17.0.4
• Nucleus version: 2.7.0
• Shadow Manager version: 2.2.1

Additional context Add any other context about the problem here.

E.g. what is the impact of the bug?

[MikeDombo]

Hi, Can you please provide the full configuration that you applied to the shadow manager component which shows the maximum size setting?

Have you tried increasing the limit even beyond 30KB?

Note that the max shadow size includes all the JSON including both the reported and desired states, not just the one key you may have changed.

[crispinboylan-int]

aws.greengrass.ShadowManager: componentVersion: 2.2.1 configurationUpdate: merge: '{"shadowDocumentSizeLimitBytes": 30720, "synchronize":{"coreThing":{"classic":false,"namedShadows": "RoomConfig"}}}'

from my calculations it is still under 30720, total size including reported and desired. but in any case why is it rejecting it if the cloud side accepted?

[MikeDombo]

What do you say that the size is?

There is a local limit in order to protect you from trying to set a shadow which cannot be synchronized to the cloud. AWS IoT shadows have a 8KB limit by default and this size can only be increased by requesting a service limit increase.

[crispinboylan-int]

the original size was 30463. we have had limit increased to 30kb via a support request.

[MikeDombo]

When calling this API, you provide a byte array in the payload key of UpdateThingShadowRequest. What is the size of that byte array? This is the array which Shadow Manager checks to ensure you are under the limit.

[crispinboylan-int]

that was 30,463

this can also be provoked by sending in an update via the aws managaement console, the update is accepted there, the resulting document is less than 30,720 but the shadowmanager complains when syncing it to the local shadow

[MikeDombo]

There isn't any size checking when synchronizing to the local from cloud, only when updating the shadow locally. Can you show the exception you see when changing the value in the cloud?

I want to make sure that the size limit is properly applied. To that end, if you have the local greengrass cli installed please run it with the command "components list". If you do not have the local cli, please shut down greengrass and then provide the effective config file from /greengrass/v2/config/effectiveConfig.yaml.

[crispinboylan-int]

there is currently nothing on the local side which would update this shadow. the exception happens during startup of greengrass when it is syncing the shadows.

[MikeDombo]

OK. I have a suspected issue. Please provide the effective config file or the output of the component list command.

After that, please try restarting Greengrass. If it works after a restart, then I know what the problem is.

[crispinboylan-int]

effectiveConfig.yaml.txt

[crispinboylan-int]

a restart does not help

[MikeDombo]

Thank you for that information. Unfortunately, since a restart did not help, it isn't what I suspected and thus more investigation will be required. We'll get back to you after attempting to reproduce your issue.

[crispinboylan-int]

i'm just wondering is it taking into accout the delta in this instance?

for example right now i'm getting the issue in the shadow with a payload of 19541 in the reported state, so there is a big delta. which taken into accout would blow the limit. we wont be updating the desired part currently so should be under the limit, excluding the delta of course.

but for another shadow I have 27,481 bytes across the reported and desired state and have no issues.

[rbattle]

It shouldn't be taking delta into account. I do see a discrepancy in how we are validating the size - it is looking at the size of the entire update request instead of just the size of the desired and reported nodes in the shadow after the update has been applied.

Is it possible for you to share the contents of a shadow and an update request so we can replicate this behavior and confirm a fix?

[cris-b]

have attached two updates:

the first is the one I initially made. This as accepted, but when synced got the too large error. the second I nulled two of the top level entries and then the shadow could sync.

neither is anywhere near the 30k limit, or even half that.

bad.json.txt ok.json.txt

[crispin@yossarian stacks]$ jq -c '.' bad.json.txt| wc -c 13831 [crispin@yossarian stacks]$ jq -c '.' ok.json.txt | wc -c 11821

[cris-b]

is there any schedule for the release of this fix?

[jbutler]

Hi @cris-b , sorry for the delay. This was released today in Shadow Manager version 2.2.4