search

tailscale / tailscale

The easiest, most secure way to use WireGuard and 2FA.
https://tailscale.com
BSD 3-Clause "New" or "Revised" License
19.52k stars 1.53k forks source link

State File Size May Exceed AWS SSM Standard Param Size #6784

Closed jglien closed 1 year ago

jglien commented 1 year ago

What is the issue?

Tailscale is hard-coded to only support using a standard AWS SSM parameter to store state.

The size limit for an AWS SSM standard parameter is 4KB, however, we have observed that the state file may exceed that limit, requiring the parameter be upgraded to an advanced parameter which supports up to 8KB.

Steps to reproduce

Using a subnet router that is regularly updated and stores state in AWS SSM, eventually the state file will exceed the limit of 4KB. The state file will then fail to update with a size exceeded error, or fail with a cannot downgrade to standard error if the parameter is upgraded to advanced manually.

Are there any recent changes that introduced the issue?

No response

OS

Linux, Other

OS version

Docker

Tailscale version

1.34.1

Bug report

BUG-739b33cd6db22952785195c6b80c59500b4e99e4391672daa3077e3d448cf13f-20221219190832Z-1736d598881ecbb9

DentonGentry commented 1 year ago

If we can figure out how to express it in aws-sdk-go-v2, letting the parameter store figure out whether to use Standard or Advanced seems pretty compelling.

https://aws.amazon.com/about-aws/whats-new/2019/08/aws-systems-manager-parameter-store-announces-intelligent-tiering-to-enable-automatic-parameter-tier-selection/

DentonGentry commented 1 year ago

Probably ParameterTierIntelligentTiering

https://github.com/evalphobia/aws-sdk-go-v2-wrapper/blob/b5915d66f6a5f4841d42e141979e32d5d477ac76/ssm/type_enum.go#L12

marwan-at-work commented 1 year ago

I confirmed in AWS that changing to intelligent tiering can indeed upgrade the state to "advanced" allowing the state to have a size <= 8kb. Hence the PR linked above should be a safe change.

However, that only increases the capacity a few more kilobytes which our state file can easily exceed. My two follow up suggestions would be:

  1. Store each parameter separately, AWS docs they give you up to 10 thousand parameters in standard tiering for free, so it should actually be a money saver.
  2. Provide an S3 implementation that users can switch to if the Param Store is not suitable for their use case.