feat: sample message too large to s3

[pauldambra]

We get MessageTooLarge error from Kafka, we know this means that the event content is too large for kafka, and based on how we process messages we can be pretty sure this is caused by full snapshots (which aren't chunk-up-able)

This adds a new sample rate config to dump a selection of these messages to S3 so we can analyse them

• the sample rate must be over 0
• it must be <= 0.001 (we want to be very sure that we don't overwhelm capture)
• we wrap the s3 upload in a try/except block to try (pun intended) not to break capture unexpectedly

I'm anticipating manually setting the environment variable and monitoring the bucket until we have >10 and <100 samples and then turning it off so we can validate the contents

once we've gathered enough samples to take some remedial steps in the SDK or during ingestion, we'll remove this code

[sentry-io[bot]]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ ObjectStorageError: write failed /s/[#].* View Issue

_{Did you find this useful? React with a 👍 or 👎}