SQS ingestion

[Samrose-Ahmed]

Tracking issue for SQS ingestion support.

Goal

Someone can send logs through SQS to Matano.

Design

• Likely have a separate ingest queue per log source.

• SQS has a max message size of 256 KB. For efficiency, we can store many log 'rows' in one SQS message (similar to how Kinesis packs messages).

• Aim to basically encode each message very similarly to an S3 file.

• Can also add support compression through base64 encoding.

Notes

• SQS ingestion can skip Data batcher (necessary for S3 to have predictable work sent to Transformer).
  • Instead, set SQS event source to max batch window (10000) and Lambda will send payload up to max 6MB.
• Transformer to be updated to be able to handle SQS messages. Lambda will be invoked with array of messages.
  • Implement a method very similar to current read_events_s3 (https://github.com/matanolabs/matano/blob/main/lib/rust/transformer/src/main.rs#L370) for SQS that converts the raw SQS messages into values.