How query logs should be queued for batch INSERT INTO?

[ghalimi]

Every query must be logged into an Iceberg table using an INSERT INTO query. Batching multiple such queries into one would make it more efficient, but would require some queuing mechanism. Since low latency is not an absolute requirement for query logs, Amazon SQS could be used for such a purpose, but should other options be considered?

[ghalimi]

Using an Iceberg table for logs is probably adding more complexity than is necessary. Amazon ElastiCache for Redis is probably a better option.

[alexey-milovidov]

I don't think it's the best approach. For example, ClickHouse logs the queries directly into ClickHouse (itself) into the system.query_log table. That's how they can be analyzed with low latency.

[ghalimi]

We're actually going in that direction now.