search

d-Rickyy-b / certstream-server-go

This project aims to be a drop-in replacement for the certstream server by Calidog. This tool aggregates, parses, and streams certificate data from multiple certificate transparency logs via websocket connections to the clients.
MIT License
88 stars 8 forks source link

Add SQS / SNS Support #35

Open goliel opened 7 months ago

goliel commented 7 months ago

Websockets are not very good when you want to process a lot of messages in a large pipeline due to the lack of consumers that can process the messages.

Would like to see some kind of SQS support, this way multiple consumers can subscribe and process the messages as scale out rather than scale up.

d-Rickyy-b commented 7 months ago

This project was initially thought of as a 1:1 replacement for the original certstream server. I thought about implementing something like Kafka support myself, additionally to websockets.

I haven't worked with SNS / SQS yet so I can't say if that's something I'll consider adding. But I'll definitely have a look into it.

Cheers

alberts-s commented 3 months ago

When I first found this repository I thought at first that having support for a queue (e.g SQS) would be useful.

However after giving this more though - maybe it would be better to prioritize adding support for storing the messages in an S3-compatible file store? Combining this with the out of the box feature to receive S3 events in SQS (guide here) it would also accomplish the goal of supporting SQS. GCP supports similar mechanism out of the box too.

Moreover supporting an S3-compatible store would allow also to easily query results using AWS Athena/Glue and GCP BigQuery.

meinder-a commented 1 week ago

Well, I thought that having this feature could make it easier to implement other distributed message-queuing services like Kafka, which is also widely used. I'm not entirely sure though. I like the concept of uploading messages to S3, but I’d prefer a small script that reads the messages using Kafka and uploads them to S3 afterwards.