search

CaliDog / certstream-server

Certificate Transparency Log aggregation, parsing, and streaming service written in Elixir
https://certstream.calidog.io
MIT License
271 stars 75 forks source link

Keeping the CT server list up to date without restarting certstream-server #97

Closed mzpqnxow closed 1 year ago

mzpqnxow commented 2 years ago

Hello, I was looking through the source to better understand how certstream-server worked and I had a question that I thought you might be able to easily answer

I see that certstream-server uses the metadata that Google maintains when it bootstraps, to get the list of CT server URLs:

https://github.com/CaliDog/certstream-server/blob/41c054704316f9ade21a0cc89db19d51e10469e6/lib/certstream/ct_watcher.ex#L22-L28

What are your thoughts on how to handle when a new CT server may be added to (or removed from) this list? Do you think this should be entirely up to the operator of certstream-server? Or do you think it would be appropriate to include some mechanism within certstream-server to intelligently detect and handle that case, maybe by polling every 24h?

I see that you use Supervisor as a high-level management mechanism/interface for the workers. Perhaps that offers some clean way to do this, either via a signal from an external process, or via a dedicated worker that could be part of certstream-server, whose job it is to simply monitor for changes in all_logs_list.json, and then either terminate a worker or add a new worker when a CT server has been removed or added to the all_logs_list.json

It's certainly not a major problem to just kick the server over every week, month, etc. via cron (or whatever) to ensure that there's always a fresh set of CT servers being polled. But it would be nice to not have to worry about that

Thanks again for your work on this project (as well as certstream-python) - it has significantly simplified a project of mine which had a very clunky solution in place for trying to track CT data

hendursaga commented 1 year ago

It's certainly not a major problem to just kick the server over every week, month, etc. via cron (or whatever) to ensure that there's always a fresh set of CT servers being polled.

It IS Elixir, is it not? "Let it crash!" :smile: