search

CaliDog / certstream-server

Certificate Transparency Log aggregation, parsing, and streaming service written in Elixir
https://certstream.calidog.io
MIT License
271 stars 75 forks source link

Ability to search historical data [Question/Feature Request] #33

Closed gmkbenjamin closed 4 years ago

gmkbenjamin commented 4 years ago

This is an awesome product with a lot of great use cases.

It would be super useful to be able to fetch and search historical data.

realalexandergeorgiev commented 4 years ago

I tried doing this for a subset of the data (DNS) here: https://pdns.daloo.de

Edit: now at https://passivedns.daloo.de

Fitblip commented 4 years ago

Hi @gmkbenjamin I agree, historical DNS searching is a great thing to have, though I see certstream as more a feeding stream into a larger aggregation and storage service, and to keep things simple/flexible we’re probably not going to build in historical retention or anything like that.

That said, pull requests are always welcome and if you (or someone else) want to take the time to scope out things and do the legwork, I’m sure we can figure something out (I.e. I’d probably be fine with a complicated feature like this if someone else does the legwork and it’s well-written, but I don’t think I’m going to dedicate my own cycles to it).

Fitblip commented 4 years ago

Also @realalexandergeorgiev - that service is very cool! Good work :)

mschfh commented 1 year ago

@Fitblip Is increasing the buffer size (from 25 certs) an alternative? (to allow backfilling the latest certificates after a connection loss?)