search

d-Rickyy-b / certstream-server-go

This project aims to be a drop-in replacement for the certstream server by Calidog. This tool aggregates, parses, and streams certificate data from multiple certificate transparency logs via websocket connections to the clients.
MIT License
88 stars 8 forks source link

Not providing client X with cert because our buffer is full. The client can't keep up. #28

Closed HugooB closed 9 months ago

HugooB commented 9 months ago

Hi there! Great work on this! I relied heavily on Calidogs server and since that is offline, this seems to be a perfect replacement. I setup your certstream-server and changed the Websocket address in my certstream implementation to ws://localhost:8080. It works, but the server gives me this error:

2023/12/12 13:46:41 broadcastmanager.go:99: Not providing client 'xxxxxxx:57142' with cert because our buffer is full. The client can't keep up.

Any thoughts why?

robj192 commented 9 months ago

Great work @d-Rickyy-b. I'm in the same situation as @HugooB's context. I tried your tool and it seems to work great, but I'm also getting the same output.

Following this thread. Cheers

d-Rickyy-b commented 9 months ago

Hi there, happy you enjoy the project. This error message is shown when the certstream cilent can't process all the certificates the certstream server receives from the CT network. We're talking ~250-300 certs per second.

Possible reasons are usually slow clients, meaning either a slow network connection or a slow programming language. Alternatively this might also happen if your server has a bad network connection or if the server is trying to provide certificates to too many clients.

I assume you are using only one or a handful of clients. In case you are using Python: Python is quite a slow language by default. Make sure you either use asyncio or multiple threads. I once wrote certleak as a multithreaded python certstream client. Maybe you want to try out that as well.

Apart from that, I will make the per client buffer size configurable. In certain cases a client might be slow for a certain period of time (network hickup) and will be able to fetch the missed certs at a quicker pace later on. I tried to find a sane default and set the client buffer to a size of max. 100 certificates:

https://github.com/d-Rickyy-b/certstream-server-go/blob/0dd05827a591724e2a1dbeac6329abc3b1bdecd9/internal/web/server.go#L169-L176

d-Rickyy-b commented 9 months ago

Also make sure to check to which endpoint you are connecting to. If you only need domains, connect to the /domains-only endpoint. That saves a lot of bandwidth and parsing.

HugooB commented 9 months ago

Thanks for your swift reply! I'm indeed using a single threaded Python client and I'm not surprised that it wasn't able to handle 250-300 certs per second. I will definitely take a look at your certleak as well, great stuff man!

HugooB commented 9 months ago

Nice job! Are you going to release a new version soon? Also at Docker Hub?

d-Rickyy-b commented 9 months ago

Sure thing, probably by the end of the week if time allows.

d-Rickyy-b commented 9 months ago

New release published: https://github.com/d-Rickyy-b/certstream-server-go/releases/tag/v1.5.0

Also please let me know if the issue is fixed for you (even though I think it's not a server side issue).

uggyuggy commented 9 months ago

Hi,

please let me know if the issue is fixed for you

Thank you very much for sharing this software.

With 1.4.0, I had a lot of the buffer is full errors whatever buffer and settings values I tried to adjust into the client (running on localhost). With the new 1.5.0, and after 1 hour and processed over 1 million entries, I have not seen yet the buffer is full errors for now.

So at least for me, for now, I see a huge difference with what you changed into 1.5.0 👍 Need more tests over more time... Will post here again if error is back.