Buffer Full - Githubissues

Rceros commented 1 month ago

Hello,

I use python code to connect :

process = subprocess.Popen( ['certstream', '--url', 'ws://127.0.0.1:8080'], stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True ) output_pattern = re.compile(r'[(.?)] (https://.?) - (.*?)$')

try:
    for line in process.stdout:
        match = output_pattern.match(line.strip())
        if match:
            time, log_url, domain = match.groups()
            json_entry = {
                "id": id_counter,
                "time": time,
                "log_url": log_url,
                "domaine": domain

My python code transforme output in real time into JSON --> database.

After a few minutes of use, I get this error message and the logs pause :

2024/06/04 00:06:31 broadcastmanager.go:99: Not providing client '10.10..:54222' with cert because our buffer is full. The client can't keep up.

Is this normal and how can it be corrected please?

d-Rickyy-b commented 1 month ago

Hi Rceros, I assume that your python code is just not fast enough to process all the certificates, since certstream handles ~250-300 certs per second. Please read my answer to a similar question: https://github.com/d-Rickyy-b/certstream-server-go/issues/28#issuecomment-1852523034

gcleaves commented 1 month ago

In my case I had to run 5 parallel node workers to handle the stream.

Rceros commented 1 month ago

@d-Rickyy-b many thanks,

I've noticed other things too, do you know what they're linked to?

https://yeti2025.ct.digicert.com/log: GetRawEntries() failed: Get "https://yeti2025.ct.digicert.com/log/ct/v1/get-entries?end=107046060&start=107045961": unexpected EOF

Are we losing logs with this?

Rceros commented 1 month ago

In my case I had to run 5 parallel node workers to handle the stream.

How do you do it? Do you have any code examples?

Do you use python to do it or Go?

please help

d-Rickyy-b commented 1 month ago

I've noticed other things too, do you know what they're linked to?

The error comes from the certificate-transparency-go module that's used by certstream-server-go (click).

These logs happen when either the log returns a 429 error, indicating that we're requesting too many certificates (although this is not the case here) or that the upstream API (the certificate transparency server of digicert in your case) returned an error or sent invalid/malformed data. "unexpected EOF" usually means that the server did not respond in an expected way or closed the connection before fully transferring the requested data.

Based on the test file client_test.go test file it can be understood that the unexpected EOF is thrown when the json client tries to parse a json string that's not valid.

From my past experience, digicert logs are causing regular issues. See also:

Hope this helps.

d-Rickyy-b commented 1 month ago

How do you do it? Do you have any code examples?

I created a python tool for this purpose (certleak). I am currently not actively using it so I can't tell you if it's still up for the task, but it worked fine in the past.

Feel free to try it out and send your feedback my way.

d-Rickyy-b / certstream-server-go

Buffer Full #37