search

plausible / plausible-tracker

Frontend library to interact with Plausible Analytics
https://github.com/plausible/plausible-tracker
MIT License
214 stars 46 forks source link

Exception when Plausible is slow to respond #63

Closed dommitchell closed 7 months ago

dommitchell commented 7 months ago

Versions

Describe the bug

https://plausible.io/doaj.org

First detected via one of our internals alerts at 0100h on Sunday 10th Dec 20243 (internal: a 500 response from our OAI-PMH monitor).

We were getting a RuntimeError: can't start new thread exception from doaj-public-app-1 from the Plausible logger, which has only started occurring in the last few days. See error in Kibana:

doaj-download

The pattern here is that the exception seems to occur (increasingly in frequency) each night after 0000h, and is usually sorted by the morning. This is currently the most frequent error in our monitoring, and we'll be missing events during that period.

Does Plausible take longer to respond during that period overnight, resulting in a backlog of threads to send events?

Confirmed today that this is still happening regularly every 24 hours.

image (1)

Expected behaviour

We expect to see no exception raised.

Steps to reproduce

N/A

Your Environment

N/A

Additional context

dommitchell commented 7 months ago

At our end, to fix the unhandled exception, we released a hotfix to log when the exception occurs, so we don't get a 500 from the site for the sake of analytics.

cnkk commented 7 months ago

Hi @dommitchell!

Unfortunately the first screenshot you posted won't load.

Are we talking about our Tracking Script or Events API. Also, the error message you posted looks like a Python error, are you sure it's related to this repo?

Anyway, I haven't seen any problems on our end with the problem you describe happening around midnight.

dommitchell commented 7 months ago

Hi @cnkk I swapped out the image so hopefully you can see that now. This is for the Events API. That error message is generated by our system and I got it from our developer. I'm going to call in https://github.com/Steven-Eardley and hope that he can help here. I am sure I've introduced confusion by relaying the message. We'll be back...

Steven-Eardley commented 7 months ago

That's right, we're logging to the events API - spinning off threads to register our own API hits etc. The error can't start new thread occurs because the request latency to Plausible increases at around 0000h, and recovers during the day.

We presume this is a performance issue at Plausible's end because we are sending requests at a sustained rate, but at certain times of day the response is slow.

cnkk commented 7 months ago

As far as I can see your problem is not related to the code in this repository.

Could you please contact us via mail hello@plausible.io