logging a lot of data quickly causes a ton of errors

[aroman]

any idea how to get logtail to gracefully handle and buffer this, or else just drop messages if the queue is too big?

[aroman]

also, i'm not sure what the message there about "access control checks" has to do with anything — in fact, the logs are totally being accepted by Logtail (confirmed via logtail dashboard and looking at the network tab and seeing 202 ACCEPTED responses for the Logtail XHR requests....)

[aroman]

setting burstProtectionMax: 20 and burstProtectionMilliseconds: 1000 resolves this issue. that said, the error messages here are quite confusing (and on Chrome, you don't even get the note about 'reached max queued data' — just Safari, so it's even more confusing)

update: I can also mitigate this with batchSize: 10. it seems that my messages are very large, so the issue is not so much the frequency of messages being sent, but actually the size of each XHR request (containing multiple messages)

[curusarn]

Hi @aroman,

thank you for creating the issue! I appreciate it.

We definitely want the library to handle large messages and send them to Better Stack without errors.

My first impression is that we could change batching configuration to take the actual size of messages into account. I.e. current batchSize is in "number of log lines" but having e.g. batchSizeKb would result in logs being sent more often and not hitting the browser limits.

I'll pass this on to the team for someone to take a look.

Thanks again for rasing this!

[PetrHeinz]

Hi @aroman, thanks for reporting this 🙌

I've fixed it in v0.4.19. After reaching 48 KiB of data in queue, the queue gets flushed automatically. See #102 for details.

Tested it on this JS snippet:

const logtail = new Logtail("MY_SOURCE_TOKEN");

logtail.info("request 1 " + "X".repeat(10000))
logtail.info("request 2 " + "X".repeat(10000))
logtail.info("request 3 " + "X".repeat(10000))
logtail.info("request 4 " + "X".repeat(10000))
logtail.info("request 5 " + "X".repeat(10000))
logtail.info("request 6 " + "X".repeat(10000))
logtail.info("request 7 " + "X".repeat(10000))
logtail.info("request 8 " + "X".repeat(10000))
logtail.info("request 9 " + "X".repeat(10000))

Previous version of the library triggered the Reached maximum amount of queued data of 64Kb for keepalive requests error as you mentioned.

<script src="https://cdnjs.cloudflare.com/ajax/libs/logtail-browser/0.4.17/dist/umd/logtail.min.js" integrity="sha512-ULxpvdXTYsKoCUAaKxnqE+R/C9+2SaOs/xrrSQ3RKjIfrahLecA98mcTs82TXgtNdjQ+8y2Rf5EeyYC8djBz4g==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

After the update, data got sent to Better Stack without errors:

<script src="https://cdnjs.cloudflare.com/ajax/libs/logtail-browser/0.4.19/dist/umd/logtail.min.js" integrity="sha512-EYdnWL7Lrn+96R4BDaOxCgCWWOoZQiMSsfy+3NKYSJxK/CAt6y7cPLvGHRWONYO8Y0SsGuk5Y+lC1w3up7f7OA==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

---

There are still some cases where the queue can exceed the 64 KiB limit, but hopefully should not occur in real-case scenarios. In that case, we would have to make the queuing process bit more complicated and/or drop logs.