orangecoding
commented
2 years ago @daniel17903 Thanks for this pr.
I however see the use of those kind of delays a little bit sceptical. Because of the single threaded nature of javascript, blocking this is potentially blocking Fredy completely. Depending on how short the crawling interval has been set, we could potentially end up in a situation, where we wait longer than it takes for the next run.
I would have a different proposal.
First of, I have checked again the max size of a message and notice, it has been changed to 4096 chars (it was half the size before), thus I would go up with the number of entries within a chunk. With the messages we are sending, we can easily go up to 8 entries per chunk (potentially more) and we would still be under 4kb.
Next I would listen to the HTTP response instead of slowing down the whole process directly. If we are hitting 429, we wait a second and retry. The retry happens a max of 3 times with exponential backoff (maybe up to 10 seconds) and if it still doesn't work, we log this as an error.
I am pretty sure with these changes, we will never end up in a situation, where we send too many messages a minute. (Or better; the ONLY situation where something like this might happen would be if we crawl some page the first time, thus have a bunch of results)
What do you think?
In order to prevent HTTP 429 responses the telegram adapter should not send more than 1 message per second and not more than 20 messsages per minute (see https://core.telegram.org/bots/faq#my-bot-is-hitting-limits-how-do-i-avoid-this).
57