search

BoKKeR / RSS-to-Telegram-Bot

RSS to Telegram python script
http://t.me/rss_t_bot
GNU General Public License v3.0
260 stars 102 forks source link

Repeatedly failing to send messages to telegram due to rate limit #40

Closed we11adam closed 1 year ago

we11adam commented 2 years ago

This happens when:

  1. A lot of new items are fetched.
  2. RSS-to-Telegram-Bot tries to send all the new items in a very short period of time.

Logs:


-------checking feed: <feed>----------
[Nest] 99  - 09/01/2022, 5:40:31 AM   DEBUG [RssService] last item: https://<redacted url>?id=632024&hit=1
[Nest] 99  - 09/01/2022, 5:40:31 AM   DEBUG [RssService] new elements: 200
[Nest] 99  - 09/01/2022, 5:40:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630660&hit=1
[Nest] 99  - 09/01/2022, 5:40:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630661&hit=1
[Nest] 99  - 09/01/2022, 5:40:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630684&hit=1
[Nest] 99  - 09/01/2022, 5:40:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630692&hit=1
[Nest] 99  - 09/01/2022, 5:40:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630704&hit=1
[Nest] 99  - 09/01/2022, 5:40:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630718&hit=1
[Nest] 99  - 09/01/2022, 5:40:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630726&hit=1
[Nest] 99  - 09/01/2022, 5:40:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630728&hit=1
[Nest] 99  - 09/01/2022, 5:40:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630770&hit=1
[Nest] 99  - 09/01/2022, 5:40:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630784&hit=1
[Nest] 99  - 09/01/2022, 5:40:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630786&hit=1
[Nest] 99  - 09/01/2022, 5:40:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630787&hit=1
[Nest] 99  - 09/01/2022, 5:40:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630788&hit=1
[Nest] 99  - 09/01/2022, 5:40:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630789&hit=1
[Nest] 99  - 09/01/2022, 5:40:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630790&hit=1
[Nest] 99  - 09/01/2022, 5:40:44 AM   DEBUG [RssService] sending: https://<redacted url>?id=630792&hit=1
[Nest] 99  - 09/01/2022, 5:40:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630799&hit=1
[Nest] 99  - 09/01/2022, 5:40:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630802&hit=1
[Nest] 99  - 09/01/2022, 5:40:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630804&hit=1
[Nest] 99  - 09/01/2022, 5:40:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630818&hit=1
[Nest] 99  - 09/01/2022, 5:40:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630826&hit=1
[Nest] 99  - 09/01/2022, 5:40:50 AM   ERROR [Scheduler] Error: 429: Too Many Requests: retry after 42
[Nest] 99  - 09/01/2022, 5:45:31 AM   DEBUG [RssService]

-------checking feed: <feed>----------
[Nest] 99  - 09/01/2022, 5:45:31 AM   DEBUG [RssService] last item: https://<redacted url>?id=632024&hit=1
[Nest] 99  - 09/01/2022, 5:45:31 AM   DEBUG [RssService] new elements: 200
[Nest] 99  - 09/01/2022, 5:45:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630660&hit=1
[Nest] 99  - 09/01/2022, 5:45:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630661&hit=1
[Nest] 99  - 09/01/2022, 5:45:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630684&hit=1
[Nest] 99  - 09/01/2022, 5:45:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630692&hit=1
[Nest] 99  - 09/01/2022, 5:45:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630704&hit=1
[Nest] 99  - 09/01/2022, 5:45:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630718&hit=1
[Nest] 99  - 09/01/2022, 5:45:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630726&hit=1
[Nest] 99  - 09/01/2022, 5:45:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630728&hit=1
[Nest] 99  - 09/01/2022, 5:45:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630770&hit=1
[Nest] 99  - 09/01/2022, 5:45:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630784&hit=1
[Nest] 99  - 09/01/2022, 5:45:39 AM   DEBUG [RssService] sending: https://<redacted url>?id=630786&hit=1
[Nest] 99  - 09/01/2022, 5:45:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630787&hit=1
[Nest] 99  - 09/01/2022, 5:45:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630788&hit=1
[Nest] 99  - 09/01/2022, 5:45:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630789&hit=1
[Nest] 99  - 09/01/2022, 5:45:44 AM   DEBUG [RssService] sending: https://<redacted url>?id=630790&hit=1
[Nest] 99  - 09/01/2022, 5:45:44 AM   DEBUG [RssService] sending: https://<redacted url>?id=630792&hit=1
[Nest] 99  - 09/01/2022, 5:45:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630799&hit=1
[Nest] 99  - 09/01/2022, 5:45:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630802&hit=1
[Nest] 99  - 09/01/2022, 5:45:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630804&hit=1
[Nest] 99  - 09/01/2022, 5:45:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630818&hit=1
[Nest] 99  - 09/01/2022, 5:45:50 AM   DEBUG [RssService] sending: https://<redacted url>?id=630826&hit=1
[Nest] 99  - 09/01/2022, 5:45:50 AM   ERROR [Scheduler] Error: 429: Too Many Requests: retry after 41
[Nest] 99  - 09/01/2022, 5:50:31 AM   DEBUG [RssService]

-------checking feed: <feed>----------
[Nest] 99  - 09/01/2022, 5:50:31 AM   DEBUG [RssService] last item: https://<redacted url>?id=632024&hit=1
[Nest] 99  - 09/01/2022, 5:50:31 AM   DEBUG [RssService] new elements: 200
[Nest] 99  - 09/01/2022, 5:50:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630660&hit=1
[Nest] 99  - 09/01/2022, 5:50:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630661&hit=1
[Nest] 99  - 09/01/2022, 5:50:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630684&hit=1
[Nest] 99  - 09/01/2022, 5:50:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630692&hit=1
[Nest] 99  - 09/01/2022, 5:50:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630704&hit=1
[Nest] 99  - 09/01/2022, 5:50:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630718&hit=1
[Nest] 99  - 09/01/2022, 5:50:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630726&hit=1
[Nest] 99  - 09/01/2022, 5:50:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630728&hit=1
[Nest] 99  - 09/01/2022, 5:50:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630770&hit=1
[Nest] 99  - 09/01/2022, 5:50:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630784&hit=1
[Nest] 99  - 09/01/2022, 5:50:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630786&hit=1
[Nest] 99  - 09/01/2022, 5:50:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630787&hit=1
[Nest] 99  - 09/01/2022, 5:50:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630788&hit=1
[Nest] 99  - 09/01/2022, 5:50:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630789&hit=1
[Nest] 99  - 09/01/2022, 5:50:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630790&hit=1
[Nest] 99  - 09/01/2022, 5:50:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630792&hit=1
[Nest] 99  - 09/01/2022, 5:50:47 AM   DEBUG [RssService] sending: https://<redacted url>?id=630799&hit=1
[Nest] 99  - 09/01/2022, 5:50:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630802&hit=1
[Nest] 99  - 09/01/2022, 5:50:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630804&hit=1
[Nest] 99  - 09/01/2022, 5:50:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630818&hit=1
[Nest] 99  - 09/01/2022, 5:50:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630826&hit=1
[Nest] 99  - 09/01/2022, 5:50:49 AM   ERROR [Scheduler] Error: 429: Too Many Requests: retry after 42
[Nest] 99  - 09/01/2022, 5:55:31 AM   DEBUG [RssService]

-------checking feed: <feed>----------
[Nest] 99  - 09/01/2022, 5:55:31 AM   DEBUG [RssService] last item: https://<redacted url>?id=632024&hit=1
[Nest] 99  - 09/01/2022, 5:55:31 AM   DEBUG [RssService] new elements: 200
[Nest] 99  - 09/01/2022, 5:55:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630660&hit=1
[Nest] 99  - 09/01/2022, 5:55:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630661&hit=1
[Nest] 99  - 09/01/2022, 5:55:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630684&hit=1
[Nest] 99  - 09/01/2022, 5:55:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630692&hit=1
[Nest] 99  - 09/01/2022, 5:55:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630704&hit=1
[Nest] 99  - 09/01/2022, 5:55:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630718&hit=1
[Nest] 99  - 09/01/2022, 5:55:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630726&hit=1
[Nest] 99  - 09/01/2022, 5:55:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630728&hit=1
[Nest] 99  - 09/01/2022, 5:55:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630770&hit=1
[Nest] 99  - 09/01/2022, 5:55:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630784&hit=1
[Nest] 99  - 09/01/2022, 5:55:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630786&hit=1
[Nest] 99  - 09/01/2022, 5:55:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630787&hit=1
[Nest] 99  - 09/01/2022, 5:55:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630788&hit=1
[Nest] 99  - 09/01/2022, 5:55:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630789&hit=1
[Nest] 99  - 09/01/2022, 5:55:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630790&hit=1
[Nest] 99  - 09/01/2022, 5:55:44 AM   DEBUG [RssService] sending: https://<redacted url>?id=630792&hit=1
[Nest] 99  - 09/01/2022, 5:55:47 AM   DEBUG [RssService] sending: https://<redacted url>?id=630799&hit=1
[Nest] 99  - 09/01/2022, 5:55:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630802&hit=1
[Nest] 99  - 09/01/2022, 5:55:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630804&hit=1
[Nest] 99  - 09/01/2022, 5:55:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630818&hit=1
[Nest] 99  - 09/01/2022, 5:55:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630826&hit=1
[Nest] 99  - 09/01/2022, 5:55:50 AM   ERROR [Scheduler] Error: 429: Too Many Requests: retry after 41
[Nest] 99  - 09/01/2022, 6:00:31 AM   DEBUG [RssService]

-------checking feed: <feed>----------
[Nest] 99  - 09/01/2022, 6:00:31 AM   DEBUG [RssService] last item: https://<redacted url>?id=632024&hit=1
[Nest] 99  - 09/01/2022, 6:00:31 AM   DEBUG [RssService] new elements: 200
[Nest] 99  - 09/01/2022, 6:00:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630660&hit=1
[Nest] 99  - 09/01/2022, 6:00:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630661&hit=1
[Nest] 99  - 09/01/2022, 6:00:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630684&hit=1
[Nest] 99  - 09/01/2022, 6:00:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630692&hit=1
[Nest] 99  - 09/01/2022, 6:00:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630704&hit=1
[Nest] 99  - 09/01/2022, 6:00:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630718&hit=1
[Nest] 99  - 09/01/2022, 6:00:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630726&hit=1
[Nest] 99  - 09/01/2022, 6:00:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630728&hit=1
[Nest] 99  - 09/01/2022, 6:00:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630770&hit=1
[Nest] 99  - 09/01/2022, 6:00:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630784&hit=1
[Nest] 99  - 09/01/2022, 6:00:39 AM   DEBUG [RssService] sending: https://<redacted url>?id=630786&hit=1
[Nest] 99  - 09/01/2022, 6:00:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630787&hit=1
[Nest] 99  - 09/01/2022, 6:00:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630788&hit=1
[Nest] 99  - 09/01/2022, 6:00:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630789&hit=1
[Nest] 99  - 09/01/2022, 6:00:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630790&hit=1
[Nest] 99  - 09/01/2022, 6:00:44 AM   DEBUG [RssService] sending: https://<redacted url>?id=630792&hit=1
[Nest] 99  - 09/01/2022, 6:00:47 AM   DEBUG [RssService] sending: https://<redacted url>?id=630799&hit=1
[Nest] 99  - 09/01/2022, 6:00:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630802&hit=1
[Nest] 99  - 09/01/2022, 6:00:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630804&hit=1
[Nest] 99  - 09/01/2022, 6:00:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630818&hit=1
[Nest] 99  - 09/01/2022, 6:00:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630826&hit=1
[Nest] 99  - 09/01/2022, 6:00:50 AM   ERROR [Scheduler] Error: 429: Too Many Requests: retry after 42
[Nest] 99  - 09/01/2022, 6:05:31 AM   DEBUG [RssService]

-------checking feed: <feed>----------
[Nest] 99  - 09/01/2022, 6:05:31 AM   DEBUG [RssService] last item: https://<redacted url>?id=632024&hit=1
[Nest] 99  - 09/01/2022, 6:05:31 AM   DEBUG [RssService] new elements: 200
[Nest] 99  - 09/01/2022, 6:05:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630660&hit=1
[Nest] 99  - 09/01/2022, 6:05:31 AM   DEBUG [RssService] sending: https://<redacted url>?id=630661&hit=1
[Nest] 99  - 09/01/2022, 6:05:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630684&hit=1
[Nest] 99  - 09/01/2022, 6:05:32 AM   DEBUG [RssService] sending: https://<redacted url>?id=630692&hit=1
[Nest] 99  - 09/01/2022, 6:05:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630704&hit=1
[Nest] 99  - 09/01/2022, 6:05:33 AM   DEBUG [RssService] sending: https://<redacted url>?id=630718&hit=1
[Nest] 99  - 09/01/2022, 6:05:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630726&hit=1
[Nest] 99  - 09/01/2022, 6:05:37 AM   DEBUG [RssService] sending: https://<redacted url>?id=630728&hit=1
[Nest] 99  - 09/01/2022, 6:05:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630770&hit=1
[Nest] 99  - 09/01/2022, 6:05:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630784&hit=1
[Nest] 99  - 09/01/2022, 6:05:38 AM   DEBUG [RssService] sending: https://<redacted url>?id=630786&hit=1
[Nest] 99  - 09/01/2022, 6:05:42 AM   DEBUG [RssService] sending: https://<redacted url>?id=630787&hit=1
[Nest] 99  - 09/01/2022, 6:05:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630788&hit=1
[Nest] 99  - 09/01/2022, 6:05:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630789&hit=1
[Nest] 99  - 09/01/2022, 6:05:43 AM   DEBUG [RssService] sending: https://<redacted url>?id=630790&hit=1
[Nest] 99  - 09/01/2022, 6:05:44 AM   DEBUG [RssService] sending: https://<redacted url>?id=630792&hit=1
[Nest] 99  - 09/01/2022, 6:05:47 AM   DEBUG [RssService] sending: https://<redacted url>?id=630799&hit=1
[Nest] 99  - 09/01/2022, 6:05:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630802&hit=1
[Nest] 99  - 09/01/2022, 6:05:48 AM   DEBUG [RssService] sending: https://<redacted url>?id=630804&hit=1
[Nest] 99  - 09/01/2022, 6:05:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630818&hit=1
[Nest] 99  - 09/01/2022, 6:05:49 AM   DEBUG [RssService] sending: https://<redacted url>?id=630826&hit=1
[Nest] 99  - 09/01/2022, 6:05:49 AM   ERROR [Scheduler] Error: 429: Too Many Requests: retry after 42```
BoKKeR commented 2 years ago

Check the develop branch bokker/rss.to.telegram:develop, for now I just added some space between each request but I will have to rework the project to use a queue where each job can be retried if failed

luc-ass commented 1 year ago

Was just about to report the same issue. Good to see, that this is already worked on. Keep up the great work! 🚀

BoKKeR commented 1 year ago

This is solved with the queue implementation in develop.