Telegram notifications quickly run into rate limit

[sven-simonsen]

Is your feature request related to a problem? Please describe. I have set up a search with 4 of the big flat search providers and set up a Telegram bot to inform me and my girlfriend of the new flats it finds. At least when sending to a channel the telegram bot has a rate limit of only 10 messages every minute, and fredy regularly goes over this when I turn it on in the morning. Also I would prefer to set the bots messages to only have 1 flat per message, so we can delete uninteresting messages easily, but if I change the code setting for how many flats get sent in a message I go over the rate limit even faster.

Describe the solution you'd like I would love it if fredy would know of the telegram rate limit and wait for a minute every 9 messages to slowly but error free feed me my flat search results. Regrettably my Typescript skills are not up to implement this change.

Describe alternatives you've considered If this is not possible I might just set up a telegram bot for every search provider. It seems that Fredy is not crosschecking for duplicates between providers anyway?

Additional context

Error: Request failed with status code 429
    at createError (D:\dev\experiments\fredy\node_modules\axios\lib\core\createError.js:16:15)
    at settle (D:\dev\experiments\fredy\node_modules\axios\lib\core\settle.js:17:12)
    at IncomingMessage.handleStreamEnd (D:\dev\experiments\fredy\node_modules\axios\lib\adapters\http.js:322:11)
    at IncomingMessage.emit (node:events:538:35)
    at endReadableNT (node:internal/streams/readable:1345:12)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [Function: httpAdapter],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    validateStatus: [Function: validateStatus],
    headers: {
      Accept: 'application/json, text/plain, */*',
      'Content-Type': 'application/json',
      'User-Agent': 'axios/0.26.1',
      'Content-Length': 333
    },
    method: 'post',
    url: 'https://api.telegram.org/bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage',
    data: '{"chat_id":"-677533133","text":"<i>Mietwohnung um Bad Oldesloe</i> (immonet) found <b>3</b> new listings:\\n\\n<a href=\\"https://www.immonet.de/angebot/47748318\\"><b>HH-Harburg Zentrum: 3 Zimmer Whg. mit Balkon</b></a>\\nHamburg Harburg | Miete zzgl. NK 1.200 € | ca. 89.0 m²\\n\\n","parse_mode":"HTML","disable_web_page_preview":true}',
    'axios-retry': { retryCount: 0, lastRequestTime: 1657048583661 }
  },
  request: <ref *1> ClientRequest {
    _events: [Object: null prototype] {
      abort: [Function (anonymous)],
      aborted: [Function (anonymous)],
      connect: [Function (anonymous)],
      error: [Function (anonymous)],
      socket: [Function (anonymous)],
      timeout: [Function (anonymous)],
      prefinish: [Function: requestOnPrefinish]
    },
    _eventsCount: 7,
    _maxListeners: undefined,
    outputData: [],
    outputSize: 0,
    writable: true,
    destroyed: false,
    _last: true,
    chunkedEncoding: false,
    shouldKeepAlive: false,
    maxRequestsOnConnectionReached: false,
    _defaultKeepAlive: true,
    useChunkedEncodingByDefault: true,
    sendDate: false,
    _removedConnection: false,
    _removedContLen: false,
    _removedTE: false,
    _contentLength: null,
    _hasBody: true,
    _trailer: '',
    finished: true,
    _headerSent: true,
    _closed: false,
    socket: TLSSocket {
      _tlsOptions: [Object],
      _secureEstablished: true,
      _securePending: false,
      _newSessionPending: false,
      _controlReleased: true,
      secureConnecting: false,
      _SNICallback: null,
      servername: 'api.telegram.org',
      alpnProtocol: false,
      authorized: true,
      authorizationError: null,
      encrypted: true,
      _events: [Object: null prototype],
      _eventsCount: 10,
      connecting: false,
      _hadError: false,
      _parent: null,
      _host: 'api.telegram.org',
      _readableState: [ReadableState],
      _maxListeners: undefined,
      _writableState: [WritableState],
      allowHalfOpen: false,
      _sockname: null,
      _pendingData: null,
      _pendingEncoding: '',
      server: undefined,
      _server: null,
      ssl: [TLSWrap],
      _requestCert: true,
      _rejectUnauthorized: true,
      parser: null,
      _httpMessage: [Circular *1],
      [Symbol(res)]: [TLSWrap],
      [Symbol(verified)]: true,
      [Symbol(pendingSession)]: null,
      [Symbol(async_id_symbol)]: 2443,
      [Symbol(kHandle)]: [TLSWrap],
      [Symbol(kSetNoDelay)]: false,
      [Symbol(lastWriteQueueSize)]: 0,
      [Symbol(timeout)]: null,
      [Symbol(kBuffer)]: null,
      [Symbol(kBufferCb)]: null,
      [Symbol(kBufferGen)]: null,
      [Symbol(kCapture)]: false,
      [Symbol(kBytesRead)]: 0,
      [Symbol(kBytesWritten)]: 0,
      [Symbol(connect-options)]: [Object],
      [Symbol(RequestTimeout)]: undefined
    },
    _header: 'POST /bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage HTTP/1.1\r\n' +
      'Accept: application/json, text/plain, */*\r\n' +
      'Content-Type: application/json\r\n' +
      'User-Agent: axios/0.26.1\r\n' +
      'Content-Length: 333\r\n' +
      'Host: api.telegram.org\r\n' +
      'Connection: close\r\n' +
      '\r\n',
    _keepAliveTimeout: 0,
    _onPendingData: [Function: nop],
    agent: Agent {
      _events: [Object: null prototype],
      _eventsCount: 2,
      _maxListeners: undefined,
      defaultPort: 443,
      protocol: 'https:',
      options: [Object: null prototype],
      requests: [Object: null prototype] {},
      sockets: [Object: null prototype],
      freeSockets: [Object: null prototype] {},
      keepAliveMsecs: 1000,
      keepAlive: false,
      maxSockets: Infinity,
      maxFreeSockets: 256,
      scheduling: 'lifo',
      maxTotalSockets: Infinity,
      totalSocketCount: 4,
      maxCachedSessions: 100,
      _sessionCache: [Object],
      [Symbol(kCapture)]: false
    },
    socketPath: undefined,
    method: 'POST',
    maxHeaderSize: undefined,
    insecureHTTPParser: undefined,
    path: '/bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 429,
      statusMessage: 'Too Many Requests',
      client: [TLSSocket],
      _consuming: false,
      _dumped: false,
      req: [Circular *1],
      responseUrl: 'https://api.telegram.org/bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage',
      redirects: [],
      [Symbol(kCapture)]: false,
      [Symbol(kHeaders)]: [Object],
      [Symbol(kHeadersCount)]: 18,
      [Symbol(kTrailers)]: null,
      [Symbol(kTrailersCount)]: 0,
      [Symbol(RequestTimeout)]: undefined
    },
    aborted: false,
    timeoutCb: null,
    upgradeOrConnect: false,
    parser: null,
    maxHeadersCount: null,
    reusedSocket: false,
    host: 'api.telegram.org',
    protocol: 'https:',
    _redirectable: Writable {
      _writableState: [WritableState],
      _events: [Object: null prototype],
      _eventsCount: 3,
      _maxListeners: undefined,
      _options: [Object],
      _ended: true,
      _ending: true,
      _redirectCount: 0,
      _redirects: [],
      _requestBodyLength: 333,
      _requestBodyBuffers: [],
      _onNativeResponse: [Function (anonymous)],
      _currentRequest: [Circular *1],
      _currentUrl: 'https://api.telegram.org/bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage',
      [Symbol(kCapture)]: false
    },
    [Symbol(kCapture)]: false,
    [Symbol(kNeedDrain)]: false,
    [Symbol(corked)]: 0,
    [Symbol(kOutHeaders)]: [Object: null prototype] {
      accept: [Array],
      'content-type': [Array],
      'user-agent': [Array],
      'content-length': [Array],
      host: [Array]
    }
  },
  response: {
    status: 429,
    statusText: 'Too Many Requests',
    headers: {
      server: 'nginx/1.18.0',
      date: 'Tue, 05 Jul 2022 19:16:29 GMT',
      'content-type': 'application/json',
      'content-length': '109',
      connection: 'close',
      'retry-after': '5',
      'strict-transport-security': 'max-age=31536000; includeSubDomains; preload',
      'access-control-allow-origin': '*',
      'access-control-expose-headers': 'Content-Length,Content-Type,Date,Server,Connection'
    },
    config: {
      transitional: [Object],
      adapter: [Function: httpAdapter],
      transformRequest: [Array],
      transformResponse: [Array],
      timeout: 0,
      xsrfCookieName: 'XSRF-TOKEN',
      xsrfHeaderName: 'X-XSRF-TOKEN',
      maxContentLength: -1,
      maxBodyLength: -1,
      validateStatus: [Function: validateStatus],
      headers: [Object],
      method: 'post',
      url: 'https://api.telegram.org/bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage',
      data: '{"chat_id":"-677533133","text":"<i>Mietwohnung um Bad Oldesloe</i> (immonet) found <b>3</b> new listings:\\n\\n<a href=\\"https://www.immonet.de/angebot/47748318\\"><b>HH-Harburg Zentrum: 3 Zimmer Whg. mit Balkon</b></a>\\nHamburg Harburg | Miete zzgl. NK 1.200 € | ca. 89.0 m²\\n\\n","parse_mode":"HTML","disable_web_page_preview":true}',
      'axios-retry': [Object]
    },
    request: <ref *1> ClientRequest {
      _events: [Object: null prototype],
      _eventsCount: 7,
      _maxListeners: undefined,
      outputData: [],
      outputSize: 0,
      writable: true,
      destroyed: false,
      _last: true,
      chunkedEncoding: false,
      shouldKeepAlive: false,
      maxRequestsOnConnectionReached: false,
      _defaultKeepAlive: true,
      useChunkedEncodingByDefault: true,
      sendDate: false,
      _removedConnection: false,
      _removedContLen: false,
      _removedTE: false,
      _contentLength: null,
      _hasBody: true,
      _trailer: '',
      finished: true,
      _headerSent: true,
      _closed: false,
      socket: [TLSSocket],
      _header: 'POST /bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage HTTP/1.1\r\n' +
        'Accept: application/json, text/plain, */*\r\n' +
        'Content-Type: application/json\r\n' +
        'User-Agent: axios/0.26.1\r\n' +
        'Content-Length: 333\r\n' +
        'Host: api.telegram.org\r\n' +
        'Connection: close\r\n' +
        '\r\n',
      _keepAliveTimeout: 0,
      _onPendingData: [Function: nop],
      agent: [Agent],
      socketPath: undefined,
      method: 'POST',
      maxHeaderSize: undefined,
      insecureHTTPParser: undefined,
      path: '/bot5116548208:AAHO8HuYUnlOFe8GTBHQdn9qNEWk5nRW1hU/sendMessage',
      _ended: true,
      res: [IncomingMessage],
      aborted: false,
      timeoutCb: null,
      upgradeOrConnect: false,
      parser: null,
      maxHeadersCount: null,
      reusedSocket: false,
      host: 'api.telegram.org',
      protocol: 'https:',
      _redirectable: [Writable],
      [Symbol(kCapture)]: false,
      [Symbol(kNeedDrain)]: false,
      [Symbol(corked)]: 0,
      [Symbol(kOutHeaders)]: [Object: null prototype]
    },
    data: {
      ok: false,
      error_code: 429,
      description: 'Too Many Requests: retry after 5',
      parameters: [Object]
    }
  },
  isAxiosError: true,
  toJSON: [Function: toJSON]
}

[orangecoding]

Hi @sven-simonsen ,

the issue I see with this is how the architecture of Fredy is built. Currently, an interval is triggering Fredy every x minutes (configurable). Then, all providers are being queried in a loop. The result of each provider is routed to the configured notification adapter. Each adapter is returning a promise which resolved when all notifications have been sent. Only then, the next provider is being queried.

Now when I delay sending messages to a specific provider like telegram, I run into multiple issues, as the whole pipeline is delayed. In the worst case, if somebody configures a crawl interval of let's say 2 minutes, the delays will add up until we eat up all of the system resources.

Because of this, I currently have no real idea, how to fix this. Any clues?

[sven-simonsen]

Sounds like there are actually 3 steps to what fredy does and they are currently happening in the same task loop. Step 1 is collecting all the relevant Anzeigen with name and link. Step 2 is filtering for duplicates. Step 3 is sending out notifications on all adapter channels.

I think to make some of the improvements like filtering over all providers together and timing message sends to comply with rate limits, a structural change would be necessary where steps 1 and 3 no longer run in the same loop.

For example saving a bit more information about every Anzeige would allow filtering to happen for every batch of new Anzeigen found before adding to the db. And then step 3 could run on its own loop depending on adapter rate limits and just work through the db entries and mark the ones where notifications have been sent.

[orangecoding]

Yeah, that was something I was playing around with in my mind for a little longer already.

As you might imagine, this is quite a big change in the architecture of Fredy. It requires not only structural changes, but also changes in the database which in turn means migrating existing data.

In the long run, I am planning to move away from lowdb to SQLite. Also I am planning to run the steps mentioned by you in sequences. Each of which returning a promise, resolving when done..

However, I am not sure when I will have the time to do those changes, as I just became a father ;)

[orangecoding]

@sven-simonsen rather than trying to fix telegram, I am thinking about ditching the integration in favor of pushover.net.

If has a great free Tier and it has almost no rate limit.

What is your take on it?

[sven-simonsen]

Seems like users would need to install an app exclusively for use with fredy. I would personally not use that adapter. I would rather go back to email then and change my workflow with my partner to send emails back and forth discussing the flats. To me the advantage of Telegram is that I can send the flat announcements to a channel and then have a decent space and tools for collaboration.

[orangecoding]

Ok got it

[orangecoding]

Hey. Just a little status update;

Splitting the jobs is way harder than is seems. I think instead of dramatically changing the architecture of Fredy, I will try to change the telegram adapter...

[orangecoding]

@sven-simonsen do you have a link where i can read more about the telegram rate limit? All I find is the rate limit to not send more than 1 message per second

[sven-simonsen]

https://core.telegram.org/bots/faq#my-bot-is-hitting-limits-how-do-i-avoid-this This is probably where I got my info. Note the part that says: "Also note that your bot will not be able to send more than 20 messages per minute to the same group." Some months ago I claimed 10 was the limit and looking back I have been running into the limit less, so maybe they changed it? I also changed my use of fredy to have less parallel searches, that might also have helped.

[orangecoding]

I'm going to close this for now, as I cannot reproduce it. Maybe they changed their api, however I was able to send 50+ messages without any issue