Bot silently stops pulling/responding

[trashbytes]

Hi there!

I am running this bot on my Raspberry Pi and it always just stops pulling the messages after a period of time.

I am running it with nohup to daemonize it like this:

nohup node app.js &

The nohup.out never throws an error, it just stops pulling the messages until I kill and restart the bot. Then it processes all missed messages and goes on for a few hours. My Raspberry Pi is running rock solid for more than 120 days straight now.

Any idea?

[shernshiou]

@trashbytes How long did you run the application? Do you have any snippets so I can debug?

[trashbytes]

Of course, it is not much since most of the heavy stuff is handled in another script called "data" (it is a db made up of several arrays which are usually searched through, processed and the result returned).

var Bot = require('node-telegram-bot');
require('node-import');

global = imports('data/data');

var commands = ['search'];

var bot = new Bot({
    token: 'TOKEN'
});
bot.on('message', function (message) {
    if (message.text) {
        var msg = message.text.toLowerCase();
        if (msg.indexOf('/') == 0) {
            var params = msg.substr(1).split(' ');
            var command = params.shift();
            var param = params.join(' ');
            if (!isNaN(parseFloat(command)) && isFinite(command)) {
                bot.sendMessage({
                    parse_mode: 'Markdown',
                    chat_id: message.chat.id,
                    text: getUnit(msg.substr(1))
                });
            } else if (command == 'search') {
                bot.sendMessage({
                    parse_mode: 'Markdown',
                    chat_id: message.chat.id,
                    text: search(param)
                });
            } else {
                bot.sendMessage({
                    parse_mode: 'Markdown',
                    chat_id: message.chat.id,
                    text: '*Error!*\n\nPlease enter a valid command!'
                });
            }
        }
    }
})
bot.start();

As you can see it's just a modification of your echo bot example. I added a few things to parse the message.text and split it into the command and the parameters. It works great but how long it runs is kind of random. Usually a few hours.

I am by no means an expert at JavaScript so please be lenient :)

[longstone]

just my two cents to this topic: i'm pretty sure i've the same problem on a dedicated instance. from time to time, the bot stop interacting. But the logs are quiet. how can we investigate on this?

[shernshiou]

@longstone @trashbytes It might be caused by the interval polling. Can it be memory leakage?

BTW, I use this for a production environment since August. I never encountered this problem.

[trashbytes]

Well you are pretty limited on a Raspberry Pi ram-wise. I will further test this with memory leakage in mind. @longstone when this happens, are you using it on a weaker machine, too?

[derhuerst]

I'm not using it on a Raspi, but it doesn't work reliably either on a laptop. Also, when reaching the memory limit, node will tell you when crashing.

[longstone]

@trashbytes I'm using it on openshift. I'm from time to time, i realise the bot isn't working anymore. so i have to restart the whole nodejs application. in logs, nothing happen (I catch thread exceptions as well see here). Are you all using polling instead of webhooks?

[trashbytes]

@longstone Yes, I am using polling as well. I also have to restart node to get it working again.

[derhuerst]

@longstone polling as well.

[trashbytes]

I just noticed that node doesn't crash. The bot just stops polling after a while. Same thing happened a while ago when I was trying out this API with pm2, which keeps node alive forever. Back then the bot also stopped polling after a while although it was still running, if I remember it correctly.

[derhuerst]

Can confirm that.

[shernshiou]

Thanks for the report. @trashbytes @derhuerst @longstone, I will start debugging by catching memory leakage.

[JasperAlgra]

I might have the same issue but can't confirm memory. I do have a crashlog (running with supervisorctl):

Error: Unknown error. Status code: 502 at Request._callback (/home/jasper/telegram-npm/node_modules/node-telegram-bot/lib/Bot.js:240:28) at Request.self.callback (/home/jasper/telegram-npm/node_modules/request/request.js:198:22) at Request.EventEmitter.emit (events.js:98:17) at Request.<anonymous> (/home/jasper/telegram-npm/node_modules/request/request.js:1082:10) at Request.EventEmitter.emit (events.js:117:20) at IncomingMessage.<anonymous> (/home/jasper/telegram-npm/node_modules/request/request.js:1009:12) at IncomingMessage.EventEmitter.emit (events.js:117:20) at _stream_readable.js:920:16 at process._tickCallback (node.js:415:13)

Update: just done some more reading, apparently the 502 is a gateway error (of telegram servers?). Maybe a harmless error (well, you can retry to fetch messages I guess) but the bot does die after this.

[shernshiou]

@JasperAlgra thanks.. I am currently running the bot for a few days to catch any memory leak.

[trashbytes]

@JasperAlgra I also thought about a network issue. To me it seems like the bot doesn't retry to get messages if it fails once.

[trashbytes]

I just got this error.

Error: Unknown error. Status code: 504
    at Request._callback (/home/tobias/optc_bot/node_modules/node-telegram-bot/$
    at Request.self.callback (/home/tobias/optc_bot/node_modules/node-telegram-$
    at Request.emit (events.js:110:17)
    at Request.<anonymous> (/home/tobias/optc_bot/node_modules/node-telegram-bo$
    at Request.emit (events.js:129:20)
    at IncomingMessage.<anonymous> (/home/tobias/optc_bot/node_modules/node-tel$
    at IncomingMessage.emit (events.js:129:20)
    at _stream_readable.js:908:16
    at process._tickCallback (node.js:355:11)

[shernshiou]

@trashbytes can you update your code to latest master and listen to 'error' ?

[trashbytes]

Will do! Thanks.

[trashbytes]

@shernshiou the bot is running rock solid since yesterday noon. No errors, no stops, no crashes. It never lasted that long so you may have done it! Thanks a ton!

[shernshiou]

@trashbytes glad to hear that. I will keep monitoring for anomaly.

[longstone]

@shernshiou will there be an release? I think a lot of users will benefit from the latest change...

[shernshiou]

@longstone I will release now.

[trashbytes]

I recently updated to the new version and this issue reappeared.

[shernshiou]

@trashbytes is it related to the new issue you opened?

[trashbytes]

No, they are not related.

[trashbytes]

As I rolled back to a previous build I noticed that a Gateway error is never thrown or catchable in the latest version. I get quite a few over the days when running the working build so I assume that something in that area changed which causes the bot to again stop pulling on such an error.

[seeya]

I am also facing this issue, I get [Error: Gateway error.] this error and the bot just stops and die. I'm using polling. It's about 2-3 days before I get this error and have to manually restart the bot.

[shernshiou]

@seeya I faced this error when my network is down.

[Kaos1337]

Do you have any suggestion to manage the "gateway error" due a temporary network down? Like a wait for ...

[trashbytes]

What works is catching and logging/ignoring. It will try again.

[AlecSchneider]

I have the same problem, running my bot on forever and after like 20h it just stopps, error log empty, looks like it just stops pulling... may using webhooks fix this?

[trashbytes]

Use an older build or switch to Yago Perez API.

[AlecSchneider]

not really the solution I was hoping for, on which build is it working?

[trashbytes]

https://github.com/depoio/node-telegram-bot/tree/9777fec0e254af667b4aa20729a31086b78834c5

I think this one.

[shernshiou]

sorry guys.. i do not have my computer for a few months. I will rewrite this asap after i got my com.