[Question] GatewayTimeout with interaction attempt after bot is idle for a few days/weeks?

[sherifsarhan]

Bot Info

• App ID: d6399230-27f6-46d7-a77f-636607844ba9
• SDK Platform: Node.js
• SDK Version: 3.13.0
• Active Channels: Facebook Messenger, Web Chat
• Deployment Environment: Azure App Service

Issue Description

After not interacting with the bot for days/weeks, GatewayTimeout Error appears. This issue is fixed and the bot is able to respond when re-deploying to Azure or even simply opening up its Log Stream on Azure. I've read elsewhere someone experienced similar issues with logging. For now, I have disabled all logs on Azure for my bot in the hopes that this issue goes away.

Reproduction Steps

1. Don't interact with your bot for a few days/weeks
2. Try interacting with it again
3. GatewayTimeout error gets thrown and unable to interact with bot

[sherifsarhan]

Possible Labels: Azure App Service, bot deployment, Node.js SDK, waiting for reply

[FranciscoPonceGomez]

Hi @sherifsarhan,

You could set up a timer to refresh the token periodically. The expiration of the token can cause latency and timeout issues. It could be beneficial to check if this could be the cause of your issue:

let chatConnector = ...;

... 
setInterval(() => {
    chatConnector.getAccessToken((error) => {
            console.log(JSON.stringify(error));
        }, (token) => {
            console.log(`token refreshed: ${token}`); 
        });
}, 30 * 60 * 1000 /* 30 minutes in milliseconds*/ );

https://blog.botframework.com/2017/11/02/optimizing-latency-bot-framework/

[sherifsarhan]

Thanks @FranciscoPonceGomez. I'll try this out and report back with results.

[sherifsarhan]

@FranciscoPonceGomez Just happened again. Facebook messenger. 1 Day of inactivity, token being refreshed periodically.

User: "hi" Chatbot: no response botframework throws: There was an error sending this message to your bot: HTTP status code GatewayTimeout User: "hello" Chatbot replies correctly, no GatewayTimeout thrown.

It's as if the deployed bot needs to be "woken up" before it can be used. "Woken up" through a re-deploy or even viewing log stream as I mentioned previously. Or even through someone messaging it first and not getting a response until they message again.

[sherifsarhan]

@FranciscoPonceGomez I'm also using the bot state API still. I'll make a switch to saving state on my own and see if the issue is fixed.

[FranciscoPonceGomez]

Hi @sherifsarhan,

That is a very good idea. Let me know if you still see the problem after saving the bot state by yourself.

Regards, Francisco

[sushovannits]

var restify = require('restify');
var builder = require('botbuilder');
var botbuilder_azure = require("botbuilder-azure");
var path = require('path');
var documentDbOptions = {
    host: '<db uri>', 
    masterKey: '<key>', 
    database: 'botdocs',   
    collection: 'botdata'
};
var docDbClient = new botbuilder_azure.DocumentDbClient(documentDbOptions);
var cosmosStorage = new botbuilder_azure.AzureBotStorage({ gzipData: false }, docDbClient);
// Create chat connector for communicating with the Bot Framework Service
var connector =  new botbuilder_azure.BotServiceConnector({
    appId: "<appid>",
    appPassword: "<app password>"
});

var bot = new builder.UniversalBot(connector, [
    function(session, results, next) {
        session.send(`Message 1`, session.message.text);
        next();
    },
    function(session, results, next) {
        session.endDialog(`Bye bye`);
    }
]).set('storage', cosmosStorage);

bot.localePath(path.join(__dirname, './locale'));

module.exports = {
    default: connector.listen()
}

This is my current code using cosmos db for storage. But still I am getting issues. I am unable to test from dev bot portal webchat .

The azure function log is:

I get a function started streaming log but nothing further. The same code was working just two days back.

[JasonSowers]

@sushovannits this is completely unrelated to the original issue, please open a new issue

[sherifsarhan]

@JasonSowers @sushovannits @FranciscoPonceGomez I have narrowed down the issue. I'd like to preface by saying that I have correctly implemented CosmosDB and the issue still occurs.

Yesterday, I interacted with the chatbot on botframework.com and FB Messenger and got responses.

After a day of inactivity, I go on to the web chat on dev.botframework.com and FB Messenger to see if I can still reproduce the issue. I send "Hi" to each of the chat sessions.

Immediately, I receive: "There was an error sending this message to your bot: HTTP status code GatewayTimeout" in both channel logs.

I wait approximately 10 seconds and then send the same message again in each chat session.

It works and I receive replies.

I am thinking this is something related to the Azure deployment of the bot. I've followed all instructions for the deploy, and I'm not quite sure how to debug this.

[sherifsarhan]

@JasonSowers @sushovannits @FranciscoPonceGomez I think I may have found the solution. Azure App Service has the "Always On" Feature which is not available to the Free Tier. I believe, given the description of my issue, this would fix it. So I think we can mark this as resolved.

[sushovannits]

@JasonSowers The issue seemed related because it was the same gateway timeout error that I was receiving. @sherifsarhan Exactly my observation is the same. Initially I thought that the botbuilder package must have got some new updates which was causing the slow down. So, I even downgraded to 3.7. This actually improved the performance, I could get the response after a minute of getting the first timeout. Yes with always on it started to work fine, but however always is on app service plan so it will cost more I believe. I am guessing there is something happening in backed of Azure + botbuilder which is causing the high loading time of botbuilder module. It may be that the IO is slow and hence I am thinking of giving webpack a try.

[JasonSowers]

@sherifsarhan thanks for the update. If the always-on feature does not work, the other option you have is to send your bot a ping activity (ActivityType.Ping) periodically I would suggest 30 minutes or less. This can be done with a web job or something similar.

Unfortunately, the downside for AWS Lambda and Azure Functions because they have to "spin up" after they have been put to sleep it causes increased latency with the first message. This is known and expected in these environments.