search

microsoft / botframework-sdk

Bot Framework provides the most comprehensive experience for building conversation applications.
MIT License
7.5k stars 2.44k forks source link

HTTP status code GatewayTimeout (Is error in Azure Channel Registration?) #4833

Closed wladneto closed 6 years ago

wladneto commented 6 years ago

Bot Info

Issue Description

I have a chatbot in production (more than a month), and yesterday at night I've had some issues regarding exceptions being thrown within the chatbot, resulting in the "Sorry my bot code is having an issue" message. It´s happen only in production... In localhost works normally. My application is in South Central US region...

Is it an problem with the registration channel in azure?

Code Example

Reproduction Steps

  1. Send a message in production (in any channel)
  2. Bot response - POST to BotXPTO timeout after 15s or Sorry, my bot code is having an issue

Expected Behavior

Solve that situation... :(

Actual Results

Channel Registration (Azure) image image image

ItsMeArthur commented 6 years ago

Hi @wladneto. I have already reported this on issue #4559. It's a really weird issue and really critical IMO. I have more than 10 bots facing this problem right now, and it seems like from yesterday to today the problem has really worsened.

PS: Quão estranho é que nós dois estamos reportando de Uberlândia?

wladneto commented 6 years ago

@ItsMeArthur pois é... rssss será que existe algum erro na conectividade do Brasil?

14geronimo commented 6 years ago

I'm having the same issue. Was working fine before lunch ~1 h ago, I wanted to do a demo back from lunch, now I keep getting these Gateway errors. I get them in both Web App and Skype channels.

I restarted the App Service, since then the Web Chat channel is fine (both within Azure Portal and from a webpage embedded iframe. But the Skype channel is still kaput, I keep getting "Sorry, my bot code is having an issue."

Using West Europe region for the app, eu.luis.ai and spell check.

ItsMeArthur commented 6 years ago

@wladneto No issue que abri tem um cara até do Japão reportando o problema... Só espero que resolvam logo. Tá complicado pra quem tem bot em produção. O cliente não quer nem saber o que raios é Bot Connector.

josephktcheung commented 6 years ago

I've contacted MS support and hopefully they can tell us why this happens after investigation. Will update this post once I receive any reply from MS.

wladneto commented 6 years ago

@josephktcheung it´s a good news... so the problem is real for MS. Thanks for sharing and we waiting the answer

wladneto commented 6 years ago

Does anyone know anything that can replace this part of channel registration? Apparently the problem is in connecting the channel registration with the web application.

Carlosfbotelho commented 6 years ago

Já tentei fazer o update de todos os pacotes no Nuget mas não ajudou nada. Sin, acho que o problema é geral. Nos aqui na Europa também temos a mesma situação. Também já tentei a solução do informático (reset ao serviço do app) mas no progress. Acho que foi o Microsoft que meteu água. Não me serve de muito, mas ... espero que os gajos em redmond não demorem muito a arrumar isto.

wladneto commented 6 years ago

new return from bot... :(
POST to the bot's endpoint failed with HTTP status 401

Carlosfbotelho commented 6 years ago

That's wrong creds ... no?

josephktcheung commented 6 years ago

our bots seems to be working again now. Will wait for MS's reply nonetheless.

wladneto commented 6 years ago

no changes :( @Carlosfbotelho

vincec-msft commented 6 years ago

The Bot Framework team is aware of issues with the State Service which would affect bots on all channels. Calls to GetConversationDataWithHttpMessagesAsync and similar functions may take a long time or timeout entirely. Since most channels only wait 15 seconds for the bot to reply this could result in errors in both the client and the bot.

We created a hot fix and are expediting it through testing right now. It should be pushed to production world-wide in a few hours. Sorry for the impact and the delay!

Please note that the default State Store is not meant for Production use and has no guarantee of performance or availability. Bot developers should look at moving their bots to a dedicated State Store to insulate their bots from State Store problems.

https://docs.microsoft.com/en-us/azure/bot-service/dotnet/bot-builder-dotnet-state?view=azure-bot-service-3.0

josephktcheung commented 6 years ago

@vincec-msft thanks for the reply! We've been using dynamodb for our bot's storage, so I'm not sure previous errors happened in my bots were caused by the issues with the State Service. Anyway let's hope that this error won't happen again after the fix.

vincec-msft commented 6 years ago

Hmm, if you're already using a dedicated State Store then there might be another issue lurking. If you post a time (UTC), channel, and part of a conversation id (just the first or last couple characters should be fine) I'll look deeper.

Carlosfbotelho commented 6 years ago

@vincef-msft, thanks for looking into the issue. It worked fine for couple hours but now seems to have returned. The same gateway timeout error. Did you guys change anything ? Best, Carlos

josephktcheung commented 6 years ago

@vincec-msft here's the last error info:

conversation id: 1695****-****4886 utc time: 2018-07-06T04:48:47.266Z channel: facebook

Carlosfbotelho commented 6 years ago

I get unauthorised errors interleaved with gateaway timeout

josephktcheung commented 6 years ago

@vincec-msft This is what I found from azure app insight: Status code: GatewayTimeout Exception type: Microsoft.Bot.Schema.BotTimeoutException Failed method: Microsoft.Bot.ChannelConnector.BotAPI+d30.MoveNext Call stack: Microsoft.Bot.Schema.BotTimeoutException: at Microsoft.Bot.ChannelConnector.BotAPI+d30.MoveNext (Microsoft.Bot.ChannelConnector, Version=3.2.2.2, Culture=neutral, PublicKeyToken=null)

josephktcheung commented 6 years ago

Besides, I’m using node.js V3 of BotBuilder FYI

wladneto commented 6 years ago

My bot came back to life alone ... :) Problem solved... @ItsMeArthur voltou aí também? THANKS

Carlosfbotelho commented 6 years ago

Aqui voltou .. espero que se aguente

vincec-msft commented 6 years ago

We rolled out a fix at 2018-07-06T17:55:00Z. The service is stable again. Please let me know if you see any more persistent timeouts.

neetgupta7 commented 6 years ago

Bots up and running again. Thanks a lot.

wladneto commented 6 years ago

Thanks @vincec-msft :)

ItsMeArthur commented 6 years ago

@vincec-msft My production bots are using custom state storage with Azure Tables. I have some POC's using the Bot State Manager, and they were also affected by this issue. As I have stated on issue #4559 this is something that keeps happening from time to time, but in a smaller scale. This is the second time I've had problems that have affected all (or most of) of my chatbots. The first time was reported on issue #4645.

I really would like to see this looked into, since when I have 10 or 15 production chatbots down, and I can't do anything about it, this it's really shaking up my confidence in the service, and I hope you guys can understand that. My team has more than 30 chatbots in the pipeline to be developed this year, and this error occurring randomly it's really making us became hesitant on taking up these projects.

@wladneto Voltou sim. Agora é torcer para que continuem verificando. Esse problema tem ocorrido de maneira aleatória a um bom tempo, na realidade desde que comecei a trabalhar com a Bot Framework a quase dois anos atrás. Você já notou isso acontecendo antes?

wladneto commented 6 years ago

@ItsMeArthur sim já havia percebido, mas eram poucas ocorrências (pelo menos neste bot). Só percebi mesmo com maior frequência, depois que passaram de 100 usuários e umas 50 requisições simultâneas. Assim como você disse ai em cima estou bem preocupado em usar o BotFramework em outros projetos depois deste ocorrido. Depois quero uma aula de como usar o State Storage da forma que recomendaram, pois ainda não fiz no bot. Quando tiverem a aula me avise, vc é da cedro neh? ;)

vincec-msft commented 6 years ago

@ItsMeArthur , I fully understand your concerns and want you to be confident in the Bot Framework. Believe me, no one is more frustrated with instability than I am. We've been rolling out incremental fixes over the last couple weeks with the latest going out yesterday. Unfortunately, an unrelated issue came up after that deployment and we hotfixed that this morning. The stability looks good now but I'm watching it closely. There is no excuse for the problems you're seeing and it is the team's and my own top priority to solve them.

Keep opening issues when you see this sort of problem and mention me so I'm sure to see it. Thanks!

wladneto commented 6 years ago

@vincec-msft today i received some messages in channel registration like "There was an error sending this message to your bot: HTTP status code GatewayTimeout" :(

vincec-msft commented 6 years ago

@wladneto , sorry to hear that. There's nothing I can do without a time (UTC), channel, and part of a conversation id (just the first or last couple characters should be fine). Please share those and I'll see if this is a regression or something new.

pojdrovic commented 5 years ago

I am getting errors like this all the time over the past few days on the twilio and telegram channel. Some messages go through fine and get a response back in the roughly 0.5 seconds it usually takes, but every few messages, i'll get a gatewaytimeout error like the guys above me, and it will persist for a few minutes. Then, all the responses will arrive to the user at the same time. Almost as if something is stuck, the message responses get queued up, and once it's unstuck they are fired off at the same time. I know it can't be a code problem on my end because it's working perfectly in emulator, online chat, and azure portal test chat.

pojdrovic commented 5 years ago
Screen Shot 2019-11-12 at 4 36 44 PM

This screenshot shows the error results I get back from the app log stream

pojdrovic commented 5 years ago

And the delay always happens when context.sendActivity(message) or it's dialog counterparts are called. @vincec-msft

pojdrovic commented 5 years ago
Screen Shot 2019-11-12 at 6 34 04 PM Screen Shot 2019-11-12 at 6 34 17 PM

Sometimes I see the following behavior as well. I know that the API calls I'm making aren't the cause of the timeouts because in the instances where it times out, i can verify that the API call took less than a second and that the timeout occurs when botframework is returning a response

Kadae commented 3 years ago

Year 2021. Still problem exists.

didz-br commented 3 years ago

@vincec-msft Hi Vince

I´m experiencing this same issues from time to time as shown in the picture below. The bot works fine the most of the time, except when this "Gateway Timeout" happens. I´ll be glad if you can take a look ! Thanks

Captura de tela 2021-11-05 191504

vincec-msft commented 3 years ago

I´m experiencing this same issues from time to time as shown in the picture below. The bot works fine the most of the time, except when this "Gateway Timeout" happens.

@didz-br: This issue is 2 years old and Closed. And I no longer work on the Bot Framework. Please open a new issue and someone on the team will take a look. Thanks.