search

MicrosoftDocs / msteams-docs

Source for the Microsoft Teams developer platform documentation.
https://aka.ms/teamsdev
Creative Commons Attribution 4.0 International
288 stars 511 forks source link

MS Teams - Bot framework bot is slow #11583

Closed gykung closed 2 weeks ago

gykung commented 2 months ago

Steps to reproduce

I have a bot, written in Python, bot framework, run on a local VM, connected via ngrok to my bot app for my enterprise. User messages take about 5-7 seconds before it reaches the bot/VM. Does anyone else face a similar issue? What’s the typical latency expected?

One interesting observation: during the crowdstrike incident, the bot responded significantly faster. My enterprise was not affected by crowdstrike.

Sorry for asking this question here, I am not sure where else to ask this.

Expected behavior

Nearly zero lag.

Actual behavior

5-7 seconds lag.

Error details

No response

Meghana-MSFT commented 2 months ago

Hello @gykung - Thanks for raising your query:

The latency experienced by you where user messages take about 5-7 seconds to reach the bot/VM when running on a local VM connected via NgRok is not typically expected. This delay could be due to various factors such as network configuration, VM performance, or ngrok settings.

It's recommended to check and optimize these aspects to reduce latency: Optimize VM performance: Ensure the VM has enough resources (CPU, memory) to handle requests efficiently. Create and host a dev tunnel: (Alternative for NgRok and recommended by Microsoft) https://learn.microsoft.com/en-us/azure/developer/dev-tunnels/get-started?tabs=windows Upgrade ngrok plan: Consider using a paid ngrok plan for better routing and reduced latency. Test direct hosting: Consider hosting the bot on a cloud provider (e.g., Azure, AWS) instead of a local VM to reduce the latency caused by local network and NgRok/Dev Tunnel.

Meghana-MSFT commented 1 month ago

@gykung - Could you please share the bot id with us at microsoftteamsdev@microsoft.com. Also, please note that

A host (and a bot is just a webapp host) not receiving a message or it's slow getting there is 999 times out of 1000 a network issue.

Python, like JS, is a single threaded core. So, it's not hard to imagine in a very high load situation where the bot is just backed up. But this is rare since, A) Very high load, B) Mostly applies to compute bound situations. i.e., The bot is running some very complex algorithm during each message.

gykung commented 1 month ago

Hi Meghana,

Thank you very much for your reply. The MicrosoftAppId is "515d62a9-34b7-4f90-b213-8b4c7721951f". In case there are analytics on your end that could help us solve the issue, feel free to share with me.

I agree it could ultimately be a network issue as it is taking several seconds for the incoming request to reach the Python bot app. I've tried ping but there is no lag. If there are specific Azure admin settings, Azure-related network issues we should trace, feel free to suggest.

Thank you.

Regards, Guangyi @gykung

On Thu, Sep 19, 2024 at 1:26 PM Meghana-MSFT @.***> wrote:

@gykung https://github.com/gykung - Could you please share the bot id with us at @.*** Also, please note that

A host (and a bot is just a webapp host) not receiving a message or it's slow getting there is 999 times out of 1000 a network issue.

Python, like JS, is a single threaded core. So, it's not hard to imagine in a very high load situation where the bot is just backed up. But this is rare since, A) Very high load, B) Mostly applies to compute bound situations. i.e., The bot is running some very complex algorithm during each message.

— Reply to this email directly, view it on GitHub https://github.com/MicrosoftDocs/msteams-docs/issues/11583#issuecomment-2360006383, or unsubscribe https://github.com/notifications/unsubscribe-auth/AQWH4JVEWJPUUXBQ4FGJHMDZXJN7PAVCNFSM6AAAAABOE6567GVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGNRQGAYDMMZYGM . You are receiving this because you were mentioned.Message ID: @.***>

Meghana-MSFT commented 1 month ago

@gykung - We have shared the MicrosoftAppId with engineering team, we will keep you posted on the investigation details. Thank you.

gykung commented 1 month ago

Hi Meghana,Would you or your team have any insights/findings? Thank you.Guangyi @gykung On 20 Sep 2024, at 5:08 PM, Meghana-MSFT @.***> wrote: @gykung - We have shared the MicrosoftAppId with engineering team, we will keep you posted on the investigation details. Thank you.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: @.***>

gykung commented 1 month ago

I presume that the traffic flows this way: User-->teams.microsoft.com-->botframework.com-->ngrok tunnel --> VM hosting the bot, and then in reverse direction to return the response to the user.

I've tried tracert with directly to my ngrok address, with teams.microsoft.com, and botframework.com.

Notably, botframework.com has the most significant lag.

image

gykung commented 2 weeks ago

@Meghana-MSFT I've fixed the issue. It was an unexpected issue in the app. Somehow the tracert and network tracing had been giving false clues. Thanks for the time spent on this.

gykung commented 2 weeks ago

@Meghana-MSFT I've fixed the issue. It was an unexpected issue in the app. Somehow the tracert and network tracing had been giving false clues. Thanks for the time spent on this.

microsoft-github-policy-service[bot] commented 2 weeks ago

Tell us about your experience!

Hi gykung! This issue is closed in our system. We would like your feedback on your experience with our support team and Platform.

Best regards, Teams Platform