microsoft / botframework-sdk

Bot Framework provides the most comprehensive experience for building conversation applications.
MIT License
7.5k stars 2.44k forks source link

[Direct Line] Performance & latency issues with BotBuilder #1892

Closed gregk closed 7 years ago

gregk commented 7 years ago

So, we really like the concept of BotBuilder to design sophisticated CI applications. We easily moved our solution to LUIS because of the power of its intents.

However, we are seeing 4 to 6 second turnaround times in BotBuilder between communication to and from the service as well as actions within the service. We aren't finding any issues with LUIS which is sub 500 ms. In seems like others have reported this problem as far back as August. See issue #1618. This makes the service unusable for any interaction beyond a single utterance.

Is there any plan to dramatically increase performance? If so, do you have a timeline?

dandriscoll commented 7 years ago

Hi @gregk, which channel are you using? Is this Direct Line like the issue you linked?

gregk commented 7 years ago

Hi @dandriscoll yes, it is direct line. All communication is between our service and MBF since we don't make use of outside services today.

dandriscoll commented 7 years ago

If you send a conversation ID, I can get a breakdown and let you know where things stand.

One item of note is that POSTing a message takes longer than it does to write to the service. The call includes transit to our service, persisting the call, transit to the bot, waiting for the bot to complete (which sometimes involves calling back into the service), returning the error code from the bot, and then transit back to our service and finally your client. Your client sees the worst-case scenario because we want to return an HTTP status code that reflects your bot's status.

We do have a performance-oriented storage rewrite in testing now. I'll know more about the impact when I can see timing on your calls.

gregk commented 7 years ago

Your write up follows along the lines of our analysis. I'm going to share this with my team. We are writing a hack to work around these issues but would much prefer to use an approved method. Do you have a timeline for moving it out of testing? We would be happy to work an early version and give feedback. Really appreciate your team focusing on this.

areckkimo commented 7 years ago

I have the same problem. My bot response slow between 2 and 4 second The channel of the bot is facebook messager. It deploys on azure's bot service.

ebattulga commented 7 years ago

I have same problem. Using LUIS and Facebook Messenger. Bot service deployed in local dedicated server. But response delay 4-8 second.

ebattulga commented 7 years ago

When user send message on Facebook Messenger It seems to like this. is it correct?

  1. Facebook send message to facebook.botframework.com
  2. botframework.com send message to my server
  3. my server send to LUIS
  4. LUIS calculate and respond to my server
  5. my server respond corresponding message to botframework.com
  6. botframework.com send to facebook

How to skip botframework.com or improve performance? My target user is only FB Messenger

gilesbutler commented 7 years ago

Also seeing very slow response times from the Bot framework, even just using the emulator in the portal we're getting 2-4 second delays.

dandriscoll commented 7 years ago

Hi @gregk, contact me offline.

dandriscoll commented 7 years ago

Hi @ebattulga, that's correct. Using the connectors is currently the only supported way to use the Bot Framework. There are unsupported libraries to connect direct to FB but I'm not familiar with them. (Example: https://github.com/andrew-makarenko/botbuilder-facebook)

dandriscoll commented 7 years ago

Hi @gilesbutler, at the moment it depends a lot on where you're loading the dev portal from. In some regions, the dev portal is making trans-oceanic calls to get to a datacenter. We have some enhancements for this coming soon.

gilesbutler commented 7 years ago

Hi @dandriscoll, I'm based in Sydney, Australia, so I presume that is why it's slow. Looking forward to the enhancements 👍

areckkimo commented 7 years ago

messenger<-->fb server<-->bot connect<-->bot server How to get seconds of every stage?

lcschiltz commented 7 years ago

Hi @dandriscoll any update ? I am facing the same issue. My bot is hosted on azure west europe server

jrios6 commented 7 years ago

Hi @dandriscoll my azure bot is hosted in South East Asia, FB Messenger takes 4-6s to respond. Anyway to resolve this issue?

nwhitmont commented 7 years ago

Bot Framework has undergone some major performance upgrades recently. If you are still experiencing problems, open a new issue. Thx

roskodav commented 7 years ago

Still very slow response

ghost commented 7 years ago

Being that the Azure Bot Service is an httpTriggered Azure Function, I think it would be nice to be able to see the function threads that the bot is currently using. We're planning to process upwards 500/s directline requests and seeing the scaling on Azure is critical to diagnostic/debugging. Is this possible already? How to do this? Application Insights seem to be pretty limited for the Bot Service, at least whats shown on the Analytic tab in the resource on Azure portal.

I know httpTrigger Azure Functions are having some problems scaling/load balancing in general and since Bot Service is httpTriggered, it's probably experiencing the same issues. Yes / No ?

Furthermore, is there any plan to expand to allow the Bot Service to act as a ServiceBus? QueueProcessor? I know ServiceBus functions can scale and handle 1000+/sec requests when BrokeredMessage method is used (pretty impressive to see a single thread scale up to 12 and push through 100/s each).

What advice/experience can anyone share about hitting the DirectLine channel this hard? Starting to run tests now and will share when I have anything conclusive to add.

dandriscoll commented 7 years ago

Hi @MikeDrips please do not perform any load testing on Direct Line.

ghost commented 7 years ago

Okay. When will it be functional for high volumes of traffic?
Mike 

dandriscoll commented 7 years ago

It is already. However, it's intended for user-generated traffic.

Our recommendation for performing load testing is to connect directly to your bot's endpoint. We're still expanding the documentation for this but the core of it is to request a token using the same configuration that the emulator uses and set up a sink that can receive callbacks from your bot.

ghost commented 7 years ago

Thanks @dandriscoll. We appreciate your help.

We don't have any user-generated traffic per say. We have a prior existing system that all traffic funnels through, passing it off to the bot through DirectLine in the meantime for AI processing. If don't need to using DirectLine we won't, the only reason we are is because we we're told to by Microsoft at a BotHackFest.

Does this mean we can send/receive Activity objects to/from the bot without sending through the DirectLine channel? Can you outline how this would/should be done? Will it have better performance under high loads? How does auth work in this case? Is the ?code= in the endpoint the token you're speaking of? How can we regen it or find it's place of origin?

We have our own custom bot state service as well. So we really just need the bot to process and Activity and reply with one. We're using the Bot framework because in the future we'll open this up to all the other channels. Right now we just need it to be able to handle high loads from one source. Our whole system is Azure Functions as well so we're hoping the bot function can scale well enough to keep up.

I know you guys are working on it, but this stuff needs doc or it's a long road to come back from if it isn't the right choice.

Cheers

On Tue, Jun 13, 2017 at 11:16 AM, dandriscoll notifications@github.com wrote:

It is already. However, it's intended for user-generated traffic.

Our recommendation for performing load testing is to connect directly to your bot's endpoint. We're still expanding the documentation for this but the core of it is to request a token using the same configuration that the emulator uses and set up a sink that can receive callbacks from your bot.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Microsoft/BotBuilder/issues/1892#issuecomment-308150452, or mute the thread https://github.com/notifications/unsubscribe-auth/Ab6EPqCmO0L-LDh1QPinQ92FxhzNya3Aks5sDqfHgaJpZM4LSXKc .