"Empty" messages comming from DirectLine

[pedro-ramirez-suarez]

Describe the bug

Recently we started to get "empty" or incomplete messages from directline randomly, this messages represent between 1% and 2% of the total messages.

Usually we get a response like this:

{
    "type": "message",
    "id": "EepDdNGyZlBdtvN116472-us|0000006",
    "timestamp": "2021-10-11T15:51:49.3327947Z",
    "channelId": "directline",
    "from": {
        "id": "DevMainBotNew",
        "name": "DevMainBotNew"    },
    "conversation": {
        "id ": "EepDdNGyZlBdtvN116472-us"    },
    "attachmentLayout": "list",
    "inputHint": "expectingInput",
    "attachments": [{
        "contentType": "application/vnd.microsoft.card.hero",
        "content": {
            "text ": "Please use your phone keypad to enter your five digit zipcode now.",
            "lgtype ": "HeroCard "        }
    }],
    "entities": [],
    "replyToId ": "EepDdNGyZlBdtvN116472-us|0000004" 
}

And this is the response that we are randomly getting:

{
  "type": "message",
  "id": "D6jHza4IrvW77I9VmDdQfo-6|0000040",
  "timestamp": "2021-10-06T21:04:16.57671Z",
  "channelId": "directline",
  "from": {
    "id": "BOTPRODMainBot",
    "name": "BOTPRODMainBot"
  },
  "conversation": {
    "id": "D6jHza4IrvW77I9VmDdQfo-6"
  },
  "inputHint": "expectingInput",
  "attachments": [],
  "entities": [],
  "replyToId": "D6jHza4IrvW77I9VmDdQfo-6|0000039"
}

That "empty" response is returned no matter if the expected response it's a normal message, a hero card or an activity card, sometimes retrying with the same watermark returns the right response but sometimes the same "empty" message is returned again and again.

Version

1.4.1

Browser

• [X] Electron distribution
• [ ] Chrome
• [ ] Safari
• [ ] Firefox
• [ ] Edge

OS

• [X] macOS
• [X] Windows
• [ ] Ubuntu

To Reproduce

Steps to reproduce the behavior: This happens randomly and at different steps of the conversation and it only happens after the bot has been deployed to Azure, never on the local machine.

Expected behavior

I would expect to not receive "empty" messages

Additional context

As mentioned earlier, this was not happening before, it started to happen a couple of weeks ago; the changes that were made to the bot were minor flow changes and we have re deployed a few times.

[tonyanziano]

Transferring this issue to BF Services because it is only happening on remote bots in Azure. Can someone on the service side inspect the traffic for this particular bot and figure out what's causing these activities to be sent?

[craigjensen]

I took a look at our server logs and don't see any errors associated with message D6jHza4IrvW77I9VmDdQfo-6|0000040. It's possible that there's something about the content that's causing it to get dropped. Can you attach a network trace that shows the message content of a request that resulted in an empty message? We don't log the content on our end so there's no way for us to tell from our server logs what was originally posted.

[pedro-ramirez-suarez]

I'll get the network trace.

[pedro-ramirez-suarez]

Sorry for the late response, we captured the network traffic with Wireshark, you can download the session from here: https://drive.google.com/file/d/1z6Li_MN3v0BPLk-MZhjGxNftyFBuCSir/view?usp=sharing The empty message was present a few times. let me know if you need any other thing from our side.

[stevkan]

Hi @pedro-ramirez-suarez, I'm assisting on the investigation of this issue. Thank you for the saved Wireshark session you provided. Unfortunately, it doesn't provide all the data that I was hoping to see, primarily the data/content being posted. However, if you are willing to capture another session with a slight tweak, I think we can get it. At least, that is what my tests using Wireshark are showing me.

For generating a new capture, you will need to run your bot locally using ngrok to generate a messaging endpoint (placed in your Azure bot's configuration). You can use Emulator as the client and connect to the same messaging endpoint.

When you are ready to start capturing, you will want to select "Adapter for loopback traffic capture" as the interface to capture thru. Either double click or right click and choose "Start capture" on the "Adapter" interface to begin.

After capturing, filtering on "http" in the session should show you results. If not, you can try connecting straight to the localhost address in Emulator (i.e., http://localhost:3978/api/messages).

[stevkan]

@pedro-ramirez-suarez, were you able to re-run the capture session using the above instructions?

[pedro-ramirez-suarez]

I have not but we think that we have identified the cause and seems related with some time outs, I'll post more details later.

[stevkan]

Closing due to inactivity. If the problem persists, please feel free to re-open.