search

microsoft / BotFramework-WebChat

A highly-customizable web-based client for Azure Bot Services.
https://www.botframework.com/
MIT License
1.56k stars 1.51k forks source link

Live Streaming | second live stream is rendered on top of the first one, #5222

Closed HesselWellema closed 3 days ago

HesselWellema commented 1 week ago

Is it an issue related to Adaptive Cards?

No

Is this an accessibility issue?

No

What version of Web Chat are you using?

Latest production

Which distribution are you using Web Chat from?

NPM

Which hosting environment does this issue primarily affect?

Web apps

Which browsers and platforms do the issue happened?

Browser: Edge (latest)

Which area does this issue affect?

Development experience

Which theme pack does this issue affect?

I did not test it on other theme packs

What is the public URL for the website?

No response

Please describe the bug

If an answer is streamed to the frontend, the first answer is rendered in chunks correctly. When a second answer is send to webchat, it is rendered on top of the first one.

Do you see any errors in console log?

No

How to reproduce the issue?

Use sample a.minimizable-web-chat Update to latest version: npm install botframework-webchat@main

  1. Send answer 1 to the webchat
{"channelData":{"streamSequence":1,"streamType":"streaming"},"type":"typing","text":"Typing an answer .."}
{"channelData":{"streamId":"c053d730-2fb9-11ef-9f75-136888712e32","streamType":"final"},"text":"Hello to you!","type":"message"}
  1. Send answer 2 to the webchat
{"channelData":{"streamSequence":1,"streamType":"streaming"},"type":"typing","text":"Typing an answer .."}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":2,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":3,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":4,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a wonderful"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":5,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a wonderful day"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":6,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a wonderful day!"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamType":"final"},"text":"Farewell to you, have a wonderful day!","type":"message"}

What do you expect?

I would expect the second answer to be rendered below the question is answers

What actually happened?

It is rendered above the first answer

Do you have any screenshots or recordings to repro the issue?

image

Adaptive Card JSON

No response

Additional context

The events send to webchat from my bot.

First answer stream

{"channelData":{"streamSequence":1,"streamType":"streaming"},"type":"typing","text":"Typing an answer .."}
{"channelData":{"streamId":"c053d730-2fb9-11ef-9f75-136888712e32","streamType":"final"},"text":"Hello to you!","type":"message"}

Second answer stream

{"channelData":{"streamSequence":1,"streamType":"streaming"},"type":"typing","text":"Typing an answer .."}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":2,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":3,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":4,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a wonderful"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":5,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a wonderful day"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamSequence":6,"streamType":"streaming"},"type":"typing","text":"Farewell to you, have a wonderful day!"}
{"channelData":{"streamId":"d2e36d70-2fb9-11ef-9f75-136888712e32","streamType":"final"},"text":"Farewell to you, have a wonderful day!","type":"message"}
OEvgeny commented 1 week ago

Adding timestamp to activities may help for now. Full conversation transcript would be helpful, so we could repro this with our tests suite.

HesselWellema commented 1 week ago

I will add timestamps and see if that helps Do you have an endpoint I can use to make sure it is not my own botcode?

or even better: share the botcode you guys used for testing. I can use that as a starting point.

OEvgeny commented 1 week ago

We mostly use our tests to continuously check things. You can find related by searching for streamId

And here is an example bot from William: https://github.com/compulim/experiment-webchat-streamingbot

HesselWellema commented 1 week ago

Tx mate. Will keep you posted.

HesselWellema commented 1 week ago

I added timestamps. Still not ok.

Bot

{"type":"typing","id":"G9zqRneUmCX4MQ1NIxlUro-eu|7JrmOwAwbnH","timestamp":"2024-06-23T10:51:39.9730338Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"G9zqRneUmCX4MQ1NIxlUro-eu"},"locale":"nl-NL","text":"Typing an answer ..","channelData":{"streamSequence":1,"streamType":"streaming","timestamp":"2024-06-23T10:51:39.835Z"},"replyToId":"G9zqRneUmCX4MQ1NIxlUro-eu|0000005"}

User question: How much is 2 * 21. Only give the answer```

Bot event 1

{"type":"typing","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|ISt7TasNiXb","timestamp":"2024-06-23T10:53:39.162762Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"Typing an answer ..","channelData":{"streamSequence":1,"streamType":"streaming","timestamp":"2024-06-23T10:53:39.026Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000005"}

Bot event 2

{"type":"message","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000006","timestamp":"2024-06-23T10:53:39.3334534Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"42","channelData":{"streamId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|ISt7TasNiXb","streamType":"final","timestamp":"2024-06-23T10:53:39.202Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000005"}

Bot evenr 3

{"type":"message","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000007","timestamp":"2024-06-23T10:53:39.4750714Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"Stel gerust een vervolgvraag of klik op het huisje om opnieuw een dossier te kiezen.","inputHint":"acceptingInput","replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000005"}

User question: What time it it? Answer in English

Bot event 1

{"type":"typing","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|EoVg5w2gDnM","timestamp":"2024-06-23T10:56:38.6987624Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"Typing an answer ..","channelData":{"streamSequence":1,"streamType":"streaming","timestamp":"2024-06-23T10:56:38.584Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000008"}

Bot event 2

{"type":"typing","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|3zMZKfnMxvj","timestamp":"2024-06-23T10:56:39.5194385Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"It is currently 12","channelData":{"streamId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|EoVg5w2gDnM","streamSequence":2,"streamType":"streaming","timestamp":"2024-06-23T10:56:39.392Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000008"}

Bot event 3

{"type":"typing","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|F6XxTvLGTnU","timestamp":"2024-06-23T10:56:39.6914206Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"It is currently 12:","channelData":{"streamId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|EoVg5w2gDnM","streamSequence":3,"streamType":"streaming","timestamp":"2024-06-23T10:56:39.527Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000008"}

Bot event 4

{"type":"typing","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|AicK1PDGL2X","timestamp":"2024-06-23T10:56:39.874014Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"It is currently 12:39","channelData":{"streamId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|EoVg5w2gDnM","streamSequence":4,"streamType":"streaming","timestamp":"2024-06-23T10:56:39.717Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000008"}

Bot event 5

{"type":"typing","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|LhOUASFp8On","timestamp":"2024-06-23T10:56:40.0787362Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"It is currently 12:39.","channelData":{"streamId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|EoVg5w2gDnM","streamSequence":5,"streamType":"streaming","timestamp":"2024-06-23T10:56:39.926Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000008"}

Bot event 6

{"type":"message","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000009","timestamp":"2024-06-23T10:56:40.2810958Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"It is currently 12:39.","channelData":{"streamId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|EoVg5w2gDnM","streamType":"final","timestamp":"2024-06-23T10:56:40.106Z"},"replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000008"}

Bot event 7

{"type":"message","id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000010","timestamp":"2024-06-23T10:56:40.5052304Z","channelId":"directline","from":{"id":"bot4aamelegalauthenticatedstreaming","name":"Bot","role":"bot"},"conversation":{"id":"5hzRJ5kIZ3rLyJWEA1K7N2-eu"},"locale":"nl-NL","text":"Stel gerust een vervolgvraag of klik op het huisje om opnieuw een dossier te kiezen.","inputHint":"acceptingInput","replyToId":"5hzRJ5kIZ3rLyJWEA1K7N2-eu|0000008"}

image

HesselWellema commented 1 week ago

I also checked the botcode you shared and implemented it with no good result. There seems to be a mismatch between the botcode and the pullrequest wth the requirement of the events mentioned in the pullrequest Differences i noticed No streamType: 'streaming' in the botcode No streamType: 'final' in the botcode The botcode is jus sending typing events Also in the botcode there is this: 'willContinue' in context.adapter && context.adapter.willContinue(context); What is the purpose and do I also need it in my implementation

I would really appreciate your guidance because I feel we are almost there.

OEvgeny commented 1 week ago

@HesselWellema could you try the following:

You should be able to start a completely new stream the same way: by sending a new typing activity with a fresh id. Looks like you're doing something similar, but maybe subsequent activity ids confuse WebChat.

I have a note to look here to debug this, but haven't got time yet due to other workstreams

HesselWellema commented 1 week ago

@OEvgeny. Did what you suggested but second answer is still on top (so before) the first answer. Reproducible on the a.minimizable-web-chat sample.

.1 I removed all other events that are send by the bot. The only events it received are typing events. Just to make sure that does not cause any issues. .2 Also tested for three and four answers. The third is rendered between the 1th and 2nd. The fourth between the 3d and 4th. .3 @compulim. If you have a sample of events that do render in the right order, I am interested. I should be able to create those in my bot code. I did notice in you little movie that it should work.

My latest results based on two questions:

Answer 1

the requests I send via await context.sendActivity(request);

{"channelData":{"streamSequence":1},"type":"typing","text":"Typing an answer .."}
{"channelData":{"streamId":"F3YgYHw44MhAEJTEDrltry-eu|DvzgnR1yzXB","streamSequence":2},"type":"typing","text":"42"}

typing events received in webchat

{
    "type": "typing",
    "id": "F3YgYHw44MhAEJTEDrltry-eu|DvzgnR1yzXB",
    "timestamp": "2024-06-25T07:18:50.5493476Z",
    "channelId": "directline",
    "from": {
        "id": "bot4aamelegalauthenticatedstreaming",
        "name": "Bot",
        "role": "bot"
    },
    "conversation": {
        "id": "F3YgYHw44MhAEJTEDrltry-eu"
    },
    "locale": "nl-nl",
    "text": "Typing an answer ..",
    "channelData": {
        "streamSequence": 1
    },
    "replyToId": "F3YgYHw44MhAEJTEDrltry-eu|0000009"
}

{
    "type": "typing",
    "id": "F3YgYHw44MhAEJTEDrltry-eu|BryxZrF6dPf",
    "timestamp": "2024-06-25T07:18:50.6805386Z",
    "channelId": "directline",
    "from": {
        "id": "bot4aamelegalauthenticatedstreaming",
        "name": "Bot",
        "role": "bot"
    },
    "conversation": {
        "id": "F3YgYHw44MhAEJTEDrltry-eu"
    },
    "locale": "nl-nl",
    "text": "42",
    "channelData": {
        "streamId": "F3YgYHw44MhAEJTEDrltry-eu|DvzgnR1yzXB",
        "streamSequence": 2
    },
    "replyToId": "F3YgYHw44MhAEJTEDrltry-eu|0000009"
}

Answer 2 (which is rendered before answer 1

the requests I send via await context.sendActivity(request);

{"channelData":{"streamSequence":1},"type":"typing","text":"Typing an answer .."}
{"channelData":{"streamId":"F3YgYHw44MhAEJTEDrltry-eu|8X7OePylskP","streamSequence":2},"type":"typing","text":"1999"}

typing event received in webchat

{
    "type": "typing",
    "id": "F3YgYHw44MhAEJTEDrltry-eu|8X7OePylskP",
    "timestamp": "2024-06-25T07:21:24.0934264Z",
    "channelId": "directline",
    "from": {
        "id": "bot4aamelegalauthenticatedstreaming",
        "name": "Bot",
        "role": "bot"
    },
    "conversation": {
        "id": "F3YgYHw44MhAEJTEDrltry-eu"
    },
    "locale": "nl-nl",
    "text": "Typing an answer ..",
    "channelData": {
        "streamSequence": 1
    },
    "replyToId": "F3YgYHw44MhAEJTEDrltry-eu|0000017"
}

{
    "type": "typing",
    "id": "F3YgYHw44MhAEJTEDrltry-eu|CAyqP0Owhat",
    "timestamp": "2024-06-25T07:21:24.24051Z",
    "channelId": "directline",
    "from": {
        "id": "bot4aamelegalauthenticatedstreaming",
        "name": "Bot",
        "role": "bot"
    },
    "conversation": {
        "id": "F3YgYHw44MhAEJTEDrltry-eu"
    },
    "locale": "nl-nl",
    "text": "1999",
    "channelData": {
        "streamId": "F3YgYHw44MhAEJTEDrltry-eu|8X7OePylskP",
        "streamSequence": 2
    },
    "replyToId": "F3YgYHw44MhAEJTEDrltry-eu|0000017"
}
LuisM000 commented 1 week ago

I'm experiencing the same issue. It seems to work for me if I send the messages in the following way:

First chunk:

{
  "channelData": {
    "streamSequence": 1,  
    "streamType": "streaming"
  },
  "type": "message"
}

Subsequent chunks:

{
  "channelData": {
    "streamId": "<activity ID of the first chunk>",
    "streamSequence": 2,  // Must be a monotonically increasing positive integer
    "streamType": "streaming"
  },
  "type": "typing"
}

Final chunk:

{
  "channelData": {
    "streamId": "<activity ID of the first chunk>",
    "streamType": "final"
  },
  "type": "typing"
}

Basically, compared to the PR https://github.com/microsoft/BotFramework-WebChat/pull/5141, I am changing the initial message type to "message" and the final message type to "typing." With these changes, it is working for me.

HesselWellema commented 1 week ago

@LuisM000 This seems to work. Great, you saved my day. We will see if this was by design. Again, thanks a lot!

HesselWellema commented 6 days ago

@LuisM000 small addition. the typing events where flooding my transscriptlogger. I decided to ignore them in the transcript logger and use type 'messsage' as event for the final chunk. That also works.

HesselWellema commented 3 days ago

I will close the issue because my problem is solved.