search

microsoft / BotFramework-Services

Microsoft Bot Framework Services
Creative Commons Attribution 4.0 International
38 stars 11 forks source link

smba.trafficmanager.net/amer returning 404 when sending Teams message #349

Closed martinlarosa closed 1 year ago

martinlarosa commented 1 year ago

I'm receiving a lot of errors from the bot backend when trying to reply to a user in Teams. This is not happening to my particular user so I can't reproduce it.

The exception I'm seeing being logged is a Not Found being returned by: POST https://smba.trafficmanager.net/amer/v3/conversations/{id}/activities/{id}

I'm using a C# bot with Bot Framework Composer and using 4.17.2 version of the Bot Builder SDK Anything you know or that I should consider for troubleshooting this?

boydc2014 commented 1 year ago

Hi, @martinlarosa this url is the Teams service url where bot sending messages to, is this a transient issue or a persistent issue?

If you have a few particular timestamps of failed requests, i can route this to Teams service team for further investigation.

martinlarosa commented 1 year ago

It used to happen sparsely, but now it's happening a lot. Multiple times an hour when I created the ticket, less so now.

2022-11-11 01:56:22.352 UTC 2022-11-11 01:56:34.082 UTC 2022-11-11 02:19:13.028 UTC 2022-11-11 02:34:40.545 UTC

martinlarosa commented 1 year ago

Hi! I wanted to know if there's any news on this?

boydc2014 commented 1 year ago

Sorry for the delay, @martinlarosa, I've opened a ticket to Teams messaging API team. Will let you know once they get feedback

martinlarosa commented 1 year ago

Hi @boydc2014, thank you for opening that ticket. Is that publicly available or something internal? It's happening a lot and caused us a production issue with a customer last week.

aosolis commented 1 year ago

The response to the 404 should contain a code that indicates what could not be found. See https://learn.microsoft.com/en-us/microsoftteams/platform/bots/how-to/conversations/conversation-messages?tabs=dotnet#status-codes-from-bot-conversational-apis

image

For this POST /v3/conversations/{id}/activities endpoint, the usual reason is the referenced conversation ID has been deleted.

martinlarosa commented 1 year ago

I don't see this in the logs, the message is being sent through Bot Framework .net sdk.

luhan2017 commented 1 year ago

@martinlarosa, It's an internal ticket, they are asking for more information Can you please share more details like tenant id, request id? Here is the link: https://portal.microsofticm.com/imp/v3/incidents/details/353458654/home

aosolis commented 1 year ago

The error code is in the body of the response, which is captured in the Exception or Error object thrown by the SDK. The content will be similar to the one below:

{"error":{"code":"ConversationNotFound","message":"Conversation not found."}}

See for example the Body property of ErrorResponseException: https://github.com/microsoft/botbuilder-dotnet/blob/main/libraries/Microsoft.Bot.Connector/ErrorResponseException.cs

If you provide your bot ID and recent timestamps of instances of this error (within 2 weeks), we can check our logs.

luhan2017 commented 1 year ago

@martinlarosa is this issue still valid? could you please provide more details for this?

martinlarosa commented 1 year ago

Hi, it is still happening, and I'm even getting some 403s

We'll add some more logging to try to find the real error message

martinlarosa commented 1 year ago

The response to the 404 should contain a code that indicates what could not be found. See https://learn.microsoft.com/en-us/microsoftteams/platform/bots/how-to/conversations/conversation-messages?tabs=dotnet#status-codes-from-bot-conversational-apis

image

For this POST /v3/conversations/{id}/activities endpoint, the usual reason is the referenced conversation ID has been deleted.

Regarding that, could it be that the conversation ID expires? We didn't expect that. We are actually caching the conversation ids in one of the features where we send proactive messages so that we don't hit the Graph API if we already have the value

martinlarosa commented 1 year ago

@aosolis @luhan2017 would you comment on my previous question?

YunnyChung commented 1 year ago

Hey @martinlarosa, to your question - could it be that the conversation ID expires? We didn't expect that. We are actually caching the conversation ids in one of the features where we send proactive messages so that we don't hit the Graph API if we already have the value I think most likely the thread was deleted by the user. But to really understand the root cause, I need to check logs.

Do you still hit the same issue? If so, could you share the timestamp in UTC, bot id , tenant id and the error code?

YunnyChung commented 1 year ago

Any updates on this, @martinlarosa?

martinlarosa commented 1 year ago

Is it really necessary to share the bot id and tenant id in a public channel?

Here's an occurrence that has just happened in UTC: 15/3/2023, 16:09:43.5809337

https://smba.trafficmanager.net/amer/v3/conversations/{someId}/activities/{someId} Result code: 404 OperationId 71a0c935423f289582c1be0d9be85fc9

YunnyChung commented 1 year ago

Hey @martinlarosa , do you have access to the link that @luhan2017 shared? If so, could you share the requested information in the ticket? https://portal.microsofticm.com/imp/v3/incidents/details/353458654/home image

We rely on bot id and tenant id to narrow down to specific requests & cross checks due to the amount of traffic.

With the given information, I searched the send activity requests during the given timestamp, and there are quite a few requests with 404 across different bots :/

If you don't have access to the link, so therefore you cannot share the info.. if you can share multiple time stamps (in UTC), I can check each of the requests and infer the necessary data.

martinlarosa commented 1 year ago

I don't have access to the link. Here are some timestamps from the last 24hs in UTC:

2023-3-20T14:34:23.6738341 2023-3-20T14:34:23.8924655 2023-3-20T10:37:46.743094 2023-3-20T10:37:46.9479881 2023-3-19T15:48:23.7102831 2023-3-19T15:48:23.9064328 2023-3-19T15:48:23.7102831 2023-3-20T05:53:52.3286748 2023-3-20T05:53:52.6622061 2023-3-19T19:27:49.4917231 2023-3-19T19:27:49.7409761 2023-3-20T01:13:26.6598817 2023-3-20T01:13:26.3532828 2023-3-20T14:34:23.6738341 2023-3-20T14:34:23.8924655

Wajeed-msft commented 1 year ago

@martinlarosa - could you please share you Bot Id and Tenant Id to Microsoft Teams Community Help alias?

martinlarosa commented 1 year ago

Done, thank you!

Wajeed-msft commented 1 year ago

@martinlarosa - could you please share Tenant Id as well?

martinlarosa commented 1 year ago

I missed your email from last week. I just replied with extra information and new timestamps.

Thank you

martinlarosa commented 1 year ago

Any news?

luhan2017 commented 1 year ago

@martinlarosa @YunnyChung , could you please follow up through email? I am going to close this ticket because it is not a real ABS issue. Please reopen if needed.

Steffi3rd commented 1 year ago

Hello guys, I have the same issue.

Do you find a solution?

Our bot receive ConversationNotFound since this weekend whereas user still have the conversation and the user exist in the storage.

When I delete the user, I have "User Not found" (so this is normal).

The bug is ConversationNotFound whereas conversation exists for the user.

chu-tianshu commented 1 year ago

Hi @Steffi3rd , is the issue still happening to you? Could you share more info for us to troubleshoot?

Steffi3rd commented 1 year ago

@cts2003258 I found an alternative solution here : https://github.com/OfficeDev/TeamsFx/issues/8759#issuecomment-1561159073