microsoft / BotFramework-Services

Microsoft Bot Framework Services
Creative Commons Attribution 4.0 International
38 stars 11 forks source link

BadArgument: Incoming request was redirected to the wrong geo! Current geo: EMEA, Target geo: EUDB #339

Closed michaelwittig closed 2 years ago

michaelwittig commented 2 years ago

Hi,

we recently received a new error that we never saw before when calling the GET /v3/conversations/${conversationId}/members API.

 {
    "code": "BadArgument",
    "message": "Incoming request was redirected to the wrong geo! Current geo: EMEA, Target geo: EUDB"
}

The tenant's service URL so far was https://smba.trafficmanager.net/de/. Is it possible that this value changes? is there an event indicating the change? Could this error be related to the "Multi-Geo capabilities"?

Thanks

stevkan commented 2 years ago

@michaelwittig, this appears to be related to an on-going initiative that Microsoft has been pursuing for the last year. In short, the EUDB stands for EU Data Boundary and represents a greater commitment to not transfer data outside the EU. You can read the latest FAQ here, EU Data Boundary for the Microsoft Cloud, as well as this article for additional context, EU Data Boundary for the Microsoft Cloud: A progress report.

The "updated" date on the first article is May 12 which is the day prior to when the issue started for you (or, at least, when you posted this issue). This customer experienced a similar issue however in their case it resolved itself. Is this still blocking you?

Unfortunately, beyond the above, I don't have much more information around how to resolve it. The service URL you posted suggests to me that your service is located in Germany, or thereabouts. But a related service is located in the EMEA region? My assumption is the EUDB is blocking the sending of the "members" data outside the EU to the resource located in the EMEA region.

To start, can you verify for me if any of the above regarding regions is true, and which services are located where, both regionally and with respect to one another in Azure (i.e. are they in the same resource group, rely on the same AAD app registration, etc.)? Knowing this may help to determine who would be best to help.

stevkan commented 2 years ago

@michaelwittig, I also notified the Services team. Someone from that team will move forward in assisting you.

michaelwittig commented 2 years ago

Thanks @stevkan , let me answer your questions below:

We first saw this error at around 2022-05-13T14:07:24.576 UTC which should be 07:07 PDT (similar to what is reported on SO).

I don't know if the issue is/was transient. We removed the message from our queue too early to see the error go away. I have not seen this error again since then (not sure if this is good or bad).

We store the service url together with each tenant. If we send a message to a channel, we lookup the tenants service url and use it to deliver the message. Our service itself runs on AWS in us-east-1. We send raw HTTP requests to the API. No SDK/tool/magic used.

Regarding your Azure questions... I'm not very knowledgeable in this area but I will try my best:

I hope that helps.

luhan2017 commented 2 years ago

@michaelwittig, based on your service url: "https://smba.trafficmanager.net/de/", your request is sending to SMBA service which is a Teams specific scenario and is not owned by ABS team. I have created an incident to SMBA team, I will update this thread when I get a response from them.

michaelwittig commented 2 years ago

Thanks @luhan2017

luhan2017 commented 2 years ago

@michaelwittig, I didn't got any response from smba team for more than 10 days, do you know how to create an incident for them? Maybe you can open an issue for them directly?

michaelwittig commented 2 years ago

@luhan2017 no, sorry. I don't know what smba stands for, nor who is responsible for helping developers write chatbots for Teams. So far I have opened issues on GitHub and mostly received a response....

luhan2017 commented 2 years ago

@michaelwittig I got a response from Teams team, here is the response.

We did get IcM for that issue in May as well but it should be fixed by now (probably starting from May 20th it's fixed). Please check with a customer if they can still a repro. From our telemetry we no longer see those errors

Could you please try again and let me know if it still not works.

michaelwittig commented 2 years ago

@luhan2017 Thanks. I have not seen this error again as well.