send message to AU/APAC with error 400,payload is incorrect

[Jeff-Zhang-Github]

Version

springboot

Describe the bug

adapter.continueConversation( appId, reference, turnContext -> turnContext .sendActivity(message) .thenApply(resourceResponse -> null) );

when bot send message to AU(https://smba.trafficmanager.net/au/)/APAC(https://smba.trafficmanager.net/apac/) with error 400,payload is incorrect. but rest is normal(https://smba.trafficmanager.net/amer/).

java.util.concurrent.CompletionException: com.microsoft.bot.connector.rest.ErrorResponseException: Status code 400, {"error":{"code":"BadSyntax","message":"Payload is incorrect"}} at java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:314) at java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:319) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:645) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) at retrofit2.CompletableFutureCallAdapterFactory$ResponseCallAdapter$2.onResponse(CompletableFutureCallAdapterFactory.java:119) at retrofit2.OkHttpCall$1.onResponse(OkHttpCall.java:129) at okhttp3.RealCall$AsyncCall.execute(RealCall.java:174) at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: com.microsoft.bot.connector.rest.ErrorResponseException: Status code 400, {"error":{"code":"BadSyntax","message":"Payload is incorrect"}} at jdk.internal.reflect.GeneratedConstructorAccessor56.newInstance(Unknown Source) at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:490) at com.microsoft.bot.restclient.ServiceResponseBuilder.build(ServiceResponseBuilder.java:118) at com.microsoft.bot.connector.rest.RestConversations.replyToActivityDelegate(RestConversations.java:494) at com.microsoft.bot.connector.rest.RestConversations.lambda$replyToActivity$5(RestConversations.java:471) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:642) ... 9 common frames omitted

converstaion

I didn't change the conversation

{ "activityId": "activityId", "user": { "id": "2id", "aadObjectId": "aadObjectId" }, "bot": { "id": "botId", "name": "botName" }, "conversation": { "group": false, "isGroup": false, "conversationType": "personal", "tenantId": "tenantId", "id": "id" }, "channelId": "msteams", "serviceUrl": "https://smba.trafficmanager.net/au/" }

To Reproduce

    1.Receiving a User Message
    2.Save the conversation locally
    3.Logical processing
    4.Send a message to the user using the saved conversation
    5.Error sending message to Australian user，US user can receive messages normally

[InfinytRam]

Hi @Jeff-Zhang-Github, we are investigating.

[InfinytRam]

@Jeff-Zhang-Github, Could you please share the steps to reproduce this issue?

[Jeff-Zhang-Github]

Hi,@ramfattah, I updated the issue, and now the service seems normal. It has been restored around 2022-09-13 09:31:04 UTC,Let me know if you find out.

[hami-sh]

+1 -- we also experienced this outage in the AU region. Sorted itself out around the same time.

Any reason why this occurred?

[Jeff-Zhang-Github]

This problem occurred again today, around 2022-09-15 08:55UTC, for ten minutes or so, including EMEA and AMER.

[InfinytRam]

Hi @Jeff-Zhang-Github,

Sorry, I'm not sure why that is happening. I've transferred this issue to the BotFramework-Services repository as there seem to be outages for users in Australia. Maybe someone here will have some ideas.

CC: @boydc2014

[boydc2014]

Thank you @ramfattah. This url is Teams service endpoint, i will open an internal ticket to Teams and see what they get.

[boydc2014]

Sorry for the late response. I've opened ticket to Teams team in internal system and they've confirmed this is due to a recent regression they got related to Newstonsoft.Json update. They've fixed it at 9/13. Things should be back then.