Failure to invoke dapr actors on production environment

[onionhammer]

This issue is a: (mark with an x)

• [x] bug report -> please search issues before submitting
• [ ] documentation issue or request
• [x] regression (a behavior that used to work and stopped in a new release)

Issue description

Currently when trying to invoke dapr actors hosted on one ACA app from another ACA app in the same environment, I am seeing the following error:

time="2024-02-06T22:28:45.315297799Z" level=debug msg="{ERR_ACTOR_INVOKE_METHOD error invoke actor method: failed to invoke target 169.254.11.18:50002 after 3 retries. Error: rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 169.254.11.18:50002: i/o timeout\"}" app_id=my-app-name instance=my-app-name--snip scope=dapr.runtime.http type=log ver=1.11.6

Steps to reproduce

1. Have a buggy ACA environment

Expected behavior [What you expected to happen.] Actor invocations should work

Actual behavior [What actually happened.] time="2024-02-06T22:28:45.315297799Z" level=debug msg="{ERR_ACTOR_INVOKE_METHOD error invoke actor method: failed to invoke target 169.254.11.18:50002 after 3 retries. Error: rpc error: code = Unavailable desc = last connection error: connection error: desc = \"transport: Error while dialing: dial tcp 169.254.11.18:50002: i/o timeout\"}" app_id=my-app-name instance=my-app-name--snip scope=dapr.runtime.http type=log ver=1.11.6

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context

This exact code was working until I tore down the environment and re-creating it.

Ping me on Discord @Onionhammer

@berndverst Tried opening a support ticket, but the techs there arent super helpful, they are reading the logs but cant do anything

[ItalyPaleAle]

I believe the issue was fixed in your environment after engaging with Azure Support @onionhammer

We are rolling out a permanent fix for all environments, which is expected to be completed in the next days/weeks. This will coincide with the update to Dapr 1.12

[onionhammer]

I believe the issue was fixed in your environment after engaging with Azure Support @onionhammer

We are rolling out a permanent fix for all environments, which is expected to be completed in the next days/weeks. This will coincide with the update to Dapr 1.12

This has been hotfixed twice, and broken again twice, it's currently not working so I would not call this resolved at all.

[ItalyPaleAle]

CC: @xiangj1 @sanchitmehta who control the releases

[marchagele]

I am running into the same problem currently. Are there any news?

[onionhammer]

@marchagele you should open a support ticket and refer to this issue. What region are you in?

[marchagele]

@marchagele you should open a support ticket and refer to this issue. What region are you in?

@onionhammer WestEurope, which platform to open the support ticket? any link to it? Kind regards

[onionhammer]

any link to it?

https://portal.azure.com/#view/Microsoft_Azure_Support/HelpAndSupportBlade/~/overview

[marchagele]

Thanks @onionhammer!