azure-cli: Operation returned an invalid status 'Forbidden'

Expected behavior:

After applying all steps in https://goteleport.com/docs/application-access/cloud-apis/azure/ (Managed Identity is affected to the VM, Reader Role as been given, I tested also Contributor and Owner)

tsh apps login azure-cli --azure-identity teleport-azure --debug

should return

<<Azure JSON>>

Logged into Azure app "azure-cli".
Your identity: xxxx

As explained in https://goteleport.com/docs/application-access/cloud-apis/azure/

Current behavior:

Instead I get an Operation returned an invalid status 'Forbidden':

$ tsh apps login azure-cli --azure-identity teleport-azure 
Operation returned an invalid status 'Forbidden'
ERROR: exit status 1

ERROR: failed to automatically login with `az login` using identity "/subscriptions/00000-000000-0000000/resourcegroups/bastion/providers/microsoft.managedidentity/userassignedidentities/teleport-azure"; run with --debug for details
exit status 1

Bug details:

Teleport version: 3.3.1
Recreation steps: Execute https://goteleport.com/docs/application-access/cloud-apis/azure/ (note one change: I did not create a separate VM I used my main teleport VM)
Debug logs

On the managed VM:
$ /usr/bin/az login --identity -u /subscriptions/000000-0000000-00000/resourcegroups/bastion/providers/microsoft.managedidentity/userassignedidentities/teleport-azure

<<List of usable subscriptions>>
So the az login is working on the VM.

On my workstation:
$ tsh apps login azure-cli --azure-identity teleport-azure --debug
...
2023-08-04T12:23:47+02:00 DEBU [KEYAGENT]  Validated host teleport.internal.sgforge.com:443. client/keyagent.go:373
2023-08-04T12:23:47+02:00 INFO [CLIENT]    Successful auth with proxy teleport.internal.sgforge.com:443. client/api.go:3055
2023-08-04T12:23:47+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-08-04 21:18:02 +0000 UTC". client/client_store.go:106
2023-08-04T12:23:47+02:00 DEBU [KEYSTORE]  Teleport TLS certificate valid until "2023-08-04 21:18:02 +0000 UTC". client/client_store.go:106
2023-08-04T12:23:47+02:00 INFO [LOCALPROX] Starting HTTP access proxy alpnproxy/local_proxy.go:305
2023-08-04T12:23:47+02:00 DEBU [TSH]       Running command: "/usr/bin/az login --identity -u /subscriptions/000000-000000-0000000/resourcegroups/bastion/providers/microsoft.managedidentity/userassignedidentities/teleport-azure" tsh/app_azure.go:182
2023-08-04T12:23:48+02:00 DEBU             Started forwarding request for "azure-msi.teleport.dev:443". alpnproxy/forward_proxy.go:357
2023-08-04T12:23:48+02:00 INFO [CA]        Generating TLS certificate SERIALNUMBER=295123642963878312798032944828672738502,CN=azure-msi.teleport.dev,O=Teleport dns_names:[azure-msi.teleport.dev] key_usage:5 not_after:2023-08-04 21:18:02 +0000 UTC tlsca/ca.go:1111
2023-08-04T12:23:48+02:00 INFO [AZURE_MSI] MSI: returning token for identity /subscriptions/000000-000000-0000000/resourcegroups/bastion/providers/microsoft.managedidentity/userassignedidentities/teleport-azure alpnproxy/azure_msi_middleware.go:128
2023-08-04T12:23:48+02:00 DEBU             Stopped forwarding request for "azure-msi.teleport.dev:443". alpnproxy/forward_proxy.go:363
2023-08-04T12:23:48+02:00 DEBU             Started forwarding request for "azure-msi.teleport.dev:443". alpnproxy/forward_proxy.go:357
2023-08-04T12:23:48+02:00 INFO [AZURE_MSI] MSI: returning token for identity /subscriptions/000000-000000-0000000/resourcegroups/bastion/providers/microsoft.managedidentity/userassignedidentities/teleport-azure alpnproxy/azure_msi_middleware.go:128
2023-08-04T12:23:48+02:00 DEBU             Stopped forwarding request for "azure-msi.teleport.dev:443". alpnproxy/forward_proxy.go:363
2023-08-04T12:23:48+02:00 DEBU             Started forwarding request for "management.azure.com:443". alpnproxy/forward_proxy.go:357
2023-08-04T12:23:48+02:00 INFO [CA]        Generating TLS certificate SERIALNUMBER=295123642963878312798032944828672738502,CN=management.azure.com,O=Teleport dns_names:[management.azure.com] key_usage:5 not_after:2023-08-04 21:18:02 +0000 UTC tlsca/ca.go:1111
Operation returned an invalid status 'Forbidden'
2023-08-04T12:23:48+02:00 DEBU             Stopped forwarding request for "management.azure.com:443". alpnproxy/forward_proxy.go:363
2023-08-04T12:23:48+02:00 INFO [LOCALPROX] HTTP access proxy stopped alpnproxy/local_proxy.go:331

ERROR REPORT:
Original Error: *exec.ExitError exit status 1
Stack Trace:
    github.com/gravitational/teleport/tool/tsh/tsh.go:509 main.(*CLIConf).RunCommand
    github.com/gravitational/teleport/tool/tsh/app_azure.go:192 main.(*azureApp).RunCommand
    github.com/gravitational/teleport/tool/tsh/app_azure.go:62 main.onAzure
    github.com/gravitational/teleport/tool/tsh/tsh.go:1341 main.Run
    github.com/gravitational/teleport/tool/tsh/tsh.go:530 main.main
    runtime/proc.go:250 runtime.main
    runtime/asm_amd64.s:1598 runtime.goexit
User Message: exit status 1

ERROR REPORT:
Original Error: *exec.ExitError exit status 1
Stack Trace:
    github.com/gravitational/teleport/tool/tsh/tsh.go:509 main.(*CLIConf).RunCommand
    github.com/gravitational/teleport/tool/tsh/app.go:158 main.onAppLogin
    github.com/gravitational/teleport/tool/tsh/tsh.go:1271 main.Run
    github.com/gravitational/teleport/tool/tsh/tsh.go:530 main.main
    runtime/proc.go:250 runtime.main
    runtime/asm_amd64.s:1598 runtime.goexit
User Message: failed to automatically login with `az login` using identity &#34;/subscriptions/000000-000000-0000000/resourcegroups/bastion/providers/microsoft.managedidentity/userassignedidentities/teleport-azure&#34;; run with --debug for details
    exit status 1

gravitational / teleport

azure-cli: Operation returned an invalid status 'Forbidden' #30029