crossplane-contrib / provider-upjet-azure

Official Azure Provider for Crossplane by Upbound.
Apache License 2.0
53 stars 69 forks source link

[Bug]: OIDC does not work with sovereign clouds. Client/azure.go is missing environment variable in func oidcAuth #743

Open DE-Wizard opened 1 month ago

DE-Wizard commented 1 month ago

Is there an existing issue for this?

Affected Resource(s)

internal/clients/azure.go ProviderConfig

Resource MRs required to reproduce the bug

--- apiVersion: azure.upbound.io/v1beta1 kind: ProviderConfig metadata: name: default spec: credentials: source: OIDCTokenFile msiEndpoint: https://login.microsoftonline.us/<mytenant>/v2.0/.well-known/openid-configuration oidcTokenFilePath: /var/run/secrets/azure/tokens/azure-identity-token environment: usgovernment subscriptionID: <mysubscription> tenantID: <mytenant> clientID: <myclient>

Steps to Reproduce

I have tried using the OIDC token provided by Azure Workload Identity.

What happened?

My resource-group import failed and the provider-family-azure put out a log saying AADSTS900382: Confidential Client is not supported in Cross Cloud request. I looked at the code and I noticed that in the func oidcAuth section that it was missing the if statement to check the environment to use the right endpoint as you are already doing in func spAuth and func msiAuth.

Relevant Error Output Snippet

2024-05-16T00:06:56Z    DEBUG   provider-azure  Connecting to the service provider  {"uid": "0bbb7209-6f70-426a-a874-887e24cf4f45", "name": "rg-mgmt-cluster", "gvk": "azure.upbound.io/v1beta1, Kind=ResourceGroup"}
2024-05-16T00:06:57Z    DEBUG   provider-azure  Cannot connect to provider  {"controller": "managed/azure.upbound.io/v1beta1, kind=resourcegroup", "request": {"name":"rg-mgmt-cluster"}, "uid": "0bbb7209-6f70-426a-a874-887e24cf4f45", "version": "4061648", "external-name": "rg-mgmt-cluster", "error": "cannot initialize the Terraform plugin SDK async external client: cannot get terraform setup: failed to configure the no-fork Azure client: failed to configure the provider: [{0 building account: could not acquire access token to parse claims: clientCredentialsToken: received HTTP status 400 with response: {\"error\":\"invalid_request\",\"error_description\":\"AADSTS900382: Confidential Client is not supported in Cross Cloud request. Trace ID: a183dc20-c730-41db-87dd-39f532600e00 Correlation ID: 1653a396-4a20-4790-8558-26bf6be7b9bd Timestamp: 2024-05-16 00:06:56Z\",\"error_codes\":[900382],\"timestamp\":\"2024-05-16 00:06:56Z\",\"trace_id\":\"a183dc20-c730-41db-87dd-39f532600e00\",\"correlation_id\":\"1653a396-4a20-4790-8558-26bf6be7b9bd\"}  []}]", "errorVerbose": "failed to configure the provider: [{0 building account: could not acquire access token to parse claims: clientCredentialsToken: received HTTP status 400 with response: {\"error\":\"invalid_request\",\"error_description\":\"AADSTS900382: Confidential Client is not supported in Cross Cloud request. Trace ID: a183dc20-c730-41db-87dd-39f532600e00 Correlation ID: 1653a396-4a20-4790-8558-26bf6be7b9bd Timestamp: 2024-05-16 00:06:56Z\",\"error_codes\":[900382],\"timestamp\":\"2024-05-16 00:06:56Z\",\"trace_id\":\"a183dc20-c730-41db-87dd-39f532600e00\",\"correlation_id\":\"1653a396-4a20-4790-8558-26bf6be7b9bd\"}  []}]\ngithub.com/upbound/provider-azure/internal/clients.configureNoForkAzureClient\n\tgithub.com/upbound/provider-azure/internal/clients/azure.go:125\nmain.main.TerraformSetupBuilder.func11\n\tgithub.com/upbound/provider-azure/internal/clients/azure.go:116\ngithub.com/crossplane/upjet/pkg/controller.(*TerraformPluginSDKConnector).Connect\n\tgithub.com/crossplane/upjet@v1.3.0/pkg/controller/external_tfpluginsdk.go:231\ngithub.com/crossplane/upjet/pkg/controller.(*TerraformPluginSDKAsyncConnector).Connect\n\tgithub.com/crossplane/upjet@v1.3.0/pkg/controller/external_async_tfpluginsdk.go:55\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:246\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:892\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650\nfailed to configure the no-fork Azure client\nmain.main.TerraformSetupBuilder.func11\n\tgithub.com/upbound/provider-azure/internal/clients/azure.go:116\ngithub.com/crossplane/upjet/pkg/controller.(*TerraformPluginSDKConnector).Connect\n\tgithub.com/crossplane/upjet@v1.3.0/pkg/controller/external_tfpluginsdk.go:231\ngithub.com/crossplane/upjet/pkg/controller.(*TerraformPluginSDKAsyncConnector).Connect\n\tgithub.com/crossplane/upjet@v1.3.0/pkg/controller/external_async_tfpluginsdk.go:55\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:246\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:892\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650\ncannot get terraform setup\ngithub.com/crossplane/upjet/pkg/controller.(*TerraformPluginSDKConnector).Connect\n\tgithub.com/crossplane/upjet@v1.3.0/pkg/controller/external_tfpluginsdk.go:234\ngithub.com/crossplane/upjet/pkg/controller.(*TerraformPluginSDKAsyncConnector).Connect\n\tgithub.com/crossplane/upjet@v1.3.0/pkg/controller/external_async_tfpluginsdk.go:55\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:246\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:892\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650\ncannot initialize the Terraform plugin SDK async external client\ngithub.com/crossplane/upjet/pkg/controller.(*TerraformPluginSDKAsyncConnector).Connect\n\tgithub.com/crossplane/upjet@v1.3.0/pkg/controller/external_async_tfpluginsdk.go:57\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*NopDisconnecter).Connect\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:246\ngithub.com/crossplane/crossplane-runtime/pkg/reconciler/managed.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/reconciler/managed/reconciler.go:892\ngithub.com/crossplane/crossplane-runtime/pkg/ratelimiter.(*Reconciler).Reconcile\n\tgithub.com/crossplane/crossplane-runtime@v1.16.0-rc.1.0.20240424114634-8641eb2ba384/pkg/ratelimiter/reconciler.go:54\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tsigs.k8s.io/controller-runtime@v0.17.3/pkg/internal/controller/controller.go:227\nruntime.goexit\n\truntime/asm_amd64.s:1650"}

Crossplane Version

1.15.2

Provider Version

1.1.0

Kubernetes Version

1.28.9

Kubernetes Distribution

AKS US Gov GCC-High

Additional Info

func oidcAuth should look like this: (see added if statement just before return nil) `func oidcAuth(pc v1beta1.ProviderConfig, ps terraform.Setup) error { if pc.Spec.SubscriptionID == nil || len(pc.Spec.SubscriptionID) == 0 { return errors.New(errSubscriptionIDNotSet) } if pc.Spec.TenantID == nil || len(pc.Spec.TenantID) == 0 { return errors.New(errTenantIDNotSet) } if pc.Spec.ClientID == nil || len(pc.Spec.ClientID) == 0 { return errors.New(errClientIDNotSet) } // OIDC Token File Path defaults to a projected-volume path mounted in the pod running in the AKS cluster, when workload identity is enabled on the pod. ps.Configuration[keyOidcTokenFilePath] = defaultOidcTokenFilePath if pc.Spec.OidcTokenFilePath != nil { ps.Configuration[keyOidcTokenFilePath] = pc.Spec.OidcTokenFilePath } ps.Configuration[keySubscriptionID] = pc.Spec.SubscriptionID ps.Configuration[keyTenantID] = pc.Spec.TenantID ps.Configuration[keyClientID] = pc.Spec.ClientID ps.Configuration[keyUseOIDC] = "true" if pc.Spec.Environment != nil { ps.Configuration[keyEnvironment] = pc.Spec.Environment } return nil

}`

Because of environment restrictions I have an issue trying to test this minor change and I also cannot GPG sign my commits from my work system. Even if I make the change and sign it from my personal computer I don't have a good way to test. This should be a non-breaking change as the constructs are already there.

waterfoul commented 3 weeks ago

Did that fix work for you? I tried it and it didn't fix the issue

waterfoul commented 3 weeks ago

Hmmm, looking at the provider that should work....

DE-Wizard commented 3 weeks ago

It did work for me as it is the match for the other auth types that use the same sdk.

ERIC DEITRICK

Chief Infrastructure Engineer

M: 386.576.3441 @.**@.>

Arcfield Proprietary E-MAIL CONTENT CLASSIFICATION NOTICE CONFIDENTIAL AND/OR PROPRIETARY INFORMATION OF ARCFIELD. This e-mail message and/or its attachment contains confidential and/or proprietary information of Arcfield that may only be received, disclosed, or used as authorized by Arcfield. The information in this message may be exempt from release under the Freedom of Information Act. If you received this message in error, please delete all copies and promptly notify the sender.


From: Aaron Aichlmayr @.> Sent: Wednesday, June 12, 2024 6:01:35 PM To: crossplane-contrib/provider-upjet-azure @.> Cc: Deitrick, Eric (STC) @.>; Author @.> Subject: [EXTERNAL] Re: [crossplane-contrib/provider-upjet-azure] [Bug]: OIDC does not work with sovereign clouds. Client/azure.go is missing environment variable in func oidcAuth (Issue #743)

Hmmm, looking at the provider that should work....

— Reply to this email directly, view it on GitHubhttps://urldefense.us/v2/url?u=https-3A__github.com_crossplane-2Dcontrib_provider-2Dupjet-2Dazure_issues_743-23issuecomment-2D2163973968&d=DwMFaQ&c=L4IogQUxJwxc-F14PJeZBy0aL1nbCId3JgRjMgunPoM&r=BPK4M53lslq4H95CXMHdJHr5QwdWJy-t3oSV0vZLyOE&m=KMasjlnHg-N0lWC9ZHYrRCqTN8tkMt94XyCiMJCjlSMVnYup4MaWd6vGNC-AhzNF&s=Nmz0q1I3R4r9BthF9GeQ5_muu_CwLOl_4RXtnhE9WIM&e=, or unsubscribehttps://urldefense.us/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_BDXHF56IQJNZZFO5X42IPHDZHDAL7AVCNFSM6AAAAABHZEMLDKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNRTHE3TGOJWHA&d=DwMFaQ&c=L4IogQUxJwxc-F14PJeZBy0aL1nbCId3JgRjMgunPoM&r=BPK4M53lslq4H95CXMHdJHr5QwdWJy-t3oSV0vZLyOE&m=KMasjlnHg-N0lWC9ZHYrRCqTN8tkMt94XyCiMJCjlSMVnYup4MaWd6vGNC-AhzNF&s=R_TpDef36ckpS80qLAGDpbJ8kDKXe4-GSrTlc8XF5ZY&e=. You are receiving this because you authored the thread.Message ID: @.***>

waterfoul commented 3 weeks ago

Looks like my first attempt didn't apply correctly. The fix works