AzureAD OIDC and Terragrunt issues

[maciob]

Hi @MatthewJohn, I'm trying to set up SSO with AzureAD OIDC. Im quite sure that my setup is right, but I'm getting following errors from it:

ERROR:terrareg.server:Exception on /openid/login [GET] Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request rv = self.dispatch_request() ^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request return self.ensure_sync(self.view_functions[rule.endpoint])(view_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/flask_restful/init.py", line 467, in wrapper resp = resource(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/flask/views.py", line 107, in view return current_app.ensure_sync(self.dispatch_request)(kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/flask_restful/init.py", line 582, in dispatch_request resp = meth(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/app/terrareg/server/error_catching_resource.py", line 34, in get return self._get(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/terrareg/server/api/open_id_initiate.py", line 13, in _get redirect_url, state = terrareg.openid_connect.OpenidConnect.get_authorize_redirect_url() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/terrareg/openid_connect.py", line 74, in get_authorize_redirect_url auth_url = cls.obtain_issuer_metadata().get('authorization_endpoint', None) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ AttributeError: 'NoneType' object has no attribute 'get'

I've set up: OPENID_CONNECT_CLIENT_ID: xxx OPENID_CONNECT_CLIENT_SECRET: xxx OPENID_CONNECT_DEBUG: "True" OPENID_CONNECT_ISSUER: "https://login.microsoftonline.com//v2.0" OPENID_CONNECT_LOGIN_TEXT: "Login using AzureAD" OPENID_CONNECT_SCOPES: '["openid","profile","email","offline_access","groups"]' Still Im getting this error. Version: 3.8.0 Might want to take a look at that?

[MatthewJohn]

Created gitlab issue: https://gitlab.dockstudios.co.uk/pub/terrareg/-/issues/533 gitlab-issue-id:533

[maciob]

Update: We are encountering this error on k8 with nginx-ingress set up as a reverse proxy. I've setup local development environment and here Im getting invalid response from SSO which is pretty much what I expected for a local environment. More, magicly the button with to login using azuread appears, so i guess there may be something wrong with the way we are passing env variables.

[maciob]

Update: I've managed to set it up. For some damn reason DOMAIN_NAME which is supposed to be depricated is needed on k8 for this to work out.

[MatthewJohn]

Hmm, interesting - I think for SAML that PUBLIC_URL is required (which was the replacement for the deprecated DOMAIN_NAME config - I suspect it worked locally because it defaults to the request header (which will presumably be correct without any reverse proxies etc locallly)

Matt

Sent from my iPhone

On 17 Jul 2024, at 20:45, maciob @.***> wrote:

 Closed #55 as completed.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.

[MatthewJohn]

Having said that, I'm not too sure why it would result in that error, since the data that's None is being pulled from the ISSUER's well-known config URL, which wouldn't take DOMAIN_NAME into account

[maciob]

It was fairly missleading. I've only realized that I'm missing DOMAIN_NAME env after checking out the code. But good thing that it started working.

[MatthewJohn]

I've only realized that I'm missing DOMAIN_NAME env after checking out the code.

@maciob Where did you see this exactly in the code?

Unless there's a bug, there's a common helper to obtain the domain, protocol etc. which uses PUBLIC_URL, but falls back to DOMAIN_NAME (for backwards compatibility for now). Did you have PUBLIC_URL configured, as this is documented here: https://matthewjohn.github.io/terrareg/security/#openid-connect

[maciob]

@MatthewJohn Yeah I did had PUBLIC_URL configured. I've seen it in terrareg/openid_connect.py line 25. You return a boolean based on domain from get_public_url_details. It should have returned domain based on PUBLIC_URL. It's weird, but I think it might be the way I'm passing env variables. I basicly created a section in which I give all of the env variables in helm chart and I either set them to ""(default) or pass some new values. Bug occured when DOMAIN_NAME was set to "" and PUBLIC_URL was set to "domain.example". I encountered very simillar thing with SAML. The button occured whenever vales of saml were set to "", but they dissapeared whenever I commented them. Weird, but managable.

BTW. Does terrareg support tfr protocol? I'm trying to download the module using terragrunt, but I keep on getting 401. It works perfectly with terraform.

[MatthewJohn]

For the PUBLIC_URL, it should be set to a URL (including protocol), e.g. "https://domain.example". But yes, I think setting DOMAIN_NAME to an empty string will also cause issues. There's a lot of use of default values in the code if the env variables are unset, so having them default to an empty string is not ideal.

BTW. Does terrareg support tfr protocol?

What do you mean by tfr exactly? The terraform registry API spec?

[maciob]

So basicly terragrunt uses another syntax for source block, it looks something like this: terraform { source = "tfr://dns.example/namespace/module/provider?version=1.0.0" } Which is different to what source block in terraform looks like: module "module" { source = "dns.example/namespace/module/provider" version = "1.0.0" }

It was not a problem to get terraform version working well, but I really need it to start working with terragrunt version. Is it possible right now or terrareg just do not support it?

[MatthewJohn]

Hmm, It seems it should just be using the normal registry API. What error do you get? a 403, did you say? Was there any contents of the body from the response?

It's worth noting that if you are using/enforcing an analytics token, e.g.:

module "module" {
  source = "dns.example/blah__namespace/module/provider"
  version = "1.0.0"
}

you'd likely need to copy to the tfr one:

terraform {
  source = "tfr://dns.example/blah__namespace/module/provider?version=1.0.0"
}

[maciob]

It's the same case with the analytics token. Give me a sec I will copy paste it for you.

[maciob]

INFO[0000] Downloading Terraform configurations from tfr:///__//?version=1.0.0 into /Users/user/Desktop/.../.terragrunt-cache/TueDeyWgdtsaa82bNdpFk1N-m28/ryGMlhsuwtPIan30Wn9e917R8YA

DEBU[0001] downloading source url tfr:///__//?version=1.0.0 1 error occurred:

• error downloading 'tfr:///__//?version=1.0.0': Error downloading module from https:///v1/modules/__///1.0.0/download: error receiving HTTP data

ERRO[0001] downloading source url tfr:///__//?version=1.0.0 1 error occurred:

• error downloading 'tfr:///__//?version=1.0.0': Error downloading module from https:///v1/modules/__///1.0.0/download: error receiving HTTP data

ERRO[0001] Unable to determine underlying exit code, so Terragrunt will exit with error code 1

At the same time it works perfectly with terraform:

terraform init

Initializing the backend... Initializing modules... Downloading /__//gcp 1.0.0 for ...

in .terraform/modules/ Downloading registry.terraform.io/blah/blah/provider 19.0.0 for .blah... .blah in .terraform/modules/.blah/modules/ blah blah Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.

When I enter the https provided by terragrunt:

https:///v1/modules/__///1.0.0/download: error receiving HTTP data

I get 204, with X-Terraform-GET header which copied and pasted leads to module being downloaded by my browser.

[MatthewJohn]

In the mean time, I've just been tested Terragrunt to see what happens for me:

➜  terragrunt git:(main) ✗ cat terragrunt.hcl 
terraform {
  source = "tfr://local-dev.dock.studio/adga__test/my-first-module/aws?version=1.0.0"
}

And it seemed to work fine:

➜  terragrunt git:(main) ✗ ~/Downloads/terragrunt_linux_amd64 --terragrunt-tfpath=terraform init 
INFO[0000] Downloading Terraform configurations from tfr://local-dev.dock.studio/adga__test/my-first-module/aws?version=1.0.0 into /home/matthew/w/g/terrareg/example/terragrunt/.terragrunt-cache/3JwNGdz2WbFqCjYuOvcC1zbjX1Y/EModTpL0m063uaaYJoJROuSP4d0 

Initializing the backend...

Initializing provider plugins...
- Finding latest version of hashicorp/null...
- Installing hashicorp/null v3.2.2...
- Installed hashicorp/null v3.2.2 (signed by HashiCorp)

Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

╷
│ Warning: Provider development overrides are in effect
│ 
│ The following provider development overrides are set in the CLI
│ configuration:
│  - matthewjohn/terrareg in /home/matthew/.terraform.d/plugins
│  - mrparkers/keycloak in /home/matthew/.terraform.d/plugins
│ 
│ Skip terraform init when using provider development overrides. It is not
│ necessary and may error unexpectedly.
╵

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

[MatthewJohn]

Can you see any logs in terrareg - what response code is being returned for those failed requests? Also, for the download with x-terraform-get, which URL are you receiving? A git URL or is it a archive from Terrareg?

[maciob]

It seems that terrareg do not register these requests from terragrunt. It's archive from terrareg.

[maciob]

These are the env variables and values I'm using:

ADMIN_SESSION_EXPIRY_MINS: 60 ALLOW_CUSTOM_GIT_URL_MODULE_PROVIDER: "True" ALLOW_CUSTOM_GIT_URL_MODULE_VERSION: "True" ALLOW_FORCEFUL_MODULE_PROVIDER_REDIRECT_DELETION: "False" ALLOW_MODULE_HOSTING: "True" ALLOW_UNAUTHENTICATED_ACCESS: "False" ALLOW_UNIDENTIFIED_DOWNLOADS: "True" APPLICATION_NAME: "Terrareg" AUTOGENERATE_MODULE_PROVIDER_DESCRIPTION: "True" AUTOGENERATE_USAGE_BUILDER_VARIABLES: "True" AUTO_CREATE_MODULE_PROVIDER: "True" AUTO_CREATE_NAMESPACE: "False" AUTO_PUBLISH_MODULE_VERSIONS: "True" DATA_DIRECTORY: "/app/data" DEBUG: "True" DEFAULT_TERRAFORM_VERSION: "1.8.5" DEFAULT_UI_DETAILS_VIEW: "expanded" DELETE_EXTERNALLY_HOSTED_ARTIFACTS: "False" DISABLE_ANALYTICS: "False" DISABLE_TERRAREG_EXCLUSIVE_LABELS: "False" DOMAIN_NAME: "redacted" ENABLE_ACCESS_CONTROLS: "True" ENABLE_SECURITY_SCANNING: "True" LISTEN_PORT: "5000" OPENID_CONNECT_DEBUG: "True" OPENID_CONNECT_ISSUER: "https://login.microsoftonline.com/redacted/v2.0" OPENID_CONNECT_LOGIN_TEXT: "Login using OpenID Connect" OPENID_CONNECT_SCOPES: "openid,profile,email,offline_access" PUBLIC_URL: "redacted" SERVER: "waitress" TERRAFORM_EXAMPLE_VERSION_TEMPLATE: "{major}.{minor}.{patch}" TERRAFORM_OIDC_IDP_SIGNING_KEY_PATH: "/app/terrakey/signing_key.pem" TERRAFORM_PRESIGNED_URL_EXPIRY_SECONDS: "30" THREADED: "True" TRUSTED_NAMESPACES: "redacted" TRUSTED_NAMESPACE_LABEL: "Trusted" UPLOAD_DIRECTORY: "./data/UPLOAD" VERIFIED_MODULE_LABEL: "Verified" MIGRATE_DATABASE: "True" TERRAFORM_OIDC_IDP_SUBJECT_ID_HASH_SALT: "redacted" ADMIN_AUTHENTICATION_TOKEN: "redacted" ANALYTICS_AUTH_KEYS: "redacted" OPENID_CONNECT_CLIENT_ID: "redacted" OPENID_CONNECT_CLIENT_SECRET: "redacted" PUBLISH_API_KEYS: "redacted" SECRET_KEY: "redacted" TERRAFORM_PRESIGNED_URL_SECRET: "redacted" UPLOAD_API_KEYS: "redacted" DATABASE_URL: ""

I even started allowing unidentified downloads at this point, but it does not help in any way. It's placed behind reverse proxy provided by nginx-ingress in k8, which terminates TLS.

[MatthewJohn]

It seems that terrareg do not register these requests from terragrunt. It's archive from terrareg.

If the first request, for the /download endpoint isn't reaching terrareg (and showing in the logs), then perhaps it sounds more of a terragrunt issue.

[maciob]

That's possible. What is the version of terragrunt and terraform you are using?

[maciob]

Okay, I tested it out from another computer. It is working. It seems that terragrunt instalation on macOS is broken and it does not allow downloads. Linux installation on WSL is working perfectly. Anyway thank you for your help @MatthewJohn. It seems that I'm done with my helm chart. I can prepare open-source version for you, with description on env variables, etc. EDIT: nevermind, it was cached. Still not working.

[maciob]

I've searched nginx-ingress logs and I found this:

[21/Jul/2024:10:41:18 +0000] "GET /.well-known/terraform.json HTTP/2.0" 200 249 "-" "Go-http-client/2.0" 65 0.017 [terrareg-terrareg-443] [] :5000 249 0.016 200 e6b9879829fc2ea4f5db6b626f726d99 [21/Jul/2024:10:41:18 +0000] "GET /v1/modules/__///1.0.0/download HTTP/2.0" 401 236 "-" "Go-http-client/2.0" 102 0.003 [terrareg-terrareg-443] [] :5000 236 0.002 401 662dfb5708f970cb0576d91cc6355b11

Seems like I'm getting 401 even after authenticating with terraform login command. Started searching on how I can authenticate to registry using terragrunt and it seems like that feature is not yet implemented: https://github.com/gruntwork-io/terragrunt/issues/1771

So basicly for this to work out I need to set ALLOW_UNAUTHENTICATED_ACCESS to True, which is not ideal.

[MatthewJohn]

I've searched nginx-ingress logs and I found this:

[21/Jul/2024:10:41:18 +0000] "GET /.well-known/terraform.json HTTP/2.0" 200 249 "-" "Go-http-client/2.0" 65 0.017 [terrareg-terrareg-443] [] :5000 249 0.016 200 e6b9879829fc2ea4f5db6b626f726d99 [21/Jul/2024:10:41:18 +0000] "GET /v1/modules/__///1.0.0/download HTTP/2.0" 401 236 "-" "Go-http-client/2.0" 102 0.003 [terrareg-terrareg-443] [] :5000 236 0.002 401 662dfb5708f970cb0576d91cc6355b11

Seems like I'm getting 401 even after authenticating with terraform login command. Started searching on how I can authenticate to registry using terragrunt and it seems like that feature is not yet implemented: gruntwork-io/terragrunt#1771

So basicly for this to work out I need to set ALLOW_UNAUTHENTICATED_ACCESS to True, which is not ideal.

Ah, I see, that's a shame :/ Yes, I wasn't using authentication locally

[MatthewJohn]

I've searched nginx-ingress logs and I found this:

[21/Jul/2024:10:41:18 +0000] "GET /.well-known/terraform.json HTTP/2.0" 200 249 "-" "Go-http-client/2.0" 65 0.017 [terrareg-terrareg-443] [] :5000 249 0.016 200 e6b9879829fc2ea4f5db6b626f726d99 [21/Jul/2024:10:41:18 +0000] "GET /v1/modules/__///1.0.0/download HTTP/2.0" 401 236 "-" "Go-http-client/2.0" 102 0.003 [terrareg-terrareg-443] [] :5000 236 0.002 401 662dfb5708f970cb0576d91cc6355b11

Seems like I'm getting 401 even after authenticating with terraform login command. Started searching on how I can authenticate to registry using terragrunt and it seems like that feature is not yet implemented: gruntwork-io/terragrunt#1771

So basicly for this to work out I need to set ALLOW_UNAUTHENTICATED_ACCESS to True, which is not ideal.

I'm not sure if it's suitable for your use-case at all, but, if you did use git-based modules (as opposed to uploading them), then relaxing the security around the registry would be less risky, as the source code for the modules would still be protected by the Git server. Otherwise, I'm not entirely sure of what else could be done to improve this situation with Terragrunt