Feature: support for private module registries for Terraform module datasource

[iniinikoski]

What would you like Renovate to be able to do?

I'd like that RenovateBot would be able to access also private module registries of Terraform Cloud and Terraform Enterprise. This requires we'd need to somehow feed an API key to RenovateBot...

Did you already have any implementation ideas?

-

Are there any workarounds or alternative ideas you've tried to avoid needing this feature?

Unfortunately no... We could pull the modules from private GitHub repositories and tags - but I'm not sure if that's supported either... Probably not - as same issue would persist...

Is this a feature you'd be interested in implementing yourself?

I wish I could (no skills and also no permit to do so...)

[rarkins]

Can you provide an example to file that contains private modules? Also pointers to all relevant terraform documentation would help.

[iniinikoski]

Sure.

Here's how you pull modules from your own registry:

module "our_custom_module" {
  version            = "0.1.1"
  source             = "my-terraform.enterprise.corp.com/ORG/module/provider"
  variable_name      = true
}

See here for information: https://www.terraform.io/docs/cloud/registry/using.html

And, if you compare this to Terraform Cloud module registry, it goes like this:

module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "2.44.0"
  # insert the 14 required variables here
}

Example here: https://registry.terraform.io/modules/terraform-aws-modules/vpc/aws/2.44.0

So, to me looks like, comparing these to, there's a hostname in front of the private module registry as a difference to Terraform Cloud.

Probably someone from Hashicorp can 100% can confirm this...

[rarkins]

Can you locate documentation for how authentication is done for private registries?

[rarkins]

Doing a quick code inspection:

We should extract private modules regardless of host: https://github.com/renovatebot/renovate/blob/4665dbff6fe96c4aa2665d95022284c60be3f320/lib/manager/terraform/modules.ts#L64-L69

We don't use Terraform's service discovery protocol but instead assume a fixed path for modules on the host: https://github.com/renovatebot/renovate/blob/4665dbff6fe96c4aa2665d95022284c60be3f320/lib/datasource/terraform-module/index.ts#L68 (does this match with yours though?)

Can you clarify "how far" Renovate gets for you currently? e.g. does it query the right URL but get a 401 or 403 because of no authentication?

[rarkins]

Seems they use simple Bearer tokens, which should be possible to configure using regular hostRules.

[iniinikoski]

Hi,

Here's the DEBUG-level output from RenovateBot:

DEBUG: Datasource 404
{
  "datasource": "terraform-module",
  "lookupName": "tf-enterprise-hostname.company.com/ORG/ecr/aws",
  "url": "https://tf-enterprise-hostname.company.com/v1/modules/ORG/ecr/aws"
}
DEBUG: Failed to look up dependency tf-enterprise-hostname.company.com/ORG/ecr/aws (tf-enterprise-hostname.company.com/ORG/ecr/aws)(packageFile="aws/eu-central-1/production/main.tf", 

And yes, they use just Bearer tokens.

This is how to get Terraform working with the private registry: https://www.terraform.io/docs/commands/cli-config.html#credentials

Can we really do this with the hostRules...?

[rarkins]

If the URL is correct, then you should be able to configure the following (assuming it's self-hosted and you don't need to encrypt your token):

{
  "hostRules": [{
    "hostName": "tf-enterprise-hostname-company.com",
    "token": "abc123"
  }]
}

[iniinikoski]

Ok thanks.

We'd like to encrypt - and use the managed one - should these instructions be valid then, using the NPM private registry as an example: https://docs.renovatebot.com/private-modules/#add-an-encrypted-npm-token-to-renovate-config ?

Or, should / must we use a self-hosted one...? We can of course but, we also try to avoid this when possible...

[rarkins]

It's fine to use the hosted app. I just assumed if you have a self-hosted Terraform server you have a good chance of running self-hosted Renovate too!

In that case the instructions are similar, yours should look like this:

{
  "hostRules": [{
    "hostName": "tf-enterprise-hostname-company.com",
    "encrypted": {
      "token": "asfsfasdfasdfasdfadsfasdfads=="
    }
  }]
}

[iniinikoski]

Ok, great, thanks! :)

I tried this - and I can see that the run is setting the token correctly I think:

DEBUG: migrated config
{
  "config": {
    "extends": [
      "config:base"
    ],
    "hostRules": [
      {
        "hostName": "tf-enterprise-hostname.company.com",
        "encrypted": {
          "token": "***********"
        }
      }
    ]
  }
}
DEBUG: Found encrypted config
{
  "config": {
    "token": "***********"
  }

But, we still get 404 when it's doing the run:

          {
            "currentValue": "0.1.1",
            "depType": "terraform",
            "depName": "tf-enterprise-hostname.company.com/ORG/ecr/aws",
            "depNameShort": "tf-enterprise-hostname.company.com/ORG/ecr/aws",
            "datasource": "terraform-module",
            "depIndex": 26,
            "updates": [],
            "warnings": [
              {
                "depName": "tf-enterprise-hostname.company.com/ORG/ecr/aws",
                "message": "Failed to look up dependency tf-enterprise-hostname.company.com/ORG/ecr/aws"
              }
            ]
          },

But, what puzzles us, that if I try to access the module path (https://tf-enterprise-hostname.company.com/v1/modules/ORG/ecr/aws) - I also get an error from TFE...

I'm trying to understand is something different between TFC and TFE here which breaks this - this what I currently suspect...

[rarkins]

Let's first rule out if the URL being tried is 100% correct. You could confirm by using curl and supplying the token in the header.

[iniinikoski]

Running this curl:

    curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request GET \
https://tf-enterprise-hostname.company.com/v1/modules/ORG/ecr/aws

with the correct bearer token (which I can confirm with another curl-query doing an API call to TFE) - I get the 404.

I'll open a case with Hashicorp support on this one as well - as I think they can assist us on this.

I tried to compare the behaviours between TFC and TFE - but I cannot get wiser here...

[iniinikoski]

Hi,

Hashicorp replied :)

Here's a working curl statement:

curl \
  --header "Authorization: Bearer $TOKEN" \
  --header "Content-Type: application/vnd.api+json" \
  --request GET \
https://tf-enterprise-hostname.company.com/api/registry/v1/modules/ORG/ecr/aws

[rarkins]

Can we discover the full path using their "service discovery" protocol? I saw it mentioned in terraform docs

[secustor]

@iniinikoski Can you confirm that https://tf-enterprise-hostname.company.com/.well-known/terraform.json returns

{
  "modules.v1": "/api/registry/v1/modules"
}

Looking into the service discovery documentation the json should contain the, at the moment, hard coded part of the URL. I have never used TFE, can you enroll there your own providers too? I couldn't find any documentation regarding this topic.

[iniinikoski]

‘’’ {"modules.v1":"/api/registry/v1/modules/","state.v2":"/api/v2/","tfe.v2":"/api/v2/","tfe.v2.1":"/api/v2/","tfe.v2.2":"/api/v2/","versions.v1":"https://checkpoint-api.hashicorp.com/v1/versions/"} ‘’’

Yes - and here’s the full response.

[iniinikoski]

@secustor I would assume the provider support is coming to TFE (no access to ours right now - can check tomorrow) - as that’s what Hashicorp built into Terraform Cloud Module Registry just recently...

[secustor]

@rarkins You can assign me this issue.

@iniinikoski see you maybe next week in the HQ ;)

[iniinikoski]

@iniinikoski see you maybe next week in the HQ ;)

Yes ;) Maybe we could grab a coffee if u have a moment :)

[renovate-release]

:tada: This issue has been resolved in version 21.29.0 :tada:

The release is available on:

• GitHub release
• 21.29.0

Your semantic-release bot :package::rocket:

[iniinikoski]

I tried this as Action - but I only get this when running on DEBUG-level:

Failed to look up dependency tf-enterprise-hostname.company.com/ORG/ecr/aws

This time, no even HTTP error codes...

I'm going to test this against Terraform Cloud's private registry as well - to get more information. And with some public test code...

[iniinikoski]

I did tests with Terraform Cloud + it's private registry also. There I was able to dig up an error:

DEBUG: Datasource unauthorized (repository=iniinikoski/tf-dependencytest-repo)

I did the tests in public repos:

https://github.com/iniinikoski/tf-dependencytest-repo https://github.com/iniinikoski/tf-test-module-repo (the dependent module)

The complete Action-output can be seen here: https://github.com/iniinikoski/tf-dependencytest-repo/pull/2/checks?check_run_id=870099920#step:3:348

The token I encrypted with https://app.renovatebot.com/encrypt works when I do this:

 curl --header "Authorization: Bearer $TOKEN" --header "Content-Type: application/vnd.api+json" --request GET https://app.terraform.io/api/registry/v1/modules/iniinikoski/module-repo/test

{"id":"iniinikoski/module-repo/test/0.0.1","owner":"","namespace":"iniinikoski","name":"module-repo","version":"0.0.1","provider":"test","description":"","source":"https://github.com/iniinikoski/tf-test-module-repo","tag":"","published_at":"2020-07-14T15:49:43.114212Z","downloads":1,"verified":false,"root":{"path":"","name":"module-repo","readme":"# tf-test-module-repo","empty":false,"inputs":[],"outputs":[],"dependencies":[],"provider_dependencies":[{"name":"random","namespace":"","source":"","version":""}],"resources":[{"name":"my_pet","type":"random_pet"}]},"submodules":[],"examples":[],"providers":["test"],"versions":["0.0.1"]}

Am I missing something obvious here?

[iniinikoski]

Quick update: we do get the same error with TFE as well. Interesting...

[iniinikoski]

I enabled TRACE logging, and see this: 2020-07-14T17:12:49.5109088Z TRACE: Applying Bearer authentication for host undefined (repository=iniinikoski/tf-dependencytest-repo)

Weird...

[iniinikoski]

2020-07-14T17:44:24.6510417Z  WARN: Host error (repository=iniinikoski/tf-dependencytest-repo)
2020-07-14T17:44:24.6510657Z        "hostType": "terraform-module",
2020-07-14T17:44:24.6510918Z        "lookupName": "app.terraform.io/iniinikoski/module-repo/test",
2020-07-14T17:44:24.6511037Z        "err": {
2020-07-14T17:44:24.6511141Z          "name": "HTTPError",
2020-07-14T17:44:24.6511249Z          "host": "app.terraform.io",
2020-07-14T17:44:24.6511364Z          "hostname": "app.terraform.io",
2020-07-14T17:44:24.6511472Z          "method": "GET",
2020-07-14T17:44:24.6511748Z          "path": "/api/registry/v1/modules/iniinikoski/module-repo/test",
2020-07-14T17:44:24.6511876Z          "protocol": "https:",
2020-07-14T17:44:24.6512193Z          "url": "https://app.terraform.io/api/registry/v1/modules/iniinikoski/module-repo/test",
2020-07-14T17:44:24.6512334Z          "gotOptions": {
2020-07-14T17:44:24.6512598Z            "path": "/api/registry/v1/modules/iniinikoski/module-repo/test",
2020-07-14T17:44:24.6512725Z            "protocol": "https:",
2020-07-14T17:44:24.6512822Z            "slashes": true,
2020-07-14T17:44:24.6512930Z            "auth": null,
2020-07-14T17:44:24.6513040Z            "host": "app.terraform.io",
2020-07-14T17:44:24.6513140Z            "port": null,
2020-07-14T17:44:24.6513247Z            "hostname": "app.terraform.io",
2020-07-14T17:44:24.6513353Z            "hash": null,
2020-07-14T17:44:24.6513457Z            "search": null,
2020-07-14T17:44:24.6513647Z            "query": null,
2020-07-14T17:44:24.6513949Z            "pathname": "/api/registry/v1/modules/iniinikoski/module-repo/test",
2020-07-14T17:44:24.6514278Z            "href": "https://app.terraform.io/api/registry/v1/modules/iniinikoski/module-repo/test",
2020-07-14T17:44:24.6514417Z            "retry": {
2020-07-14T17:44:24.6514523Z              "methods": {},
2020-07-14T17:44:24.6514629Z              "statusCodes": {},
2020-07-14T17:44:24.6514737Z              "errorCodes": {},
2020-07-14T17:44:24.6514833Z              "maxRetryAfter": 60000
2020-07-14T17:44:24.6514938Z            },
2020-07-14T17:44:24.6515045Z            "headers": {
2020-07-14T17:44:24.6515330Z              "user-agent": "https://github.com/renovatebot/renovate",
2020-07-14T17:44:24.6515467Z              "accept": "application/json",
2020-07-14T17:44:24.6515712Z              "accept-encoding": "gzip, deflate"
2020-07-14T17:44:24.6515818Z            },
2020-07-14T17:44:24.6515916Z            "hooks": {
2020-07-14T17:44:24.6516026Z              "beforeRequest": [],
2020-07-14T17:44:24.6516141Z              "beforeRedirect": [null],
2020-07-14T17:44:24.6516248Z              "beforeRetry": [],
2020-07-14T17:44:24.6516355Z              "afterResponse": [],
2020-07-14T17:44:24.6516459Z              "beforeError": [],
2020-07-14T17:44:24.6516563Z              "init": []
2020-07-14T17:44:24.6516652Z            },
2020-07-14T17:44:24.6516754Z            "decompress": true,
2020-07-14T17:44:24.6516859Z            "throwHttpErrors": true,
2020-07-14T17:44:24.6516961Z            "followRedirect": true,
2020-07-14T17:44:24.6517064Z            "stream": false,
2020-07-14T17:44:24.6517168Z            "form": false,
2020-07-14T17:44:24.6517273Z            "json": true,
2020-07-14T17:44:24.6517376Z            "cache": false,
2020-07-14T17:44:24.6517480Z            "useElectronNet": false,
2020-07-14T17:44:24.6517584Z            "method": "GET",
2020-07-14T17:44:24.6517834Z            "hostType": "terraform-module",
2020-07-14T17:44:24.6517951Z            "abortOnError": true,
2020-07-14T17:44:24.6518120Z            "gotTimeout": {"request": 60000}
2020-07-14T17:44:24.6518233Z          },
2020-07-14T17:44:24.6518339Z          "statusCode": 401,
2020-07-14T17:44:24.6518445Z          "statusMessage": "Unauthorized",
2020-07-14T17:44:24.6518548Z          "headers": {
2020-07-14T17:44:24.6518655Z            "date": "Tue, 14 Jul 2020 17:44:21 GMT",
2020-07-14T17:44:24.6518941Z            "content-type": "application/vnd.api+json; charset=utf-8",
2020-07-14T17:44:24.6519179Z            "content-length": "52",
2020-07-14T17:44:24.6519289Z            "connection": "close",
2020-07-14T17:44:24.6519519Z            "cache-control": "no-cache",
2020-07-14T17:44:24.6519759Z            "vary": "Accept-Encoding, Origin",
2020-07-14T17:44:24.6519997Z            "x-content-type-options": "nosniff",
2020-07-14T17:44:24.6520233Z            "x-frame-options": "SAMEORIGIN",
2020-07-14T17:44:24.6520462Z            "x-ratelimit-limit": "30",
2020-07-14T17:44:24.6520700Z            "x-ratelimit-remaining": "29",
2020-07-14T17:44:24.6521032Z            "x-ratelimit-reset": "0.847",
2020-07-14T17:44:24.6521302Z            "x-request-id": "91774c77-7f21-b842-6d9f-3454daa4835e",
2020-07-14T17:44:24.6521542Z            "x-xss-protection": "1; mode=block"
2020-07-14T17:44:24.6521756Z          },
2020-07-14T17:44:24.6521879Z          "body": {"errors": [{"status": "401", "title": "unauthorized"}]},
2020-07-14T17:44:24.6522001Z          "message": "Response code 401 (Unauthorized)",
2020-07-14T17:44:24.6522448Z          "stack": "HTTPError: Response code 401 (Unauthorized)\n    at EventEmitter.<anonymous> (/usr/src/app/node_modules/got/source/as-promise.js:74:19)\n    at processTicksAndRejections (internal/process/task_queues.js:97:5)"
2020-07-14T17:44:24.6522596Z        }
2020-07-14T17:44:24.6522874Z  INFO: External host error causing abort - skipping (repository=iniinikoski/tf-dependencytest-repo)

I changed it to use "domainName" - and see this...

For me this looks like it's not trying to use the Bearer token at all...

[viceice]

@iniinikoski can you please share your config. Looks like you have a wrong config. You can mask your private data

[iniinikoski]

@iniinikoski can you please share your config. Looks like you have a wrong config. You can mask your private data

Sure - the whole config should be in here: https://github.com/iniinikoski/tf-dependencytest-repo (also with the open PR).

[viceice]

Ok, thanks. Now I see your problem. The renovate action uses the self hosted CLI, which can't decrypt your token because it doesn't know the nessesary private key. You should use GitHub secrets for this.

Please open a new issue in our config-help repo for further help.

[iniinikoski]

Ah, a very good point, thanks @viceice ...

I'll do this if needed yes. Thanks again!