Open Joerg-L opened 2 years ago
I am hitting the same issue. Anyone with ideas?
Same issue.
Get the same issue, just found a workaround with the system access token.
resources:
repositories:
- repository: terraform-modules
type: git
name: terraform-modules
terraform init
, change steps:
- bash: git config --global http.https://<your_org>@dev.azure.com.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"
And then you should be able to run terraform init
Get the same issue, just found a workaround with the system access token.
thanks @b1n9s I will take a look at that workaround.
It looks usefull if there is only one (or a few) repository(s), but we have for each module a single repository, so as of now 27 module repos.
@b1n9s so, I have tested the workarround
I isn't working as we have more then 20 module repos and azure is limiting that to max 20.
@srjennings @VOVELEE @b1n9s so, after many tests I was able to solve the issue now. As I can see, the solution has two parts.
First part of our solution: Script in the pipeline yml
- script: |
git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com"
displayName: 'set extra header'
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
that is what also @b1n9s has mentioned. But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total.
Second part of our solution: Deactivating the setting "Protect access to repositories in YAML pipelines"
With both parts together, everything seams to work now.
Can confirm, @Joerg-L 's solution works!
Get the same issue, just found a workaround with the system access token.
* Step 1, config your terraform module [repository resource](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/resources?view=azure-devops&tabs=schema#define-a-repositories-resource) in the pipeline
resources: repositories: - repository: terraform-modules type: git name: terraform-modules
* Step 2: add the following step before your `terraform init`, change _<your_org>_ to your organization name
steps: - bash: git config --global http.https://<your_org>@dev.azure.com.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"
And then you should be able to run
terraform init
Wow, I was hitting my had to the wall with this one, I am running a terragrunt with azure repos as git, using it inside the container job, and I tried a lot of things inlcuding using the ssh key as git_ssh_command, tried to install using the installsshkey, with this I switched the sources to use https rather than ssh, and with this little trick it finally worked. I have similar jobs with with only terraform, no terragrunt, everything works on that part but with terragrunt a new issue. Thank you on that part Now I am stuck again with a reference from a module which uses git::ssh... wow....
@srjennings @VOVELEE @b1n9s so, after many tests I was able to solve the issue now. As I can see, the solution has two parts.
First part of our solution: Script in the pipeline yml
- script: | git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com" displayName: 'set extra header' env: SYSTEM_ACCESSTOKEN: $(System.AccessToken)
that is what also @b1n9s has mentioned. But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total.
Second part of our solution: Deactivating the setting "Protect access to repositories in YAML pipelines"
With both parts together, everything seams to work now.
Thanks for that! This works for me when getting repos in the same project but fails when I try to get repos from other projects. Does anyone have any idea how to fix that?
Hey @Isaac-Irvine
This works for me when getting repos in the same project but fails when I try to get repos from other projects. Does anyone have any idea how to fix that?
The problem is that the System.AccessToken is bounded to the project specific build account.
I have just checkt it in Azure DevOps, you can grant permissions in "ProjectB" to the Build Account of "ProjectA". So I could imagine that if you update the permissions, it could work.
@srjennings @VOVELEE @b1n9s so, after many tests I was able to solve the issue now. As I can see, the solution has two parts. First part of our solution: Script in the pipeline yml
- script: | git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com" displayName: 'set extra header' env: SYSTEM_ACCESSTOKEN: $(System.AccessToken)
that is what also @b1n9s has mentioned. But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total. Second part of our solution: Deactivating the setting "Protect access to repositories in YAML pipelines" With both parts together, everything seams to work now.
Thanks for that! This works for me when getting repos in the same project but fails when I try to get repos from other projects. Does anyone have any idea how to fix that?
In case anyone is still having issues, I found that using http.
- pwsh: git config --global --add http.${{variables['System.CollectionUri']}}.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"
cleanup task:
- pwsh: git config --global --unset-all http.${{variables['System.CollectionUri']}}.extraheader
Hello, All can someone please help me with the sourcing of specific modules...
Currently, I find the below solution which works for me locally but I would like to know if there is a better approach someone follows.
module "my-module" {
source = "git::https://dev.azure.com/<org>/<project>/_git/<repo>//<module-folder>?ref=<branch||tag>"
variable = ...
variable = ...
}
++ @Joerg-L Thanks in advance :)
@tyagivasu you could also use multi-repo checkout and specify ref in the pipeline resources.
But then you would need to use a relative path for sourcing in the module, not sure if it's a better approach for you.
@tyagivasu you could also use multi-repo checkout and specify ref in the pipeline resources.
But then you would need to use a relative path for sourcing in the module, not sure if it's a better approach for you.
Hi @b1n9s thanks for your reply. Actually, that is what exactly I have today :) But I am looking to move in the HTTP:// source direction to roll out specific module versions, And would like to hear from people on any challenges, and approaches are taken while going to the https:// kind of sourcing.
@srjennings @VOVELEE @b1n9s so, after many tests I was able to solve the issue now. As I can see, the solution has two parts.
First part of our solution: Script in the pipeline yml
- script: | git config --global url."https://$SYSTEM_ACCESSTOKEN@dev.azure.com".insteadOf "https://<<ADO-ORG>>@dev.azure.com" displayName: 'set extra header' env: SYSTEM_ACCESSTOKEN: $(System.AccessToken)
that is what also @b1n9s has mentioned. But, as mentioned, adding the repos as resources isn't an option for us because of the number of repos and, what is more important, that option is limited to 20 repos in total.
Second part of our solution: Deactivating the setting "Protect access to repositories in YAML pipelines"
With both parts together, everything seams to work now.
Thanks @Joerg-L
I'm facing a similar issue for a github repo. Have not had success following these solutions. Has anyone else faced this?
Yes @bmv0161. I recently ported from ADO to GitHub too. Similar issues
@bmv0161 @gregdskb can you give more details about the setup, is it one of the following?
@bmv0161 @gregdskb can you give more details about the setup, is it one of the following?
- Caller repo and the module repo are both on Github, or
- Caller repo on Github and module repo on ADO, or
- Caller repo on ADO and module repo on Github
For me @b1n9s , it was caller repo in ADO and module repo in ADO. Then migrated my ADO pipeline to GitHub Actions. Caller repo in GitHub and module repo in GitHub. Both had similar issues. Not sure this is a TF issue. Perhaps better discussed on Stackoverflow
For me @b1n9s , it was caller repo in ADO and module repo in ADO. Then migrated my ADO pipeline to GitHub Actions. Caller repo in GitHub and module repo in GitHub. Both had similar issues. Not sure this is a TF issue. Perhaps better discussed on Stackoverflow
@gregdskb For sure this is not a TF issue.
Not sure how your auth is configured in the GitHub Actions, but one thing to note, the GITHUB_TOKEN
secret has limited access to its own repo only if that's what you are using. https://docs.github.com/en/actions/security-guides/automatic-token-authentication
The token's permissions are limited to the repository that contains your workflow.
Thanks @b1n9s , using the GITHUB_TOKEN
is not the answer. If you or @bmv0161 want to start a topic in Stackoverflow I will contribute my solutions. Hope that helps
@bmv0161 @gregdskb can you give more details about the setup, is it one of the following?
- Caller repo and the module repo are both on Github, or
- Caller repo on Github and module repo on ADO, or
- Caller repo on ADO and module repo on Github
For me, both the caller repo and the module repo are in Github. With deployment in Azure pipelines. Attempting to use the System.AccessToken with a Github Apps Service Connection does not seem to authenticate properly.
Here's another solution which is using environment variables instead. You won't have to use "global variables" so there's no risk of issues with multiple builds in parallel (and leaking tokens to other builds).
git remote -v
is also listing the original url after the build has completed as the override is gone. The access token is not included in there anymore. This is also working fine in the local dev environment as the git url in the terraform module config isn't really different.
Azure pipeline:
steps:
- script: |
export GIT_CONFIG_PARAMETERS="'url.https://$SYSTEM_ACCESSTOKEN@dev.azure.com.insteadof=https://<YOUR_ORG>@dev.azure.com'"
terraform init
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
displayName: Initialize Terraform modules
The module is included via this snippet:
module "mymodule" {
source = "git::https://<YOUR_ORG>@dev.azure.com/<YOUR_ORG>/<YOUR_PROJECT>/_git/<YOUR_REPO>//<PATH_IN_REPO>"
}
Hey everyone. I am facing the same issue right now. We are trying to use a dedicated Azure Project for terraform modules. Within that, every module gets a single repo. like @Joerg-L mentioned above we are trying to refrence each modul repo in the terraform configuration and useing the System.AccessToken in Pipeline as shown :
module "NAME" {
source = "git::https://<ORGA>@dev.azure.com/<ORGA>/<PROJECT>/_git/<REPO>?ref=<BRANCH>"
}
unfortunately none of the approaches working. Where are getting the following error message in the azure pipeline:
│ Error: Failed to download module
│
│ on main.tf line 26:
│ 26: module "MODULE" {
│
│ Could not download module "resources" (main.tf:26) source code from
│ "git::[https://@dev.azure.com/<ORGA>/<PROJECT>/_git/.....
│ error downloading
│ 'https://<ORGA>@dev.azure.com/<ORGA>/<PROJECT>/_git/<REPO>?ref=<BRANCH>'
(https://<ORGA>@dev.azure.com/<ORGA>/<PROJECT>/_git/<REPO>?ref=<BRANCH>%27):
│ /usr/bin/git exited with 128: Cloning into
│ '.terraform/modules/resources'...
│ fatal: could not read Password for 'https://<ORGA>@dev.azure.com'/: terminal
│ prompts disabled
hitting the same issue and as mentioned by @karts499 I tried all options suggested in comments here and still getting the same problem :
So I'm not a fan of disabling this project wide setting Protect access to repositories in YAML pipelines. I don't think that is a good idea. Read more about job access tokens here. There is a reason it is enabled by default.
With a test configuration repo and a test terraform modules repo hosted in the same Project I managed to get it working. Couple of things:
First adding the additional repository as a resource and explicitly in a checkout step.
resources:
repositories:
- repository: terraform_modules
type: git
name: terraform_modules
steps:
- checkout: self
- checkout: terraform_modules
When the pipeline runs you will be prompted to authenticate to the extra repository (hint, job access token now has permission to read from that repo 🤘)
Secondly, add the git config. I chose to use an environment variable which will only last per the variable scope (the job in the case of my test pipeline). If you choose to use/set git global config (especially on a self hosted agent) there may be danger ahead if the agent is shared etc.
- script: |
export GIT_CONFIG_PARAMETERS="'url.https://$SYSTEM_ACCESSTOKEN@dev.azure.com.insteadof=https://<YOUR_ORG_NAME_GOES_HERE>@dev.azure.com'"
terraform --version -json
terraform init
terraform plan
env:
SYSTEM_ACCESSTOKEN: $(System.AccessToken)
displayName: Test Init and Plan
workingDirectory: $(Build.SourcesDirectory)/config_test_repo/test1
Really that is it.
Example of a module source:
module "name" {
source = "git::https://<ORGNAME>@dev.azure.com/<ORGNAME>/<PROJECT_NAME>/_git/terraform_modules//modules/<MODULE_NAME>?ref=v2"
...
}
Success. Can use git:https
within terraform module source, from a repo hosted in the same project, all with the system access token.
https://learn.microsoft.com/en-us/azure/devops/pipelines/security/secure-access-to-repos?view=azure-devops&tabs=yaml https://learn.microsoft.com/en-us/azure/devops/pipelines/process/access-tokens?view=azure-devops&tabs=yaml#protect-access-to-repositories-in-yaml-pipelines https://learn.microsoft.com/en-us/azure/devops/pipelines/repos/multi-repo-checkout?view=azure-devops#repository-resource-definition
Hey everyone. After long nights of debugging and try and errors. We finally found a workaround. Just by simply using a bash task instead of a pwsh task:
- bash: | find $(Build.SourcesDirectory)/ -type f -name 'main.tf' -exec sed -i 's~git::https://{ORG_NAME}@dev.azure.com~git::https://$(System.AccessToken)@dev.azure.com~g' {} \;
I should basicly do the same like my try with pwsh. Dont understand what exactly made the difference.
Let me know if you have a better solution :)
Get the same issue, just found a workaround with the system access token.
- Step 1, config your terraform module repository resource in the pipeline
resources: repositories: - repository: terraform-modules type: git name: terraform-modules
- Step 2: add the following step before your
terraform init
, change __ to your organization name steps: - bash: git config --global http.https://<your_org>@dev.azure.com.extraheader "AUTHORIZATION: bearer $(System.AccessToken)"
And then you should be able to run
terraform init
I was able to get it working with these steps + 1 additional step where I had to map the additional repositories that each job consumes.
jobs:
- job:
uses:
repositories:
- terraform-modules
So, we are using Azure Devops to store our Terraform config and all the self created module code. We also want to use a DevOps Pipeline to apply the configuration.
As we are not allowed to use ssh for accessing the repos (traffic has to go trough the https-proxy), so we have to use https for the repository integration so that our source for the module looks like
Locally running
terraform init
works completely fine. When running it in the pipeline we see followingWe have tested many things right now and only with changeing source to
we were able to run
terraform init
but checking in PAT to git sounds not very right.Anyone an idea on that? Regards Joerg