search

MicrosoftDocs / azure-docs

Open source documentation of Microsoft Azure
https://docs.microsoft.com/azure
Creative Commons Attribution 4.0 International
10.28k stars 21.47k forks source link

Azure DevOps Build Pipeline can't get secrets from Key Vault when secured with vnet and firewall #38914

Closed aspnet4you closed 4 years ago

aspnet4you commented 5 years ago

I would like to use secrets stored in key vault from DevOps Build Pipeline task and I would like to follow security best practice and defense in depth. As security best practice, I want key vault to be accessible from selected virtual networks, selected azure services and from trusted internet ip's. Of course, I would use a service principal and appropriate permissions (list/get).

Unfortunately, Azure DevOps is not one of the trusted service. So, my alternative is to white-list the DevOps IPs. I found out my DevOps is in US East 2 region and I downloaded Azure Datacenter IPs (filtered with US East2). There are about 285 IP's in US East2. Key Vault firewall has a limit on how many firewall rules you can add and it's 127! So, I am out of luck!

At the moment, I can get secrets from key vault at build pipeline only if I allow all networks! Yea, I still have to authenticate to get the secrets but I lose on defense in depth. I really need to lockdown the key vault to trusted networks but I can't. Why? I can't add more than 127 firewall rules (to cover the region) and DevOps is not one of the trusted azure services!

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

mslot commented 3 years ago

There is currently a workaround for this issue.

You can use a self-hosted agent and add the following IPv4 ranges to the allow-list of the key vault firewall. Details here: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops

(The Azure DevOps team is aware of this issue and work for a permanent fix is currently in progress)

Is the work around to self host and add these IPs to the vault: https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops#im-running-a-firewall-and-my-code-is-in-azure-repos-what-urls-does-the-agent-need-to-communicate-with

swarleysez commented 2 years ago

I just ran into this issue. Yes, this is an old issue, but it still exists and, since DevOps is not considered a trusted Microsoft Service, a workaround is still the only option I could find that avoided me having to ditch my private endpoint on the keyvault.

Unfortunately, the above answers didn't exactly work for my situation or I didn't read them all thoroughly, so hopefully the following will work for anyone else who comes across this.

My Setup:

What worked:

Use the link below to get the DevOps subnet specific to your region.

Reference: https://docs.microsoft.com/en-us/azure/devops/organizations/security/allow-list-ip-url?view=azure-devops&tabs=IP-V4

davidkarlsen commented 2 years ago

@aspnet4you , Thank you for sharing the query. Whitelisting the IPs is not an efficient way, but its definitely a workaround. We are working with the Product team to get Azure Dev Ops as a Trusted service for Azure Key Vault and that would be a complete fix in all terms. Do allow us sometime and I will share my next updates with you.

Could github actions be added as a trusted service too?

hiteshtulsani commented 2 years ago

It seems this is still an issue. Can this be reopened or can someone please point me in the right direction?

jlichwa commented 2 years ago

@hiteshtulsani the above workarounds are the solution the problem. The DevOps platform does not meet requirements for trusted services. Feature request here: https://developercommunity.visualstudio.com/t/azure-devops-build-pipeline-cant-get-secrets-from-1/1152209

rambaliga commented 1 year ago

I was facing the exact same issue but we cannot have a workaround to whitelist the IPs. My Current setup looks like below:

Since my self hosted agent is on the same network I am able to fetch the secrets from the keyvault and assign their values to variables. Post which I am able to use it across all the jobs in the pipeline.

jeffpatton1971 commented 11 months ago

@hiteshtulsani the above workarounds are the solution the problem. The DevOps platform does not meet requirements for trusted services. Feature request here: https://developercommunity.visualstudio.com/t/azure-devops-build-pipeline-cant-get-secrets-from-1/1152209

Hey @jlichwa I have a key vault in azure with network access control setup, I have added these ip's, from the documentation listed earlier.

@('13.107.6.0/24','13.107.9.0/24','13.107.42.0/24','13.107.43.0/24','20.41.6.0/23')

The expected behaviour would be to setup a variable group within azure devops using my service connection which has get/list on secrets to connect to that key vault and grab the secrets I need.

Actual behaviour is an error message letting me know I should add the spn that already has get/list to the key vault with get/list. There were a few other comments in this closed and long winded thread, but this is happening right now, and the'workaround' doesn't appear to work.

additionally I run self-hosted agents, the subnet for those agents is in fact allowed in the network access control, so I really don't see how this is an agent issue.

jlichwa commented 11 months ago

This old issue and many things could change. I understand that with public access/disabled firewall everything works and it breaks when firewall is enabled only. Are you using self hosted Linux Agents like in article https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/v2-linux?view=azure-devops#im-running-a-firewall-and-my-code-is-in-azure-repos-what-urls-does-the-agent-need-to-communicate-with

For regular devops the list of IPs is different, I would recommend to find IPs which are used for the Azure DevOps product used, the actual IP will be visible in Key Vault error logs -https://learn.microsoft.com/en-us/azure/key-vault/general/howto-logging?tabs=azure-cli

jeffpatton1971 commented 11 months ago

Hi @jlichwa thanks for the response, I'm actually getting this error message when I attempt to configure my Variable Group before any agents are actually involved. I do have self-hosted linux agents, but not sure how that relates to this?

jlichwa commented 11 months ago

In general I will recommend to create Stack Overflow question. It has nothing to do with documentation for Key Vault and it would be question to DevOps. What I can only recommend is to enable Key Vault logs and see exactly what IPs and SP are used for setting up variable groups, I don't see those above are right IPs which rather apply to Agents accessing AKV, the list of IPs DevOps use is much longer and change periodically. This 4 years old thread for Key Vault documentation (not DevOps), so not right place, nor audience which can help here.

jeffpatton1971 commented 11 months ago

In general I will recommend to create Stack Overflow question. It has nothing to do with documentation for Key Vault and it would be question to DevOps. What I can only recommend is to enable Key Vault logs and see exactly what IPs and SP are used for setting up variable groups, I don't see those above are right IPs which rather apply to Agents accessing AKV, the list of IPs DevOps use is much longer and change periodically. This 4 years old thread for Key Vault documentation (not DevOps), so not right place, nor audience which can help here.

That's fair, sadly my search there sent me here! But I will create a new post and see what happens, as well as enable logging, thanks!

jeffpatton1971 commented 11 months ago

@jlichwa for future reference https://stackoverflow.com/questions/77558938/how-often-do-the-public-facing-ip-addresses-of-azure-devops-change