Retrieve secrets from Key Vault is failing

EricHunsberger commented 1 year ago

Hi Team, I've configured Azure Load Test a test to pull secrets from Key Vault, however it is failing with: Unable to find the given secret Correlation ID: GUID

I performed the following steps to enable this capability:

added secrets to KV and added parameters to to the test -How to parameterize load tests
Updated my JMX to have User Variable using __GetSecret to pull the secrets Use secrets in Apache JMeter
A system MSI is in place for the test and it has been granted Get permissions for Secrets - Grant access to Azure Key Vault

I review the AzureActivity and AzureLoadTestingOperation logs but didn't find any additional information.

My issue looks like this one, but it was resolved sometime back - #25

Is this a new occurrence? Are there additional logs I can enable/review?
Eric

AB#1659304

Nagarjuna-Vipparthi commented 1 year ago

@EricHunsberger Can you please share the test run ID? This will help us debug the issue. Thank you.

EricHunsberger commented 1 year ago

@Nagarjuna-Vipparthi this is the correlation ID from the error - 51aad157-c189-459d-8d6a-e3c076bdd012

cathalmchale commented 1 year ago

Having this exact issue. Any info that can be shared on resolving the above appreciated.

cathalmchale commented 1 year ago

So in my case I changed what I called the user defined variable secret. I originally used the same name as the KV secret entry, which included dashes. I simplified this name to "functionKey" and that seemed to resolve this error.

However I then hit another problem - The provided Key Vault reference identity does not have required permission(s) to access the Key Vault. I'm wondering is this problem due to cross-subscription access? We use a single KeyVault instance in our PROD subscription - can get this to work fine with AKS and APIM services, but maybe Azure Load Test Preview doesn't like the fact that the Azure Load Test resource is in a different Subscription to KeyVault - AAD is obviously shared across all our subscriptions.

Nagarjuna-Vipparthi commented 1 year ago

@Nagarjuna-Vipparthi this is the correlation ID from the error - 51aad157-c189-459d-8d6a-e3c076bdd012

@EricHunsberger Can you please check and confirm if you used the same managed identity for the test run and Key vault access? We see an 'Unauthorized Access to keyvault' error message suggesting that the provided Managed Identity does not have permission to access the Key Vault / Secret.

Nagarjuna-Vipparthi commented 1 year ago

@cathalmchale Can you please confirm if in the test creation, you have selected the managed identity that has access to the Key Vault? You can find this option under the 'Parameters' tab

cathalmchale commented 1 year ago

Hi @Nagarjuna-Vipparthi I've tried with both System assigned and User-assigned and ensured that these added to the Access Policy in Key Vault. However, I think the issue is probably the fact that our Key Vault isn't public. It's setup like:

So it's likely that the other Azure Services can access via policies and assigned identities etc. because they are also within a trusted vnet. I'll test this theory by setting up a new temporary Key Vault with public access for all networks and associate the same managed identities - should be able to observe one Key Vault succeed and the other fail in the Load Test if it's the root.

EricHunsberger commented 1 year ago

I have the Test (Catalogpage) configured with a System-assigned identity and the "catalogpage" application has Get permissions on the KeyVault.

My KeyVault Networking is set up very much like Cathal's.

Allow public access from specific virtual networks and IP Addresses
our internal virtual subnets are configured
Allow trusted Microsoft services to bypass this firewall is enabled

cathalmchale commented 1 year ago

Hi @EricHunsberger , I spent a bit of time root causing this one and it's definitely to do with the non-public KeyVault, rather than there being anything wrong with the Identity assignment or KeyVault access policy. So I went ahead and logged it as a more specific bug #117 Be good if you could comment on that bug as well as sounds like we've the same root cause, manifesting itself with the same error message.

I'm hoping that this will be something that Microsoft will look to resolve when the feature comes out of Preview. Afterall, we're only trying to follow some best practices in our KeyVault setup!

For now I've chosen to setup a public "dev" KeyVault that I'll store a few secrets that are important and required by the load tests. Not ideal and complicates "secret recycling" which we're trying to get better at. Also the public KeyVault is going to show up in our security scans as an issue, but we'll try justify it's limited scope and move forward for now until another resolution is possible.

EricHunsberger commented 1 year ago

Hi Cathal, Thanks for the update. I'll add my comments to the other issue.

Unfortunately I can't switch my access policy. I'll end up on our Cyber naughty list, lol. Eric

From: Cathal McHale @.> Sent: Monday, November 7, 2022 3:26 PM To: microsoft/azure-load-testing @.> Cc: Hunsberger, Eric @.>; Mention @.> Subject: [EXT] Re: [microsoft/azure-load-testing] Retrieve secrets from Key Vault is failing (Issue #112)

Hi @EricHunsbergerhttps://github.com/EricHunsberger , I spent a bit of time root causing this one and it's definitely to do with the non-public KeyVault, rather than there being anything wrong with the Identity assignment or KeyVault access policy. So I went ahead and logged it as a more specific bug #117https://github.com/microsoft/azure-load-testing/issues/117 Be good if you could comment on that bug as well as sounds like we've the same root cause, manifesting itself with the same error message.

I'm hoping that this will be something that Microsoft will look to resolve when the feature comes out of Preview. Afterall, we're only trying to follow some best practices in our KeyVault setup!

For now I've chosen to setup a public "dev" KeyVault that I'll store a few secrets that are important and required by the load tests. Not ideal and complicates "secret recycling" which we're trying to get better at. Also the public KeyVault is going to show up in our security scans as an issue, but we'll try justify it's limited scope and move forward for now until another resolution is possible.

— Reply to this email directly, view it on GitHubhttps://github.com/microsoft/azure-load-testing/issues/112#issuecomment-1306146522, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AQ5RMWOFY5LPFNYYYRSES33WHFQXNANCNFSM6AAAAAARRHU5ZM. You are receiving this because you were mentioned.Message ID: @.***>

This message (including any attachments) contains confidential information intended for a specific individual and purpose, and is protected by law. If you are not the intended recipient, you should delete this message and any disclosure, copying, or distribution of this message, or the taking of any action based on it, by you is strictly prohibited.

Deloitte refers to a Deloitte member firm, one of its related entities, or Deloitte Touche Tohmatsu Limited ("DTTL"). Each Deloitte member firm is a separate legal entity and a member of DTTL. DTTL does not provide services to clients. Please see www.deloitte.com/about to learn more.

v.E.1

Sachid26 commented 1 year ago

Hi @EricHunsberger .. we have added this to our backlog for now and will report back once it is implemented.

microsoft / azure-load-testing

Retrieve secrets from Key Vault is failing #112