search

dotnet / AspNetCore.Docs

Documentation for ASP.NET Core
https://docs.microsoft.com/aspnet/core
Creative Commons Attribution 4.0 International
12.62k stars 25.3k forks source link

ProtectKeysWithAzureKeyVault deserves more explanation #16422

Open Pyrobolser opened 4 years ago

Pyrobolser commented 4 years ago

Hello,

Going from development on localhost where everything is simple to Linux App Services, suddenly nothing is working anymore because of this whole "data protection" issue.

I am trying to follow the documentation and understand I need a Blob Storage Account and a Azure Key Vault, I setup all of those, try to generate the <blobUriWithSasToken> but apparently it needs a reference to a "blob". I go from issues to issues until I get a Root element is missing.
Now I have an empty key.xml as a blob and I don't understand what the application actually wants.
Is there an example somewhere that we can follow, the documentation is a little bit light on this side when you don't know all this.

Document Details

⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

akempe commented 4 years ago

Couldn't agree more. The documentation is woefully inadequate in explaining:

For those looking, I finally got it to work after sifting through the source code for AzureBlobXmlRepository ... but have a nagging suspicion I've missed the easy route to success here... (?)

I created a template XML file containing:

<?xml version="1.0" encoding="utf-8"?>
<keys></keys>

The name of the root element doesn't seem to be important (hopefully this is correct).

On first run, it seems to have populated the file successfully with the keys. Permissions seemed to have to be created at the Storage Account level with app service identity getting Blob Data Contributor access (not sure if this can be more restrictive?)

The app service identity was granted key wrap/unwrap access to the KeyVault.

HTH someone.

Azure team: can someone confirm this is the correct approach? If it's not, what is the right way? Either way, please can the docs make this more clear, as I'm sure many people are bumping into this continually.

Rick-Anderson commented 4 years ago

Comment hub feedback:

akempe commented 4 years ago

@Rick-Anderson side question, why can't we just store the keys directly in keyvault? why the layer of indirection here?

ghost commented 4 years ago

Good day, from what i understand,

services.AddDataProtection() .PersistKeysToAzureBlobStorage(new Uri("<blobUriWithSasToken>")) .ProtectKeysWithAzureKeyVault("<keyIdentifier>", "<clientId>", "<clientSecret>");

PersistKeysToAzureBlobStorage saves the identity cookie encryption and decryption keys to azure blob storage.

ProtectKeysWithAzureKeyVault encrypts the blob containing the keys that are created by PersistKeysToAzureBlobStorage.

But what does clientId and clientSecret mean?

mkArtakMSFT commented 4 years ago

Thanks for bringing up this issue. @Pilchie can you please assign this to an engineer who is well familiar with the Data Protection. Let's get the docs updated to address the gap.

Pilchie commented 4 years ago

@haok @blowdart - either of you want to take a look?

blowdart commented 4 years ago

It's getting replaced soon with a new library soon, so it's probably not worth the investment, however @Petermarcu then needs to assign someone to own and update the docs going forward.

ghost commented 4 years ago

Looking forward to new library.

DaleyKD commented 4 years ago

@akempe - Thank you for that. I'm trying to use the new Azure.Extensions.AspNetCore.DataProtection.Keys SDK, and I'm not convinced it requires the Blob to exist, but I'm doing it anyway.

However, I think I may be getting a race condition on startup of my ASP.NET Core app. I know this is a docs issue, and I'm not here to derail. But this definitely needs some TLC to ensure we get it right.

Putting the stack here as a just in case:

info: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[58]
      Creating key {bd0a2ab1-9338-477e-98b9-6a725797c851} with creation date 2020-07-10 13:46:43Z, activation date 2020-07-10 13:46:42Z, and expiration date 2020-10-08 13:46:42Z.
fail: Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider[48]
      An error occurred while reading the key ring.
System.Xml.XmlException: Root element is missing.
   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
   at System.Xml.Linq.XDocument.Load(XmlReader reader)
   at Azure.Extensions.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.CreateDocumentFromBlob(Byte[] blob)
   at Azure.Extensions.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.StoreElementAsync(XElement element)
   at Azure.Extensions.AspNetCore.DataProtection.Blobs.AzureBlobXmlRepository.StoreElement(XElement element, String friendlyName)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.IInternalXmlKeyManager.CreateNewKey(Guid keyId, DateTimeOffset creationDate, DateTimeOffset activationDate, DateTimeOffset expirationDate)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager.CreateNewKey(DateTimeOffset activationDate, DateTimeOffset expirationDate)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.CreateCacheableKeyRingCore(DateTimeOffset now, IKey keyJustAdded)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.Microsoft.AspNetCore.DataProtection.KeyManagement.Internal.ICacheableKeyRingProvider.GetCacheableKeyRing(DateTimeOffset now)
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingProvider.GetCurrentKeyRingCore(DateTime utcNow, Boolean forceRefresh)
info: Microsoft.AspNetCore.Authentication.Cookies.CookieAuthenticationHandler[7]
      Identity.Application was not authenticated. Failure message: Unprotect ticket failed
akempe commented 4 years ago

@DaleyKD no problem! Are you saying you're still getting this error?

System.Xml.XmlException: Root element is missing.

It looks identical to the one I had originally and lead me to initialising the blob with:

<?xml version="1.0" encoding="utf-8"?>
<keys></keys>

If you get it to work with the new SDK, please let us know as I'm sure people are going to run into this while MS resolve their docs situation. For what it's worth, this has been stable in prod for a few months now.

DaleyKD commented 4 years ago

@akempe : Let me clarify, as I wasn't as clear as I had hoped earlier.

I suspect it's because it's trying to write two or more keys (one or more dealing with the Identity cookie and one or more dealing with the anti-forgery for POST) and therefore, having a race condition

Other things I have tried that have no effect:

And finally, even when I run the app a second time, and it has a valid file in there from the previous run, and I do NOT delete it, I still get the error. Everything points back to a race condition.

nmg196 commented 3 years ago

I'm also really stuck with this. The docs surrounding this are very unclear as they don't give any examples.

The documentation implies it will create the blob on the first run, but how can it do that if the docs also show you need the full URI to the actual file including a SAS token? That's not physicially possible from what I can see - so is the URL supposed to be for a blob container and not the blob xml file itself?

Looking at the comments it seems like the whole "automatic creation" of the blob doesn't work at all. It doesn't even say what should be in the file if you want to create it manually. The only workaround which worked for me was to initially save it to the local file system, then use the generated local file to seed the Azure blob. Ideally, the API should realise an empty file has been supplied and replace it with one of the correct format to reduce "getting started friction".

As @akempe mentioned, it's also really odd that the article doesn't mention anything about how to set up permissions for the SAS token and Key Vault, given the apparent importance of security when storing keys.

arqe commented 3 years ago

@nmg196 The very same here also. This documentation doesn't make sense and is is unfollowable.

Many of security related docs are frustratingly bad overall. More so that in this particular area you definitely want to know what exactly you are doing and not to trial random to see the magic happening.

haldiggs commented 3 years ago

I am probably the most ignorant person here as to what is supposed to be happening. I am trying to use PersistKeysToAzureBlobStorage and getting nothing but frustrated.

I am using Azure to generate the shared connection strings. I am using the "Blob service SAS URL" and assigning to a string variable named blobSasUrl.

I have created a container and I seem to be successful at creating sub containers in code. I am running this in a .Net 5.0 Core WebApi application.

services.AddDataProtection() .SetApplicationName(Assembly.GetEntryAssembly().GetName().Name) .PersistKeysToAzureBlobStorage(new Uri(blobSasUrl)) .ProtectKeysWithCertificate(clientCertificate);

All I keep getting is this error that doesn't really help me figure out why it does not work. Since I am not the one who generates the Url I have to depend on Azure.

Azure.RequestFailedException: Value for one of the query parameters specified in the request URI is invalid. RequestId:bd9cedff-d01e-00ef-4a26-2b644e000000 Time:2021-04-06T20:52:23.7976519Z Status: 400 (Value for one of the query parameters specified in the request URI is invalid.) ErrorCode: InvalidQueryParameterValue Headers: Server: Microsoft-HTTPAPI/2.0 x-ms-request-id: bd9cedff-d01e-00ef-4a26-2b644e000000 x-ms-client-request-id: eed59e39-e857-4ab4-8f2a-5e146a5cc7f6 x-ms-error-code: InvalidQueryParameterValue Date: Tue, 06 Apr 2021 20:52:23 GMT Content-Length: 351 Content-Type: application/xml

I am hoping this is related to this issue you guys are having with the documentation.

Mike-E-angelo commented 2 years ago

Any update here would be appreciated. Looking at the documentation it states:

If the app uses the older Azure packages (Microsoft.AspNetCore.DataProtection.AzureStorage and Microsoft.AspNetCore.DataProtection.AzureKeyVault), we recommend removing these references and upgrading to the Azure.Extensions.AspNetCore.DataProtection.Blobs and [Azure.Extensions.AspNetCore.DataProtection.Keys] https://www.nuget.org/packages/Azure.Extensions.AspNetCore.DataProtection.Keys)

But looking at these packages on NuGet, the packages recommended to remove are actually newer and updated more frequently than the recommended packages to install in their place.

For example, Azure.Extensions.AspNetCore.DataProtection.Blobs ("newer") is dated 5/14/2021 w/ 6 published versions, while Microsoft.AspNetCore.DataProtection.AzureStorage ("older") was last released 24 days ago with 8 published versions since the last publishing of the "new" package (5/14/2021).

Very confusing.

haldiggs commented 2 years ago

omatic creation" of the blob doesn't work at all. It doesn't even say what should be in the file if you want to create it manually. The only workaround w

good luck getting an answer. there must be documentation somewhere else because there hasn't been any answers to other questions for years

datwelk commented 2 years ago

@Rick-Anderson could we get a reply from Microsoft on this issue please?

Rick-Anderson commented 2 years ago

It's getting replaced soon with a new library soon, so it's probably not worth the investment, however @Petermarcu then needs to assign someone to own and update the docs going forward.

@Petermarcu can you assign someone to own and update the docs going forward.

bradygaster commented 2 years ago

@datwelk @haldiggs @Pyrobolser please see the issue I linked: https://github.com/dotnet/AspNetCore.Docs/issues/26093

If anyone has a moment to try that and let us know if it resolves your issue, we'd be appreciative. Thanks!

Or, if possible, try a fresh deployment using the code from the doc. If it works only after you manually create a container, please confirm.

haldiggs commented 2 years ago

dang, @bradygaster I would love it too, but that code was scrapped a long time back. I will have to rely on others at this time. I will have a new project coming up that could use this. Happy to see it coming alive though

bradygaster commented 2 years ago

I do have an update. PersistKeysToAzureBlobStorage does NOT create your container. It will create the blob if it isn't there.

I'm still researching ProtectKeysWithAzureKeyVault and I'm specifically interested in the error @DaleyKD posted above - Any recollection on how you were able to make that error pop?

jwisener commented 2 years ago

Why is the documentation so fragmented and hard to follow? I too had to dig into the xml code comments of the various libraries to try and understand why the example provided would not work. The only thing I can think is that the people creating the documentation don't understand how it's supposed to work. Or it's a reflection of the communication from the team to the tech writers, aka "you only get just fragments of understanding." Then you just puke it onto some random page. It "mentions permissions", but I am not sure if that is the only permission(s) needed in key-vault. (tbh, I don't trust the documentation).

ghost commented 2 years ago

Just deployed a new app using the code from the documentation and cannot get this working. Is there any guidelines on how to set this up from scratch?

Do you need to create a empty xml-file manually in the blob before running the code the first time?

We use Azure with Managed Identity to handle permissionsso we use that to connect to the keyvault. What do we need to do to get this to work?

//Setup connection details using managed identity
 var blobStorageUri = new Uri($"https://{dpConf.AccountName}.blob.core.windows.net/{dpConf.ContainerName}/{applicationName}");

var tokenCredential = new DefaultAzureCredential(options);

      services.AddDataProtection()
        .SetApplicationName(applicationName)
        .PersistKeysToAzureBlobStorage(blobStorageUri, tokenCredential)
        .ProtectKeysWithAzureKeyVault(new Uri("keyName"), tokenCredential);
haldiggs commented 2 years ago

sorry, we decided to use the file storage method so we could move forward. I'm sure it will be better when we revisit the issue

On Thu, Jun 9, 2022 at 1:40 AM Brady Gaster @.***> wrote:

@datwelk https://github.com/datwelk @haldiggs https://github.com/haldiggs @Pyrobolser https://github.com/Pyrobolser please see the issue I linked: #16422 https://github.com/dotnet/AspNetCore.Docs/issues/16422

If anyone has a moment to try that and let us know if it resolves your issue, we'd be appreciative. Thanks!

— Reply to this email directly, view it on GitHub https://github.com/dotnet/AspNetCore.Docs/issues/16422#issuecomment-1150731212, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM5KUQHZYYQOPGDMIAJN7LVOGGUHANCNFSM4KDLCN3A . You are receiving this because you were mentioned.Message ID: @.***>

-- Hal D. - "Non sibi sed patriae" (Not for self, but for country) Half of communication is listening, and you can't listen with your mouth.

Rick-Anderson commented 1 year ago

@Petermarcu can you assign someone to own and update the docs going forward.

Rick-Anderson commented 1 year ago

Email sent to @Petermarcu

abratv commented 1 year ago
bendrick92 commented 1 year ago

Has there been any update on the documentation for configuring .NET Core Data Protection w/ Azure blob storage/key vault?

I've been trying to follow this and been getting very frustrated/lost.

EDIT: I took @haldiggs approach and gave up for now. Went with the PersistKeysToFileSystem approach until they can get the Key Vault documentation sorted out. Wasted days trying to troubleshoot all the different variables.

rwb196884 commented 1 year ago

This is still completely fucked.

AspNetCoreDataProtectionOptions aspNetCoreDataProtectionOptions = new AspNetCoreDataProtectionOptions();
builder.Configuration.GetSection("AspNetCoreDataProtection").Bind(aspNetCoreDataProtectionOptions);
aspNetCoreDataProtectionOptions.BlobUriWithSasToken = "https://whatever.blob.core.windows.net/whatever-data-protection?sp=racwdl&st=2023-08-08T09:23:11Z&se=2123-08-08T17:23:11Z&spr=https&sv=2022-11-02&sr=c&sig=bwixpImR4CVWERyi%2BDtmqGxeCDVX9P0nrzNLaEaWOTQ%3D";
aspNetCoreDataProtectionOptions.KeyIdentifier = "https://whatever-key-vault.vault.azure.net/keys/whatever-data-protection/4a1114f79ad94cceaf6761796668c712";

string keyVaultName = builder.Configuration.GetValue<string>("KeyVaultName");
string trimmedContentRootPath = builder.Environment.ContentRootPath.TrimEnd(Path.DirectorySeparatorChar);
if (!string.IsNullOrEmpty(keyVaultName))
{
    builder.Services.AddDataProtection()
        .SetApplicationName(trimmedContentRootPath)
        .PersistKeysToAzureBlobStorage(new Uri(aspNetCoreDataProtectionOptions.BlobUriWithSasToken))
        .ProtectKeysWithAzureKeyVault(new Uri(aspNetCoreDataProtectionOptions.KeyIdentifier), new DefaultAzureCredential())
        ;
}

but

RequestFailedException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
jamie-tillman commented 3 months ago

This is still completely fucked.

* Went to storage resrouce in azure and creted a new container. Used its menu to 'Generate SAS.

* Went to key vault service and generated a new key, clicked into it and copied its 'key ientifier' URL.
  Is that what you're supposed to do? Seems consistent with what @abratv says.
AspNetCoreDataProtectionOptions aspNetCoreDataProtectionOptions = new AspNetCoreDataProtectionOptions();
builder.Configuration.GetSection("AspNetCoreDataProtection").Bind(aspNetCoreDataProtectionOptions);
aspNetCoreDataProtectionOptions.BlobUriWithSasToken = "https://whatever.blob.core.windows.net/whatever-data-protection?sp=racwdl&st=2023-08-08T09:23:11Z&se=2123-08-08T17:23:11Z&spr=https&sv=2022-11-02&sr=c&sig=bwixpImR4CVWERyi%2BDtmqGxeCDVX9P0nrzNLaEaWOTQ%3D";
aspNetCoreDataProtectionOptions.KeyIdentifier = "https://whatever-key-vault.vault.azure.net/keys/whatever-data-protection/4a1114f79ad94cceaf6761796668c712";

string keyVaultName = builder.Configuration.GetValue<string>("KeyVaultName");
string trimmedContentRootPath = builder.Environment.ContentRootPath.TrimEnd(Path.DirectorySeparatorChar);
if (!string.IsNullOrEmpty(keyVaultName))
{
    builder.Services.AddDataProtection()
        .SetApplicationName(trimmedContentRootPath)
        .PersistKeysToAzureBlobStorage(new Uri(aspNetCoreDataProtectionOptions.BlobUriWithSasToken))
        .ProtectKeysWithAzureKeyVault(new Uri(aspNetCoreDataProtectionOptions.KeyIdentifier), new DefaultAzureCredential())
        ;
}

but

RequestFailedException: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.

Same issue here in July 2024. This thread having been started more than 4 years ago, it's clear no one cares about making this function usable.