docker: data protection failures bring down entire app

[natemcmaster]

Running music store in docker brings down the entire app.

Repro: dotnet publish -c release

FROM aspnetcorenightly-exp.azurecr.io/aspnetcore:1.1.0-preview1-final

ENV ASPNETCORE_URLS http://+:5000

COPY bin/release/netcoreapp1.1/publish /app
WORKDIR /app

EXPOSE 5000

ENTRYPOINT ["dotnet", "MusicStore.dll"]

On startup, the app issues this warning:

warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {9cf9edb8-4ad2-4da2-ad40-78213196b05e} may be persisted to storage in unencrypted form.

After that, subsequent requests that rely on data protection (such as session's cookie protection and antiforgery) will fail. App becomes unusable.

Examples:

``` fail: Microsoft.AspNetCore.Antiforgery.Internal.DefaultAntiforgery[7] An exception was thrown while deserializing the token. System.InvalidOperationException: The antiforgery token could not be decrypted. ---> System.Security.Cryptography.CryptographicException: The key {85956e8e-0fe1-4731-b27a-109a2ce742b3} was not found in the key ring. at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status) ```

cc @kichalla @pakrym

[danroth27]

@Eilon @muratg @glennc Did this used to work in 1.0?

[pakrym]

DataProtection stores keys in HOME directory on linux by default so when container restarts keys are lost.

[natemcmaster]

It looks like somehow the key dataprotection is looking for changes after the app starts:

# tree -al
.
|-- .aspnet
|   `-- DataProtection-Keys
|       `-- key-4e1ca6ff-281e-41b1-87e2-da086278bb15.xml

Error messages are looking for a key named "915ad8b0-11d7-4e1f-957b-e0f53fff1800".

Hosting environment: Production
Content root path: /app
Now listening on: http://+:5000
Application started. Press Ctrl+C to shut down.
warn: Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager[35]
      No XML encryptor configured. Key {4e1ca6ff-281e-41b1-87e2-da086278bb15} may be persisted to storage in unencrypted form.
warn: Microsoft.AspNetCore.Session.SessionMiddleware[7]
      Error unprotecting the session cookie.
System.Security.Cryptography.CryptographicException: The key {915ad8b0-11d7-4e1f-957b-e0f53fff1800} was not found in the key ring.
   at Microsoft.AspNetCore.DataProtection.KeyManagement.KeyRingBasedDataProtector.UnprotectCore(Byte[] protectedData, Boolean allowOperationsOnRevokedKeys, UnprotectStatus& status)

[natemcmaster]

Figured it out. Something in the session and antiforgery cookie is tied directly to the key. When the server restarts and the key refreshes, antiforgery and session get a stale cookie and try to use a key that doesn't exist. When the key doesn't exist, the app crashes.

[natemcmaster]

cc @Tratcher

[glennc]

I am fine with the fact that in docker the keys are going to go away. But I thought that someone making a request to a site with a cookie encrypted with an invalid or unfindable key should not break the app. It just means they are not authenticated or whatever the thing is that is using the cookie. I.e. they get the same experience as the very first time they ever visited the site.

@kichalla @blowdart Is that not true of Antiforgery?

[blowdart]

No. If the antiforgery token is invalid it will throw an exception

[glennc]

So it should be up to music store to handle that? What should an app do when that exception is thrown?

[Tratcher]

Here: https://github.com/aspnet/Antiforgery/blob/ad90db343c437283d809dab12f89b51116a25151/src/Microsoft.AspNetCore.Antiforgery/Internal/DefaultAntiforgeryTokenSerializer.cs#L55 Note that does not match the dataprotection behavior we use in auth or session cookies. In those cases any failures are logged and we assume you have no identity/session.

[Tratcher]

@natemcmaster what's the full stack trace?

[natemcmaster]

I don't have it handy. Would have to re-reproduce.

[xuchuanquan]

NU1001 The dependency Microsoft.Extensions.Configuration.CommandLine >= 1.2.0-* could not be resolved. How do you solve this problem ？

[natemcmaster]

We've mitigated this some in 2.0.0 by adding warnings in the logger if running Docker and the key store is a folder that would be deleted when the container is removed.

https://github.com/aspnet/DataProtection/pull/244