Support End to End Encryption

[jordanisaacs]

Is your feature request related to a problem? Please describe.

I am interested in having end to end encryption in my app that I am building. However this can't be done with how ory kratos is set up currently. Password are validated on the server before hashing. Also, it isn't possible to secure the encryption key on the server side.

Describe the solution you'd like

Support the option to turn on end to end encryption. The flows would look something like this:

Registration:

• The client is responsible for deriving an authentication key and protectedkey from the credentials
• On the server ory kratos would skip password validation. It would further hash the authentication key to derive both the verifier and the serverkey. The serverkey would encrypt the provided protectedkey to return a serverprotectedkey which would be stored with the user.

Login:

• Standard login procedure
• If successfully verified, then decrypt the serverprotectedkey to get the protectedkey
• Included in a successful response would the protectedkey

Password Change:

• Client would provide an authentication key and the new protectedkey
• Deletes the existing serverprotectedkey and updates it with the new protectedkey as during registration.

Account Recovery:

• Deletes the serverprotectedkey associated with the account. Encrypted data would become inaccessible as intended.

Other implementations/resources:

• https://bitwarden.com/help/article/what-encryption-is-used/ (client side information)
• https://www.lastpass.com/enterprise/security (under architecture)
• https://dchest.com/2020/05/25/improving-storage-of-password-encrypted-secrets-in-end-to-end-encrypted-apps/ (storing key on server side, The photos are excerpted from this article)
• https://francoisbest.com/posts/2019/how-to-store-e2ee-keys-in-the-browser (securing client key in browser)

[aeneasr]

Hey this is a cool idea! We also need a couple of other things though:

1. Parsing of the identity traits server-side to understand the identity model and do things like:
   1. sending a recovery email
   2. sending a verification email
2. be able to update profiles and again validate and interpret/parse the traits
3. be able to see if a user is already signed up with the e.g. recovery email

There's more to this list - how could we support these things?

[jordanisaacs]

I imagine that all that should be added is an option in the identity schema to toggle encryption for traits such as:

"trait": {
    "type": "string"
    "ory.sh/kratos": {
        "encryption": {
            "encrypted": true
            "validator": xxxx
        }
    }
}

If the trait is encrypted then skip validation on the server side but provide the validation that should be done client side (similar to passwords). Therefore the trait JSON model stays the same.

Thus all the standard trait management would stay the same. There would be some additional logic that would need to parse the identity schema to determine if the validation needs to be skipped before updating the server. While I am not going to comment on details of cryptographic algorithms as I am not a security expert, the standard is some form of AES. So to ensure against attacks the encrypted traits could be validated as X bits.

In regards to verifiers/identifiers they would not be allowed to be encrypted. So it would work as standard kratos. Password managers don't encrypt the emails so I take that as its standard in e2e encrypted apps that the email/identifiers won't be. They explicitly say that the emails are used for account management (verification, recovery, etc.) which is the scope of kratos.

Two whitepapers I skimmed:

• https://www.dashlane.com/download/Dashlane_SecurityWhitePaper_March2021.pdf
• https://bitwarden.com/images/resources/security-white-paper-download.pdf

[jordanisaacs]

Here is a diagram I drew of a more detailed registration flow to visualize how validation/encryption of traits would work and identifiers.

[aeneasr]

That’s pretty smart! I did not think about partial encryption of user data but that makes a ton of sense. We could also add a „vault“ to identities which has BLOB storage for encrypted data that would allow you to store any information in there (AES, assymetric, ...) and could reduce complexity on our end because the data is kept independently. However, it would make sign up processes harder for the implementor. What do you think?

[aeneasr]

And if I understand correctly the key derivation function uses the end-user’s password (what if we don’t have a password and only social sign in?) to derive a key which is used to encrypt the data - hence only the user (if we don’t leak the password in e.g. a logfile by accident and we can trust the operator - eg reverse proxy- not to listen to it) can decrypt it?

[jordanisaacs]

I like the BLOB/vault idea as it generalizes the flows to store any protected data you want. So I am imagining to support end-to-end encryption there would be three configurations needed. This is a summary of what has been discussed so far:

1. Option to turn off password validation (it would be done client side).
   • This option would enable the use of keys as passwords.
2. Separate client/server side validation for identity traits.
   • Allows for client side validation of encrypted data. while still allowing for server side validation (such as the trait is present)
   • A side effect of separating client and server validation is there can be a BLOB encrypted trait.

     "trait": {
     "type": "string"
     "ory.sh/kratos": {
     "client-validation": {
         "validator": xxxx
     }
     }
     }

3. Option to enable the vault. A protected data BLOB that is encrypted by a hash of the password/key. Follows the flows as in the first post of this issue. Would not be included in a /whoami request.
   • Would store things such as the client side encrypted encryption key.
   • It is necessary that the vault is hashed because of the problem noted in this blog post: https://dchest.com/2020/05/25/improving-storage-of-password-encrypted-secrets-in-end-to-end-encrypted-apps/.

Combining these three options would enable end-to-end encryption. Furthermore, would enable complete flexibility in how one stores traits/keys/etc. There would need to be docs walking through how to use these three features in conjunction for things like asymmetric encryption/data sharing, e2ee, etc.

There is a fourth thing that would need to be implemented, would be a whole new flow of its own, vault updates. This flow would be for doing key rotations. Vault update flow

1. Sends the password/key to the server along with the updated data BLOB.
2. Hashes the password/key as during login to ensure it is correct.
3. Protects the data BLOB by encrypting using the password/key hash
4. Updates the old protected BLOB with the new protected BLOB
5. Sends back a HTTP 200 if successful or HTTP 400 if unsuccessful

[jordanisaacs]

This is how Bitwarden works which is what I am treating as industry standard: The hashed password is not being used as a key but is used to encrypt the key. The actual encryption key is a randomly generated key created during registration. This is encrypted with a KDF before being sent to the server. This allows for password changes without having to re-encrypt all of the data. In regards to leaking the password. The password would be hashed on the client side. The result would used to encrypt the key. The result would then be hashed some more before being sent to the server. This ensures only the client can decrypt the generated key. Notably all of this is done on the client side, not ory kratos. Therefore some good docs would need to be in place to show how to use these features correctly.

I am not sure exactly how social sign in works. But assuming that social sign in produces a key during registration that doesn't change, that key could be hashed and then used to encrypt the generated encryption key.

[aeneasr]

Sorry for the long wait time. I like the vault idea also - my question however would be if we actually want to trust Ory Kratos with encrypting the data? If we do, then it's not true e2e encryption as the system receives the key to decrypt the store.

Bitwarden, for example, decrypts the vault on the client side (afaik). So the key never leaves your device. Wouldn't we want the same here?

[jordanisaacs]

No worries. We agree on kratos not doing any encryption of data. I'll walk through more clearly of the vault encryption as it seems I haven't done that well enough.

Below is the flow of what bitwarden does on both the client and server side (page 8 of [bitwarden white paper]):(https://bitwarden.com/images/resources/security-white-paper-download.pdf):

Now an example kratos vault only holding an AES key:

Client: The AES key is produced by the client (CSPRNG) on registration. All data encrypted using this encryption key. A KDF with the password as payload and identifier as salt is used to encrypt the AES key to get the client protected key. This is why we need separation of client/server side validation. Traits that are told to be encrypted on the client side still need ways to be validated.

Server: The reason further encryption is needed of the client protected key is we don't know the client's computing power. Therefore probably won't be using something like Argon2. If we do what bitwarden does and just store the client protected key in the database and the database was compromised the client protected key would be the weakest link. They would only need to attack the client protected key to find what password decrypts it. That is what is outlined in this blog post that I have linked to before.

The further encryption would be done by taking a second KDF of the authentication key using Argon2. This second KDF would then be used as the key to encrypt the client protected key to finally produce the vault. The salt used for the second KDF would be stored next to the vault.

I believe this layout makes the e2e encryption more clear. More generally, the vault essentially holds all the client protected keys. It is essentially for data that is produced in a way that is vulnerable to attacks. Maybe there can be an option to turn on/off this extra level of server side encryption if you don't see the need for it.

[jordanisaacs]

With the release of web hooks it seems that the vault could be implemented as a third party api server without needing to be a part of kratos itself which might be better. From my understanding the web hook can forward everything needed to create a vault (form data, headers, etc). I am imagining just using the after hook on the necessary flows. That way the implementation of a vault is up to the end user.

One question about web hooks is can they alter the http response to the user? For the login flow we need to somehow get the vault data to the user.

The option to disable password validation and configure client side trait validation would still need to be implemented though. What are your thoughts @aeneasr , should I create separate issues for them?

[aeneasr]

Hey @jordanisaacs sorry for the slow reply on my end - this is a difficult issue to wrap one's head around :) That concepts sounds really intriguing! I think it makes sense to have the vault capabilities built into Ory Kratos - and not rely on webhooks - because it would be great to have e2e encryption be a part of what this suite offers :)

Let's talk a bit about implementation, would you be open to contribute the plan laid out in https://github.com/ory/kratos/issues/1246#issuecomment-830855137 ?

[jordanisaacs]

No worries. It definitely is really complex!

I would be willing to contribute the plan as laid out. I am new to Go so I would work on the steps in increasing complexity: disable password validation, then client side validation, then actually working on the vault integration.

Before I start though I want to bring up this newish protocol that I found this past week that solves many of the problems I have been encountering when trying to spec out e2ee. Its called OPAQUE. It is an asymmetric PAKE (password-authenticated key exchange) with server salt. Here is a high level overview of it: https://blog.cloudflare.com/opaque-oblivious-passwords/. It is explained really well there. For an implementation of it this library, while in Rust, has really nice high-level docs overviewing the general steps of login/registration (https://docs.rs/opaque-ke/0.5.0/opaque_ke/index.html) that follows the most recent OPAQUE specs. Another nice blog post https://blog.cryptographyengineering.com/2018/10/19/lets-talk-about-pake/.

Why OPAQUE rather than an implementation like Bitwarden? A key part of Kratos is configurability and standards. As e2ee relies on the client to perform some critical security steps, using OPAQUE would provide a standard for the client side that Kratos would not need to deal with. Furthermore it allows for more configurability as it doesn't rely on using a username/password for the hash salt. Now a user can choose to login with any identifier rather than a specific identifier chosen as the salt. While OPAQUE is still in drafting stage I believe OPAQUE would serve Kratos better than a custom implementation.

How would OPAQUE fit in to Kratos? As I've stated before I am not experienced with social sign in but for passwords it would replace the password credential. A login/registration would follow the OPAQUE processes. As the authentication procedure itself holds a secret key, the vault process would be extremely simplified. The vault would just be a BLOB of data encrypted using the OPAQUE generated secret key during registration (all on the client). The client would re-encrypt the vault and update the server when the password changes (OPAQUE would generate our new secret). The vault can be provided with a simple GET request. And if the client needs to get the key to decrypt the vault again, they would just go through an authentication procedure where OPAQUE would provide the secret key. It reduces the complexity of the vault significantly as it rolls up the secret key, authentication, and client side passwords into one standard.

One thought to really make it integrate well with Kratos is accounts can have multiple credentials. However, OIDC accounts will not be secure as the I am assuming the key is provided by a third party. Therefore, the vault option is automatically enabled for any user as long as they have a password credential. To access the vault data, they would just need to re-authenticate using their password credential. So a user could quickly create/log into their account with OIDC but if they want to access their vault then they would create/authenticate with their password credentials. A sort of elevated process.

Here is the maintained Go library: https://github.com/bytemare/opaque/ (It is the one linked from the specification repo)

[aeneasr]

So I read through the different sources (including OPAQUE) and I think that we need to - before addressing which KDF to use - what use cases we want to support with e2e encryption as we’ll get lost in new methodologies and protocols as opposed to solving the concrete need.

So - imo - what we need is something along the BitWarden model where the client encrypts data before sending it to the server. To the server, the data is completely opaque and without use (so e.g. not an email address but my private diary) and the server can’t aid the client in en/decrypting the data.

Given that, the question is wether the server can implement anything that would help the client here. Should the server help in generating the KDF key? Or not? Shoild it help in encryption? Can it help?

I think we need to establish these basics before exploring different protocols! What do you think?

[jordanisaacs]

Yes, I agree. I will lay out the requirements + better explain why I introduced the idea of using OPAQUE in terms of these requirements. This is my first big idea/contribution to an open source project so I am still getting the hang of making sure I am clarifying everything in writing.

The vault idea that you brought up earlier has moved us beyond E2E encryption to the more extensible solution. The vault is at its core a secrets store. Every kratos user will have a vault (a blob of data) that only they can access. Therefore, any secrets (such as a users' actual data encryption keys) can be safely stored in the vault and synchronized across all clients. With that in mind here are the three concrete requirements I have produced, plus one requirement that needs further discussion.

Concrete Requirements

1. There is an encrypted blob of data sitting on the server that can only be fully decrypted with information that the user holds. aka the information used to encrypt is never sent to the server.

To fulfill this requirement the encryption needs to be done on the client. The encryption key and its producing information can never leave the client. Otherwise, it is not secure anymore.

2. The key to encrypt/decrypt this blob of data is hooked into login/registration. All information that the user needs to decrypt this blob of data is inputted during login/registration. As a result of this, the blob of data can be re-encrypted and updated during a password change. But will be lost during a password reset.

To fulfill this requirement, whether it is a pure KDF (Bitwarden) or a KDF encrypted envelope (OPAQUE), to enable the vault a KDF must be done on the client. Somehow, we need to derive/obtain the encryption key from the inputted data, then further obfuscate the inputted data before sending it to the server for authentication (to keep the encryption key/producing information a secret).

3. The data generating the encryption key of the blob of data cannot be tied to the unique identifier (email/username/etc). This is so any identifier type can be used during login/registration.

This requirement is where the server finally has a role. The server will generate a salt for the KDF during registration, and provide the correct salt during login. This would either be inputted client side into the pure KDF (Bitwarden) or the KDF that encrypts the envelope (OPAQUE).

The three requirements I listed means there is lots of interactions of security and cryptography. It is an order of magnitude more complex then storing a simple hashed password on a server. That is why I recommended OPAQUE. It was built specifically to enable server-side salting (requirement 3) of a client side KDF (requirement 2). A side effect is that the envelope contains a randomly generated secret key (better than a key derived from the password) that can be used to encrypt the vault (requirement 1). Utilizing a formally proven standard seems like the only way to feasibly do this. OR we can remove requirement 3, tie it to an identifier, and perform a Bitwarden style hash process (laid out earlier in this issue). But then we lose key configurability that kratos provides in terms of identifiers and are offloading the complexity to the users of a kratos server. They are required to implement all the client side KDF and logic themselves.

Requirement (Needs Discussion) This will be abstracted over whatever we use for requirements 1-3:

4. Should the vault be connected to a credential or a user? A one-to-one mapping of credentials to vaults is easy. Connecting multiple credentials (a user) to a single vault seems like it would difficult and would complicate the login and registration flows. Would somehow involve having multiple unique keys that can decrypt the same data. Is that even possible?

[aeneasr]

I see, thank you for the explanation. I think one important aspect would also be to check what methodologies are well supported in the ecosystem right now. For example, are there KDF implementations available for popular programming languages that follow the Bitwarden / OPAQUE / ... model?

So one the one hand I think we have to keep in mind the use case while also ensuring that the implementation is achievable by average implementers of the Ory stack.

[jordanisaacs]

Having a vault like this would require client-side computations as listed earlier. Therefore, a site using it would require Javascript and/or WASM which is not for every use case. Also, there is more complexity for the average implementer. Therefore, the vault should be a toggleable option for those who have a need for it. It should be stated clearly that some type of migration would require to switch to/from the vault model.

Bitwarden Model

• What is it? The model that has been discussed extensively throughout this issue. While there is no standard it has been implemented successfully by password managers such as Bitwarden and Lastpass. Here is a recap: it uses a standard KDF on the client side to derive an encryption key and an authentication key. Encryption key encrypts the vault, authentication key gets sent to the server and is treated like a password. Simple but fails requirement 3 (server side salt).
• End user implementation: As the Bitwarden model is not standardized the end-user needs to implement all the KDF logic themselves. Kratos can provide and maintain a Javascript and/or WASM reference library that implements the KDF logic correctly. High quality documentation can be provided for implementing it in their own language. It would be semi-complex but not too difficult for an implementer to build their own version in their language of choice. However, this would require Ory to create and maintain a basic standard which I am not sure if Ory really wants to do? Also fails requirement 3 (server side salt) so not sure how it would work with different identifiers.

OPAQUE Model

• What is it? An aPAKE with server salt that fulfills all three requirements listed above. It has a formal security proof and there is currently an in-progress Internet Draft. It is significantly more complex than the Bitwarden model for understanding. Here are some good documents if you want to learn: Let's talk about PAKE and OPAQUE: The Best Passwords Never Leave your Device. One issue with OPAQUE is it still is in the drafting stages but has been around long enough to be production ready. (See this exchange from the Rust implementation).
• End user implementation: This implementation is standardized and provides functions for both the client and the server. However, OPAQUE is newer, so it is only currently implemented in three languages (SAGE (reference lang), Go, and Rust). The average end-user would not be able to implement OPAQUE in their language of choice. The Rust one can be compiled to WASM. I have no experience with the Go implementation currently. I believe this would not require any standards maintaining by Ory and no client library maintaining. Just a simple example showing how to use OPAQUE to generate the correct API messages. The OPAQUE process requires a short exchange rather than a single message and response. The WASM library readme linked above shows an example how it works in Javascript. If the OPAQUE model is chosen some thought would need to go into how to best integrate the OPAQUE exchange process into existing flows in a user-friendly way.

[ycgambo]

seems like this is a lot of work. however, I just worries about plain text password on POST request, which causes middle-man attack.

[vilkinsons]

@aeneasr Appreciate you've moved this to unplanned (sorry to hear). Were there any further internal/follow-up discussions re: this you could share thoughts or outputs from? This is something we'll be interested in longer-term and I'd like to track. Thanks!

[mitar]

How does this design enforce password complexity policy? If the server-side receives an already hashed/encrypted password, then it cannot verify that password is of a required length, for example. Client-side is generally trusted, but somebody could manually do an API request/modify request to the server-side. It is true that in this case they would be making their own account less secure, but people are lazy and might want to do that.

[github-actions[bot]]

Hello contributors!

I am marking this issue as stale as it has not received any engagement from the community or maintainers for a year. That does not imply that the issue has no merit! If you feel strongly about this issue

• open a PR referencing and resolving the issue;
• leave a comment on it and discuss ideas on how you could contribute towards resolving it;
• leave a comment and describe in detail why this issue is critical for your use case;
• open a new issue with updated details and a plan for resolving the issue.

Throughout its lifetime, Ory has received over 10.000 issues and PRs. To sustain that growth, we need to prioritize and focus on issues that are important to the community. A good indication of importance, and thus priority, is activity on a topic.

Unfortunately, burnout has become a topic of concern amongst open-source projects.

It can lead to severe personal and health issues as well as opening catastrophic attack vectors.

The motivation for this automation is to help prioritize issues in the backlog and not ignore, reject, or belittle anyone.

If this issue was marked as stale erroneously you can exempt it by adding the backlog label, assigning someone, or setting a milestone for it.

Thank you for your understanding and to anyone who participated in the conversation! And as written above, please do participate in the conversation if this topic is important to you!

Thank you 🙏✌️