External KMS doesn't work with streaming

[dimapaikin]

I'm trying to encrypt and decrypt files using the TINK streaming mechanism with an external KMS integration. When I perform encryption and decryption in a single run, everything works fine. However, when I encrypt in the first run, save the encrypted file, and try to decrypt it in another run, I encounter a 'no matching key found for the ciphertext in the stream' error.

Expected behavior: The file should be decryptable in another run with the same KMS integration.

Tink version: v2.0.0 Operating System: Windows Go version: 1.21.1

Additional information: The same behavior also occurs when using HashiCorp Vault as a KMS.

package main

import (
    "bytes"
    "fmt"
    "github.com/tink-crypto/tink-go/v2/keyset"
    "github.com/tink-crypto/tink-go/v2/streamingaead"
    "github.com/tink-crypto/tink-go/v2/testing/fakekms"
    "github.com/tink-crypto/tink-go/v2/tink"
    "time"

    "io"
    "log"
    "os"
)

func main() {

    keyUri := "fake-kms://CM2b3_MDElQKSAowdHlwZS5nb29nbGVhcGlzLmNvbS9nb29nbGUuY3J5cHRvLnRpbmsuQWVzR2NtS2V5EhIaEIK75t5L-adlUwVhWvRuWUwYARABGM2b3_MDIAE"

    kmsClient, err := fakekms.NewClient(keyUri)
    if err != nil {
        log.Fatal(err)
    }
    kekAEAD, err := kmsClient.GetAEAD(keyUri)

    if err != nil {
        log.Fatal(err)
    }

    newHandle, err := keyset.NewHandle(streamingaead.AES256GCMHKDF1MBKeyTemplate())
    if err != nil {
        log.Fatal(err)
    }

    keysetAssociatedData := []byte("keyset encryption example")

    buf := new(bytes.Buffer)
    writer := keyset.NewBinaryWriter(buf)
    err = newHandle.WriteWithAssociatedData(writer, kekAEAD, keysetAssociatedData)
    if err != nil {
        log.Fatal(err)
    }
    encryptedKeyset := buf.Bytes()

    reader := keyset.NewBinaryReader(bytes.NewReader(encryptedKeyset))
    handle, err := keyset.ReadWithAssociatedData(reader, kekAEAD, keysetAssociatedData)
    if err != nil {
        log.Fatal(err)
    }

    streamingAEAD, err := streamingaead.New(handle)
    if err != nil {
        log.Fatal(err)
    }

    fileForEncryptionResult := "C:\\temp\\encryptionOutput7.txt"
    fileForEncryption := "C:\\temp\\Application architecture.pptx"
    fileForDecryptionResult := "c:\\temp\\f_result.pptx"

    startEncTime := time.Now()

    EncryptFile(streamingAEAD, fileForEncryption, fileForEncryptionResult, keysetAssociatedData)

    endEncTime := time.Now()

    DecryptFile(streamingAEAD, fileForEncryptionResult, fileForDecryptionResult, keysetAssociatedData)

    endDecTime := time.Now()

    fmt.Printf("***********   \nEncryption took %s to execute\nDecryption took %s to execute\n ", endEncTime.Sub(startEncTime), endDecTime.Sub(endEncTime))

}

func EncryptFile(streamingAEAD tink.StreamingAEAD, inputFileName string, outputFileName string, associatedData []byte) error {
    inputFile, err := os.Open(inputFileName)
    if err != nil {
        return err
    }
    defer inputFile.Close()

    outputFile, err := os.Create(outputFileName)
    if err != nil {
        return err
    }
    defer outputFile.Close()

    w, err := streamingAEAD.NewEncryptingWriter(outputFile, associatedData)
    if err != nil {
        log.Fatal(err)
    }
    if _, err := io.Copy(w, inputFile); err != nil {
        log.Fatal(err)
    }
    if err := w.Close(); err != nil {
        log.Fatal(err)
    }
    if err := outputFile.Close(); err != nil {
        log.Fatal(err)
    }
    if err := inputFile.Close(); err != nil {
        log.Fatal(err)
    }

    return nil
}

func DecryptFile(streamingAEAD tink.StreamingAEAD, ciphertextPath string, decryptedPath string, associatedData []byte) error {

    ciphertextFile, err := os.Open(ciphertextPath)
    if err != nil {
        log.Fatal(err)
    }
    decryptedFile, err := os.Create(decryptedPath)
    if err != nil {
        log.Fatal(err)
    }
    r, err := streamingAEAD.NewDecryptingReader(ciphertextFile, associatedData)
    if err != nil {
        log.Fatal(err)
    }
    if _, err := io.Copy(decryptedFile, r); err != nil {
        log.Fatal(err)
    }
    if err := decryptedFile.Close(); err != nil {
        log.Fatal(err)
    }
    if err := ciphertextFile.Close(); err != nil {
        log.Fatal(err)
    }
    return nil

}

[chuckx]

It looks like your example is based on https://developers.google.com/tink/generate-encrypted-keyset#go.

A crucial comment from the example is:

// The encrypted keyset can now be stored.

To further explain, every time you run the example code a fresh StreamingAEAD keyset is generated by:

newHandle, err := keyset.NewHandle(streamingaead.AES256GCMHKDF1MBKeyTemplate())

If you want to be able to decrypt your ciphertext in a subsequent execution, you must store the contents of encryptedKeyset (e.g. into a file). Then, when attempting to decrypt, instead generating a new keyset, you read in the ecrypted keyset from wherever you stored it. Then you continue as before (decrypt the keyset using the KMS, create the primitive, decrypt the ciphertext).

---

Judging from your code example, I believe the guidance above should address your concern. Feel free to reopen this issue if it doesn't or if you have any follow up questions.

[dimapaikin]

Thank you @chuckx for your explanation! I'd like to clarify my goals and concerns. Based on your response, it appears that I have two options:

Storing the encryption key on the machine that performs encryption/decryption. Utilizing an external Key Management Service (KMS) like Vault. I'd prefer not to store the encryption key on the machine responsible for encryption/decryption. I'm hesitant about the first option since storing sensitive data on the local machine isn't secure, and keeping it in memory isn't a reliable solution.

The second option seems more suitable, but it requires me to manage the encryption key myself using an external KMS, which means I still need to store it in the KMS.

My objective is to delegate all security-related tasks to KMS and Tink. However, it seems that if I want to implement key rotation, I would need to do that myself.

My question is, is there an option to achieve what I want, where sensitive data is stored in a KMS, and I don't have to write the code for managing the storage and retrieval of encryption keys?

[dimapaikin]

@chuckx I believe that the feature I am referring to is described in the following GitHub issue: https://github.com/google/tink/issues/527

I'm curious to know if there are any plans or intentions to implement this feature in the near future?

[chuckx]

A few things to note:

1. KMSEnvelopeAEAD is a convenience type which makes it easier to to perform DEK (data encryption keyset) key wrapping with an external KMS, but it forfeits flexibility. In particular, every cryptographic operation requires a remote KMS call as each ciphertext has a unique, single-use DEK. This can introduce cost concerns when using a cloud KMS with per-call pricing, as well as potential reliability concerns due to latency, server load (frequency/size of requests), etc.
2. More importantly, in envelope encryption the DEK is literally bundled with the ciphertext. Each ciphertext produced is prefixed with the encrypted DEK. See the wire format documentation for envelope encryption for details.
3. As mentioned in the old issue you found, we have no plans to implement envelope encryption with streaming AEAD. KMS interactions are best suited for small payloads, such as key wrapping. This is counter to one of the main purposes behind streaming AEAD, which is handling large inputs. Echoing the concerns mentioned above, encrypting large amounts of data via a KMS introduces a new scaling concern around bandwidth usage and exacerbates concerns with server load.

One thing to reconsider is your stance on storing encrypted keysets. If you were to utilize envelope encryption, you would be committing to colocating encrypted keysets with ciphertexts. In general, we do not consider this a security issue given the security properties of the encrypted keyset (i.e. the key material is only accessible via a KMS decrypt call). However, it would obviously be a problem if the DEK was stored in an unencrypted manner.

Using the encrypted keyset approach is functionally equivalent to envelope encryption, except that it provides you more flexibility with how you want to use and store DEKs. This is more responsibility than the envelope encryption approach, but it better positions you to control how you scale in the future versus locking yourself into a configuration which limits your options.