General low level primitive for ciphers (AES-GCM being the first)

[Drawaes]

Rationale

There is a general need for a number of ciphers for encryption. Todays mix of interfaces and classes has become a little disjointed. Also there is no support for AEAD style ciphers as they need the ability to provide extra authentication information. The current designs are also prone to allocation and these are hard to avoid due to the returning arrays.

Proposed API

A general purpose abstract base class that will be implemented by concrete classes. This will allow for expansion and also by having a class rather than static methods we have the ability to make extension methods as well as hold state between calls. The API should allow for recycling of the class to allow for lower allocations (not needing a new instance each time, and to catch say unmanaged keys). Due to the often unmanaged nature of resources that are tracked the class should implement IDisposable

public abstract class Cipher : IDisposable
{
    public virtual int TagSize { get; }
    public virtual int IVSize { get; }
    public virtual int BlockSize { get; }
    public virtual bool SupportsAssociatedData { get; }

    public abstract void Init(ReadOnlySpan<byte> key, ReadOnlySpan<byte> iv);
    public abstract void Init(ReadOnlySpan<byte> iv);
    public abstract int Update(ReadOnlySpan<byte> input, Span<byte> output);
    public abstract int Finish(ReadOnlySpan<byte> input, Span<byte> output);
    public abstract void AddAssociatedData(ReadOnlySpan<byte> associatedData);
    public abstract int GetTag(Span<byte> span);
    public abstract void SetTag(ReadOnlySpan<byte> tagSpan);
}

Example Usage

(the input/output source is a mythical span based stream like IO source)

using (var cipher = new AesGcmCipher(bitsize: 256))
{
    cipher.Init(myKey, nonce);
    while (!inputSource.EOF)
    {
        var inputSpan = inputSource.ReadSpan(cipher.BlockSize);
        cipher.Update(inputSpan);
        outputSource.Write(inputSpan);
    }
    cipher.AddAssociatedData(extraInformation);
    cipher.Finish(finalBlockData);
    cipher.GetTag(tagData);
}

API Behaviour

1. If get tag is called before finish a [exception type?] should be thrown and the internal state should be set to invalid
2. If the tag is invalid on finish for decrypt it should be an exception thrown
3. Once finished is called, a call to anything other than one of the Init methods will throw
4. Once Init is called, a second call without "finishing" will throw
5. If the type expects an key supplied (a straight "new'd" up instance) if the Initial "Init" call only has an IV it will throw
6. If the type was generated say from a store based key and you attempt to change the key via Init and not just the IV it will throw
7. If get tag is not called before dispose or Init should an exception be thrown? To stop the user not collecting the tag by accident?

Reference dotnet/corefx#7023

Updates

1. Changed nonce to IV.
2. Added behaviour section
3. Removed the single input/output span cases from finish and update, they can just be extension methods
4. Changed a number of spans to readonlyspan as suggested by @bartonjs
5. Removed Reset, Init with IV should be used instead

[sdrapkin]

Quick-to-judge feedback on the proposed API (trying to be helpful):

• Span<T> is not in NetStandard2.
• "Nonce" is very implementation-specific - ie. smells of GCM. However, even GCM docs (ex. NIST SP800-38D) refer to it as "IV", which - in case of GCM - happens to be a nonce. However, in case of other AEAD implementations the IV does not have to be a nonce (ex. repeating IV under CBC+HMAC is not catastrophic).
• Streaming AEAD should either work seamlessly with CryptoStream, or provide its own "AEADCryptoStream" which is just as easy to stream in/out as CryptoStream.
• AEAD API imlementations should be allowed to do internal key derivation based on AAD (Associated Data). Using AAD purely for tag calculation/verification is too restrictive and prevents a stronger AEAD model.
• "Get*" methods should return something (GetTag). If they are void, they must be setting something/changing state.
• Trying to obtain a tag before "finishing" is probably a bad idea, so "IsFinished" might be helpful.
• Folks that designed ICryptoTransform thought about reuse, multi-block support, and differently-sized input/output block sizes. These concerns are not captured.

As a proof of AEAD API sanity, the first implementation of such proposed API should not be AES-GCM, but the classic/default AES-CBC with an HMAC tag. The simple reason for it is that anyone can build AES-CBC+HMAC AEAD implementation today, with simple, existing, well-known .NET classes. Let's get a boring old [AES-CBC+HMAC] working over the new AEAD APIs first, since that's easy for everyone to MVP and test-drive.

[Drawaes]

The nonce/IV naming issue was something I was undecided about, happy with a change to IV so will change.

As for Get methods returning something, this avoids any allocations. There could be an overload Get() that returns something. Maybe it requires a naming change, but I am pretty married to the idea that the whole API needs to be basically allocation free.

As for streams etc, I am not overly bothered with that as they are higher level API's that can easily be constructed from the lower level primitives.

Obtaining a Tag before you have finished should not be allowed, however you should know when you have called finish, so I am not sure it should be in the API, however it should be a defined behaviour so I have updated the API design to include a behaviour section so we can capture any others that are thought of.

As for which cipher, I don't think any specific cipher should be the sole target, in order to prove out a new general purpose API it needs to fit a number. AES GCM, and CBC should both be covered.

(all on topic feedback good or bad is always helpful! )

[bartonjs]

• Class, or interface?
• How, if at all, do the current SymmetricAlgorithm classes interact with this?
• How would this be used for persisted keys, like TripleDESCng and AesCng can do?
• Many of these Spans seem like they could be ReadOnlySpan.

[morganbr]

@Drawaes thanks for getting the ball rolling on this API. A few thoughts:

1. Tag generation and verification is a very important part of this API since misuse of tags can defeat the whole purpose. If possible, I'd like to see tags built into the initialize and finish operations to ensure they can't be accidentally ignored. That likely implies that encrypt and decrypt shouldn't use the same initialize and finalize methods.
2. I have mixed feelings about outputting blocks during decryption before getting to the end since the data isn't trustworthy until the tag has been checked (which can't be done until all data has been processed). We'll need to evaluate that tradeoff very carefully.
3. Is Reset necessary? Should finish just reset? We do that on incremental hashes (but they don't need new IVs)

[Drawaes]

@bartonjs

1. Class, as has often been seen in the BCL with an interface you can't expand it later without breaking everything. An interface is like a puppy for life... unless default interface methods can be considered as a solution to that problem. Also a sealed class from a abstract type is actually faster (as of now) because the jitt can devirtualise the methods now ... So it's basically free. interface dispatch isn't as good (still good just not as good)
2. I dunno how would you like that to work? I have little interest in the current stuff as it's so confusing I would just patch all reasonable modern algos straight in (leave 3DES in The other classes :) But I don't have all the answers so do you have any further thoughts on this?
3. Persisted keys should be easy. Make an extension method on the key method or persistence store.

MyKeyStore.GetCipher();

It's not initialized. It's disposable so any refs can be dropped by normal disposable pattern. If they try to set the key throw an invalid operation exception.

Yes to the read-only spans I will adjust when I am not on the tube on my phone.

[Drawaes]

@morganbr no problem... I just want to see it happen more than anything ;)

1. Can you give a code snippet on how you see that working? Not sure it does but code always brings clarity
2. It's unfortunate but you really have to spit out the blocks early. With hmac and hashing you don't but you don't have interim data just the state. So in this case you would have to buffer an unknown amount of data. Let's take a look at the pipelines example and TLS. We can write 16k of plaintext but the pipeline buffers are today the size of a 4k page. So we would at best want to encrypt/decrypt 4*4k. Of you don't give me the answer until the end you need to allocate internal memory to store all of that and then I presume throw it away when I get the result? Or will you wipe it. What if I decrypt 10mb and you hold that memory after I now have to worry about latent memory use.
3. Not 100% on the init/reset thing (not your ideas, my current API shape) it doesn't sit well with me so I am open to a new suggestion!

[bartonjs]

I have little interest in the current stuff as it's so confusing I would just patch all reasonable modern algos straight in (leave 3DES in The other classes :)

The problem would be that container formats like EnvelopedCms (or EncryptedXml) may need to work with 3DES-CBC, AES-CBC, etc. Someone wanting to decrypt something encrypted with ECIES/AES-256-CBC-PKCS7/HMAC-SHA-2-256 probably wouldn't feel that they're doing old and crufty things.

If it's only supposed to be for AE, then that should be reflected somewhere in the name. Right now it's generic "cipher" (which I was/am, at some point, going to sit down with a dictionary/glossary and find out if there's a word for "an encryption algorithm in a mode of operation", since I think that "cipher" == "algorithm", therefore "Aes").

[Drawaes]

:) I was merely pointing out that it's not my subject area or of much interest to me so I am willing to defer to you and the community on this topic I haven't thought through the implications for this.

---

After quickly scanning through these, one option is allow them to take an instance of the "Cipher" or whatever it's called class. This might not be done in the first wave, but could quickly follow it up. If the API is super efficient then I see no reason they should do their own thing internally and is exactly the use case for this API.

[Drawaes]

As a side bar on the naming... I must admit its a tough one however Openssl = cipher Ruby = cipher Go = cipher package with ducktyped interfaces for AEAD etc Java = cipher

Now I am all for being different but... there is a trend. If something better is possible that is cool.

Possibly "BlockModeCipher" ... ?

[Drawaes]

I have made a few changes, I will change naming if a better name is decided upon.

[morganbr]

When I started trying to answer questions, I realized the API is already missing encryption/decryption differentiation, so in your example, it doesn't know whether to encrypt or decrypt data. Getting that put in might add some clarity.

I could imagine a couple ways that the API could enforce proper tag usage (based on the assumption that this is an AEAD API, not just symmetric encryption since we already have SymmetricAlgorithm/ICryptoTransform/CryptoStream). Don't take these as prescriptive, just as an example of enforcing the tagging. By method:

class Cipher
{
   void InitializeEncryption(ReadOnlySpan<byte> key, ReadOnlySpan<byte> iv);
   // Ensures decryptors get a tag
   void InitializeDecryption(ReadOnlySpan<byte> key, ReadOnlySpan<byte> iv, ReadOnlySpan<byte> tag);
   // Ensure encryptors produce a tag
    void FinishEncryption(ReadOnlySpan<byte> input, Span<byte> output, Span<byte> tag);
   // Throws if tag didn't verify
   void FinishDecryption(ReadOnlySpan<byte> input, Span<byte> output);
   // Update and properties are unchanged, but GetTag and SetTag are gone
}

By class:

class Cipher
{
    // Has properties and update, but Initialize and Finish aren't present
}
class Encryptor : Cipher
{
    void Initialize(ReadOnlySpan<byte> key, ReadOnlySpan<byte> iv);
    void Finish(ReadOnlySpan<byte> input, Span<byte> output, Span<byte> tag);
}
class Decryptor : Cipher
{
   void Initialize(ReadOnlySpan<byte> key, ReadOnlySpan<byte> iv, ReadOnlySpan<byte> tag);
   // Throws if tag didn't verify
   void Finish(ReadOnlySpan<byte> input, Span<byte> output);
}
class AesGCMEncryptor : Encryptor {}
class AesGCMDecryptor : Decryptor {}
}

That said, if it doesn't buffer on decryption, is it practical to ensure decryption actually gets finished and the tag gets checked? Can Update somehow figure out that it's time to check? Is that something Dispose should do? (Dispose is a bit dangerous since you may have already trusted the data by the time you dispose the object)

As far as naming, our precedent is SymmetricAlgorithm and AsymmetricAlgorithm. If this is intended for AEAD, some ideas could be AuthenticatedSymmetricAlgorithm or AuthenticatedEncryptionAlgorithm.

[sdrapkin]

Some thoughts & API ideas:

public interface IAEADConfig
{
    // size of the input block (plaintext)
    int BlockSize { get; }

    // size of the output per input-block;
    // typically a multiple of BlockSize or equal to BlockSize.
    int FeedbackSize { get; }

    // IV size; CAESAR completition uses a fixed-length IV
    int IVSize { get; }

    // CAESAR competition uses a fixed-length key
    int KeySize { get; }

    // CAESAR competition states that typical AEAD ciphers have a constant gap between plaintext length
    // and ciphertext length, but the requirement is to have a constant *limit* on the gap.
    int MaxTagSize { get; }

    // allows for AE-only algorithms
    bool IsAdditionalDataSupported { get; }
}

public interface ICryptoAEADTransform : ICryptoTransform
{
    // new AEAD-specific ICryptoTransform interface will allow CryptoStream implementation
    // to distinguish AEAD transforms.
    // AEAD decryptor transforms should throw on auth failure, but current CryptoStream
    // logic swallows exceptions.
    // Alternatively, we can create a new AEAD_Auth_Failed exception class, and
    // CryptoTransform is modified to catch that specific exception.
}

public interface IAEADAlgorithm : IDisposable, IAEADConfig
{
    // separates object creation from initialization/keying; allows for unkeyed factories
    void Initialize(ArraySegment<byte> key);

    void Encrypt(
        ArraySegment<byte> iv, // readonly; covered by authentication
        ArraySegment<byte> plaintext, // readonly; covered by authentication
        ref ArraySegment<byte> ciphertext, // must be of at least [plaintext_length + MaxTagSize] length. iv is not part of ciphertext.
        ArraySegment<byte> additionalData = default(ArraySegment<byte>) // readonly; optional; covered by authentication
        ); // no failures expected under normal operation - abnormal failures will throw

    bool Decrypt(
        ArraySegment<byte> iv, // readonly
        ArraySegment<byte> ciphertext, // readonly
        ref ArraySegment<byte> plaintext, // must be of at least [ciphertext_length - MaxTagSize] length.
        ArraySegment<byte> additionalData = default(ArraySegment<byte>), // readonly; optional
        bool isAuthenticateOnly = false // supports Authentication-only mode
        );// auth failures expected under normal operation - return false on auth failure; throw on abnormal failure; true on success

    /*  Notes:
        * Array.LongLength should be used instead of Array.Length to accomodate byte arrays longer than 2^32.
        * Ciphertext/Plaintext produced by Encrypt()/Decrypt() must be determined *only* by method inputs (combined with a key).
          - (ie. if randomness or other hidden inputs are needed, they must be a part of iv)
        * Encrypt()/Decrypt() are allowed to write to ciphertext/plaintext segments under all conditions (failure/success/abnormal)
          - some implementations might be more stringent than others, and ex. not leak decrypted plaintext on auth failures
    */

    ICryptoAEADTransform CreateEncryptor(
        ArraySegment<byte> key,
        ArraySegment<byte> iv,
        ArraySegment<byte> additionalData = default(ArraySegment<byte>)
        );

    ICryptoAEADTransform CreateDecryptor(
        ArraySegment<byte> key,
        ArraySegment<byte> iv,
        ArraySegment<byte> additionalData = default(ArraySegment<byte>)
        );

    // Streaming AEAD can be done with good-old CryptoStream
    // (possibly modified to be AEAD-aware).
}

[morganbr]

@sdrapkin , thanks for contributing thoughts. Where should tags go in your API? Are they implicitly part of the ciphertext? If so, that implies a protocol that some algorithms don't actually have. I'd like to understand what protocols people might care about to see whether implicit tag placement is acceptable or if it needs to be carried separately.

[Drawaes]

@morganbr I noticed the encrypt/decrypt issue as well but hadn't had time tonight to fix so glad for your design. I prefer the methods over the class as it allows better aggressive recycling (buffers for keys and IV's can add up).

As for a check before the dispose. It's not possible to tell the end of an operation unfortunately.

@sdrapkin interfaces I would say are a no go due to the version problem mentioned previously. Unless we rely on default implantation in the future. Also interface dispatch is slower.. the array segments also are our as span is the more versatile lower primitive. However extension methods could be added for arraysegment to span if there was demand later.

Some of your properties are of interest so will update when I am at a computer rather than on my phone.

Good feedback all round!

[sdrapkin]

@morganbr Tags are part of ciphertext. This is modelled after CAESAR API (which includes AES-GCM).

@Drawaes I've used interfaces to illustrate thoughts only - I'm perfectly ok with static methods/classes. Span does not exist. I don't care about what may or may not be coming - it is not in NetStandard2, and it is not in the normal .NET that serious projects actually use (yeah, yeah, I know it's in Core, but Core is a toy for now). I'd be happy to personally consider Span when I see it - until then ArraySegment is the closest NetStandard API that actually ships.

[Drawaes]

I will take a deeper look athe CAESAR API that is useful.

As for Span, it is shipping around the 2.1 time frame I believe which is hopefully the same time the first implementation of this API would ship (or at least the earliest possible time it could).

If you take a look at the current prerelease nuget package it supports down to .net standard 1.0 and there is no plans to change that on release.

Maybe @stephentoub can confirm that as he is doing work to add Span based API's across the framework as we speak.

(Nuget for Span)[https://www.nuget.org/packages/System.Memory/4.4.0-preview2-25405-01]

So I would say it's the only real choice for a brand new API. Then extension methods etc can be added to take an ArraySegment if you so chose and if it is useful enough then it could be added to the framework but it's trivial to turn an ArraySegment into a Span, but the other way requires copying data.

[Drawaes]

The problem I see with that API above is it will be a disaster for performance on any "chunked" data. Say for instance network traffic, if a single authenticated block is split over multiple reads from an existing stream I need to buffer it all into a single [insert data structure] and encrypt/decrypt all at once. The defeats all attempts to do any kind of zero copy on that data.

Networking frameworks such as those that Pipelines provides manage to avoid almost all copies, but then if they hit any kind of crypto in this API all of that is gone.

The separate configuration object (or bag) has actually been involved in a recent discussion on another API I have been having. I am not opposed to it in principle as if it grows in future it can become a mess to have large numbers of properties on the main object.

[bartonjs]

A couple of things have occurred to me.

• The current proposal calls TagSize an out value (well, a get-only-property). But for both GCM and CCM it's an input to encryption (and derivable from decryption since you've supplied the actual tag).
• The proposal assumes that input and output can happen at the same time, and piecemeal.
  • IIRC CCM can't do streaming encryption (the length of the plaintext is an input into the first step of the algorithm).
  • Padded modes lag decryption by (at least one) block, since until Final is called they don't know if more data is coming / if the current block needs the padding removed
• Some algorithms might consider the AD element to be required at the beginning of the operation, making it more like an Init/ctor parameter than a late-bound association.

I don't know if container formats (EnvelopedCms, EncryptedXml) would need to extract the key, or if it would simply be up to them to generate it and remember it (for as long as they need to to write it down).

(Apparently I didn't hit the "comment" button on this yesterday, so it won't be acknowledging anything after "I have made a few changes" at 1910Z)

[Drawaes]

True tag size would need to be variable. Agreed.

If we just look at encryption for now, to simplify the use case. You are correct that some ciphers will return nothing, less or more. There is a general question around what happens if you don't provide a big enough buffer.

On the new TextEncoding interfaces using span there was a suggestion around having the return type an enum to define if there was enough space to output or not, and the size actually written in an "out" param instead. This is a possibility.

In the CCM case I would just say it returns nothing and will have to internally buffer until you call finish at which point it would want to dump the whole lot. Nothing precludes you from just calling finish as your first call if you have all the data in a single block (In which case there might be a better name). Or it is possible to throw if you try an update on those ciphers. CNG returns an invalid size error if you try to do a continuation on CCM for instance.

As for when the tag is set on decryption, you often don't know it until you have read the entire packet, if we take TLS as an example we might have to read 8 * 2k network packets to get to the tag at the end of a 16k block. So we now have to buffer the entire 16k before we can start decryption and there is no chance to overlap (I am not saying this would be used for TLS just that an IO bound process is common for these types of ciphers, be it disk or network).

[sdrapkin]

@Drawaes re. chunked streams and buffering limits: You have to pick your battles. You won't be able to create a unifying API aligned with every singe nice-to-have goal in the AE world - and there are many such goals. Ex. there is a decent chunked AEAD streaming in Inferno, but it's not a standard by any stretch, and such a standard does not exist. At a higher level, the goal is "secure channels" (see this , this, and this).

However, we need to think smaller for now. Chunking/buffer-limiting are not even on the radar for standardisation efforts ("AEADs with large plaintexts" section)..

Encryption/Decryption operations are fundamentally about transforms. These transforms require buffers, and are not in-place (output buffers must be larger than input buffers - at least for Encrypt transform).

[sdrapkin]

RFC 5116 might also be of interest.

[morganbr]

@Drawaes , interesting that you bring up TLS. I would argue that SSLStream (were it to use this API) must not return any unauthenticated results to an application since the application won't have any way to defend itself.

[Drawaes]

Sure, but that is SSLStreams problem. I have prototyped this exact thing (managed TLS at the protocol level calling out to CNG and OpenSSL for the crypto bits) on top of pipelines. The logic was pretty simple, encrypted data comes in, decrypt the buffer in place, attach to the out bound and repeat until you get to the tag. At the tag call finished...

If it throws close the pipeline. If it doesn't throw then flush allowing the next stage to go to work either on the same thread or via dispatch.

My proof of concept was not ready for primetime but by using this and avoiding alot of copies etc it showed a very decent perf increase ;)

The problem with any networking is where things in the pipeline start allocating their own buffers and not using as much as possible the ones moving through the system already.

[Drawaes]

OpenSsl's crypt, and CNG have this same method Update,Update, Finish. Finish can ouput the tag as discussed. Updates must be in block size (for CNG) and for OpenSsl it does minimal buffering to get to a block size.

As they are primitives, I am not sure we would expect higher level functionality from them. If we were designing a "user" level API rather than primitives to construct those I would then argue that key generation, IV construction and entire authenticated chunks should all be implemented so I guess it depends what the target level of this API really is.

[Drawaes]

Wrong button

[morganbr]

@blowdart, who had some interesting ideas on nonce management.

[blowdart]

So nonce management is basically a user problem and is specific to their setup.

So ... make it a requirement. You must plug in nonce management ... and don't supply a default implementation, or any implementations at all. This, rather than a simple

cipher.Init(myKey, nonce);

forces users to make a specific gesture that the understand the risks.

[morganbr]

@blowdart's idea might help with both nonce management problems and differences between algorithms. I agree that it's likely important not to have built-in implementation to ensure that users understand that nonce management is a problem they need to solve. How does something like this look?

interface INonceProvider
{
    public void GetNextNonce(Span<byte> writeNonceHere);
}

class AesGcmCipher : Cipher
{
    public AesGcmCipher(ReadOnlySpan<byte> key, INonceProvider nonceProvider);
}

// Enables platform-specific hardware keys
class AesGcmCng : Cipher
{
    public AesGcmCng(CngKey key, INonceProvider nonceProvider);
}

// Example of AEAD that might not need a nonce
class AesCBCHmac : Cipher
{
    public AesCBC(ReadOnlySpan<byte> key)
}

class Cipher
{
    // As above, but doesn't take keys, IVs, or nonces
}

[Drawaes]

But whats the point of the INonceProvider? It's just an extra inteface/type, if the Init just takes a none and is needed to be called before you start any block, isn't that the same thing without an extra/interface?

[Drawaes]

Also I am no crypto expert but doesn't AES require an IV (Which isn't a nonce but needs to be provided by the user?)

[morganbr]

It's just an extra inteface/type

That's kind of the point. It essentially says that nonce management is a problem, not just passing in a zeroed or even random byte array. It might also help prevent inadvertent reuse if people interpret GetNextNonce to mean "return something different than you did last time".

It's also helpful to not need it for algorithms that don't have nonce management problems (like AES SIV or perhaps AES+CBC+HMAC).

[morganbr]

The exact IV/nonce requirements vary based on mode. For example:

• AES ECB does not require a nonce or IV
• AES GCM requires a 96-bit nonce that must never be reused or the security of the key breaks. Low entropy is fine as long as the nonce isn't reused.
• AES CBC requires a 128-bit IV that must be random. If the IV repeats, it only reveals whether the same message has been sent before.
• AES SIV doesn't need an explicit IV since it derives it from other inputs.

[Drawaes]

AES CBC needs IV right? So are you going to have an InitializationVectorProvider? Its not a nonce but nonce like and reusing the last block led to a tls attack because the iv can be predicted. You explicitly can't use say a sequential nonce for CBC.

[Drawaes]

Yeah but an IV isn't a nonce so you can't have the term nomce provider

[morganbr]

I didn't mean to imply AES CBC doesn't need an IV-- it does. I just meant to speculate about some schemes that derive the IV from other data.

[Drawaes]

Sure I guess my point is I like it generally... I can pool the provider ;) but either call it an iv provider or have 2 interfaces to be clear on intent

[sdrapkin]

@morganbr INonceProvider factories passed into cipher constructors are a bad design. It completely misses the fact that nonce does not exist by itself: the "...used once" constraint has a context. In cases of CTR and GCM (which uses CTR) modes, the context of the nonce is the key. Ie. nonce provider must return a nonce that is used only once within a context of a specific key.

Since the INonceProvider in your proposed API is not key-aware, it cannot generate correct nonces (other than via randomness, which is not what a nonce is, even if the bit space was large enough for statistical randomness to work safely).

[sdrapkin]

I'm not entirely sure what this discussion thread aims to achieve. Various Authenticated-Encryption design ideas are discussed... ok. What about Authenticated Encryption interfaces already built into .NET Core -- specifically into its ASP.NET API? There is IAuthenticatedEncryptor, etc. All these capabilities are already implemented, extensible, and shipping as part of .NET Core today. I'm not saying DataProtection crypto is perfect, but is the plan to ignore them? Change them? Assimilate or refactor?

DataProtection crypto was built by @GrabYourPitchforks (Levi Broderick). He knows the subject matter, and his opinion/input/feedback would be most valuable to this community. I enjoy crypto-themed entertainment as much as anyone, but if someone wants to get serious about crypto API design, then actual experts that are already on MS team should be engaged.

[morganbr]

@sdrapkin, nonce providers needing to be key aware is a good point. I wonder if there's a reasonable way to modify these APIs to enforce that.

DataProtection is a fine API, but it's a higher-level construct. It encapsulates key generation and management, IVs and output protocol. If somebody needs to use (say) GCM in an implementation of a different protocol, DataProtection doesn't make that possible.

The .NET crypto team includes @bartonjs, @blowdart and myself. Of course, if @GrabYourPitchforks wants to chime in, he's more than welcome.

[Drawaes]

I agree with @morganbr in that this is supposed to be a low level primitive ( in fact it says that in the title). While data protection etc are designed to be used directly in usercode and reduces the ability to shoot yourself in the foot, the way I see this primitive is to allow the framework and libraries to build higher level constructs on a common base.

With that thought in mind, the provider is fine if it has to be supplied anytime a key is supplied. It does make it a little messy let me explain using TLS (its a well known use of AES block modes for network traffic is all).

I get a "frame" (maybe over 2 + TU's with the MTU ~1500 of the internet). It contains the nonce (or part of the nonce with 4 bytes left "hidden") I then have to set this value on a shell "provider" and then call decrypt and go through my cycle of decrypting the buffers to get a single plain text.

If you are happy with that, I can live with it. I am keen to get this moving along so keen to update the design above to something we can agree on.

[SidShetye]

Thanks for forking the discussion, getting some free time to jump onto this. @Drawaes , can you confirm/update the top post as gold standard / target of this evolving conversation? If not, can you update it?

I see the current proposal having a fatal issue and then other issues with being too chatty.

// current proposed usage
using (var cipher = new AesGcmCipher(bitsize: 256))
{
    cipher.Init(myKey, nonce);
    while (!inputSource.EOF)
    {
        var inputSpan = inputSource.ReadSpan(cipher.BlockSize);
        cipher.Update(inputSpan);
        outputSource.Write(inputSpan);
    }
    cipher.AddAssociatedData(extraInformation); // <= fatal, one can't just do this
    cipher.Finish(finalBlockData);
    cipher.GetTag(tagData);
}

If you look at a true AEAD primitive, the privacy data and the authenticated data are mixed lock-step. See this for Auth Data 1 and CipherText1. This of course continues for multiple blocks, not just 1.

Since all the world's a meme, can't resist, sorry :)

Also, the API seems chatty with new, init, update etc. I'd propose this programmer's model

// proposed, see detailed comments below
using (var cipher = new AesGcmCipher(myKey, iv, aad)) // 1
{
    // 2
    while (!inputSource.EOF) 
    {
        var inputSpan = inputSource.ReadSpan(16411); // 3
        var outSpan = cipher.Encrypt(inputSpan); // 4
        outputSource.Write(outSpan); 
    }    
    var tag = cipher.Finish(finalBlockData); // 5
}

1. Typically AAD << plaintext so I've seen cipher.Init(mykey, nonce, aad); where the entire AAD is passed as a buffer and then the cipher crunches over the rest of the potentially gigabyte+ stream. (e.g. BCryptEncrypts's CipherModeInfo param). Also, size of myKey already establishes AES128, 192, 256, no need for another parameter.
2. Init becomes an optional API in case caller wishes to reuse existing class, existing AES constants and skipping AES subkey generation if AES key is same
3. cipher's API should shield caller from block size management internals like most other crypto APIs or even existing .NET APIs. Caller concerned with buffer size optimized for their use case e.g. network IO via 16K+ buffers). Demoing with a prime number > 16K to test implementation assumptions
4. inputSpan is readonly. And input. So need an outSpan
5. Update() does Encrypt or Decrypt? Just have Encrypt and Decrypt interfaces to match a developer's mental model. tag is also the most important desired data at this instant, return that.

Actually going a step further, why not just

using (var cipher = new AesGcmCipher(myKey, iv, aad))
{
    var tag = cipher.EncryptFinal(inputSpan, outputSpan);
}

[SidShetye]

Also, please steer away from INonceProvider and the sorts. The crypto primitives don't need this, just stick to byte[] iv (my favorite for small data) or Span (the supposed new cool but too much abstraction IMHO). Nonce provider operates at a layer above and the result of it could just be the iv seen here.

[blowdart]

The problem with making primitives so primitive is people will simply use them incorrectly. With a provider we can at least force some thought into their use.

[SidShetye]

We're talking about AEAD in general of which GCM is specific. So first, the generalized case (iv) should drive the design, not the specific case (nonce).

Secondly, how does merely shifting from byte[] iv to GetNextNonce(Span<byte> writeNonceHere) actually solve the nonce issue? You've only changed the name/label on the problem while simultaneously making it more complex than it should be.

Third, since we're getting into policies on iv protection should we also get into key protection policies? What about key distribution policies? Those are obviously higher level concerns.

Finally, nonce is extremely situational on usage at higher layers. You don't want to have a brittle architecture where cross-layer concerns are being coupled together.

[blowdart]

Frankly if we could hide primitives unless someone makes a gesture to say I know what I'm doing I'd push for that. But we can't. There are far too many bad crypto implementations out there because people though "Oh this is available, I'll use it". Heck look at AES itself, I'll just use that with no HMAC.

I want to see APIs be secure by default, and if that means a little more pain then frankly I'm all for it. 99% of developers do not know what they're doing when it comes to crypto and making it easy for the 1% who do should be a lower priority.

[benaadams]

Span does not exist. I don't care about what may or may not be coming - it is not in NetStandard2

@sdrapkin as @Drawaes points out Span<T> is .NET Standard 1.0 so can be used on any framework. Its also safer than ArraySegment<T> as it only lets you access the actual window referenced; rather than the whole array.

Also ReadOnlySpan<T> prevents modification to that window; again unlike array segment where anything passed it can modify and/or retain a reference to the passed array.

Span should be the general go to for sync apis (The fact an api using Span can additionally cope with stackalloc'd, native memory as well as arrays; is the icing)

i.e. With ArraySegment the readonly is suggested via docs; and no out of bounds read/modifications are prevented

void Encrypt(
    ArraySegment<byte> iv, // readonly; covered by authentication
    ArraySegment<byte> plaintext, // readonly; covered by authentication
    ref ArraySegment<byte> ciphertext, // must be of at least [plaintext_length + MaxTagSize] length. iv is not part of ciphertext.
    ArraySegment<byte> additionalData = default(ArraySegment<byte>) // readonly; optional; covered by authentication
    );

However with Span the readonly is enforced by api; as well as out of bounds reads of the arrays being prevented

void Encrypt(
    ReadOnlySpan<byte> iv, // covered by authentication
    ReadOnlySpan<byte> plaintext, // covered by authentication
    Span<byte> ciphertext, // must be of at least [plaintext_length + MaxTagSize] length. iv is not part of ciphertext.
    ReadOnlySpan<byte> additionalData = ReadOnlySpan<byte>.Empty) // optional; covered by authentication
    );

It conveys intent with the parameters far better; and is less error prone with regards to out of bounds reads/writes.

[sdrapkin]

@benaadams @Drawaes never said that Span<T> was in NetStandard (any shipped NetStandard). What he did say is (1) agree that Span<T> is not in any shipped NetStandard; (2) that Span<T> will be "shipping around the 2.1 time frame".

For this particular Github issue, however, (read-only) Span<T> discussion is bikeshedding right now - there is no clarity on scope or purpose of the API to be designed.

Either we go with raw low-level primitive AEAD API (ex. similar to CAESAR):

• Pros: nice fit for AES-GCM/CCM, existing test vectors from good sources (NIST, RFC). @sidshetye will be happy. @blowdart will meditate over "making primitives so primitive", but will eventually see the Yin and Yang because primitives are primitive and there is no way to childproof them.
• Cons: Expert users (the proverbial 1%) will use the low-level APIs responsibly, while the other non-expert users (99%) will misuse it to write broken .NET software that will be responsible for the vast majority of .NET CVEs, which will greatly contribute to the perception that .NET is an insecure platform.

Or we go with high-level misuse-impossible or -resistant AEAD API:

• Pros: 99% of non-expert users will continue to make mistakes, but at least not in AEAD code. @blowdart 's "I want to see APIs be secure by default" resonates deeply in the ecosystem, and security, prosperity, and good karma befall all. Many good API designs & implementations are already available.
• Cons: No standards. No test vectors. No consensus on whether AEAD is even the right goal to target for high-level online streaming API (spoiler: it's not - see Rogaway's paper).

Or, we do both. Or we enter analysis paralysis and might as well close this issue right now.

[SidShetye]

I strongly feel that being part of the core language, crypto needs to have a solid, low level foundational API. Once you have that, creating high level APIs or "training wheels" APIs can be bridged quickly by the core or community. But I challenge anyone to do the reverse elegantly. Plus the topic is "General low level primitive for ciphers" !

@Drawaes is there a timeline to converge and resolve this? Any plans on involving non-Microsoft folks beyond such GitHub alerts? Like a 30 min conference call? I'm trying to stay out of a rabbit hole but we're betting that .NET core crypto will be at a certain level of maturity and stability .. so can triage for such discussions.

[morganbr]

We're still paying attention and working on this. We've met with the Microsoft Cryptography Board (the set of researchers and other experts who advice Microsoft's usage of cryptography) and @bartonjs will have more information to share soon.

[bartonjs]

Based on a little data flow doodling and the advice of the Crypto Board we came up with the following. Our model was GCM, CCM, SIV and CBC+HMAC (note that we're not talking about doing SIV or CBC+HMAC right now, just that we wanted to prove out the shape).

public interface INonceProvider
{
    ReadOnlySpan<byte> GetNextNonce(int nonceSize);
}

public abstract class AuthenticatedEncryptor : IDisposable
{
    public int NonceOrIVSizeInBits { get; }
    public int TagSizeInBits { get; }
    public bool SupportsAssociatedData { get; }
    public ReadOnlySpan<byte> LastNonceOrIV { get; }
    public ReadOnlySpan<byte> LastTag { get; }

    protected AuthenticatedEncryptor(
        int tagSizeInBits,
        bool supportsAssociatedData,
        int nonceOrIVSizeInBits) => throw null;

    protected abstract bool TryEncrypt(
        ReadOnlySpan<byte> data,
        ReadOnlySpan<byte> associatedData,
        Span<byte> encryptedData,
        out int bytesWritten,
        Span<byte> tag,
        Span<byte> nonceOrIVUsed);

    public abstract void GetEncryptedSizeRange(
        int dataLength,
        out int minEncryptedLength,
        out int maxEncryptedLength);

    public bool TryEncrypt(
        ReadOnlySpan<byte> data,
        ReadOnlySpan<byte> associatedData,
        Span<byte> encryptedData,
        out int bytesWritten) => throw null;

    public byte[] Encrypt(
        ReadOnlySpan<byte> data,
        ReadOnlySpan<byte> associatedData) => throw null;

    // some variant of the Dispose pattern here.
}

public sealed class AesGcmEncryptor : AuthenticatedEncryptor
{
    public AesGcmEncryptor(ReadOnlySpan<byte> keySize, INonceProvider nonceProvider)
        : base(128, true, 96)
    {
    }
}

public sealed class AesCcmEncryptor : AuthenticatedEncryptor
{
    public AesCcmEncryptor(
        ReadOnlySpan<byte> key,
        int nonceSizeInBits,
        INonceProvider nonceProvider,
        int tagSizeInBits)
        : base(tagSizeInBits, true, nonceSizeInBits)
    {
        validate nonceSize and tagSize against the algorithm spec;
    }
}

public abstract class AuthenticatedDecryptor : IDisposable
{
     public abstract bool TryDecrypt(
        ReadOnlySpan<byte> tag,
        ReadOnlySpan<byte> nonceOrIV,
        ReadOnlySpan<byte> encryptedData,
        ReadOnlySpan<byte> associatedData,
        Span<byte> data,
        out int bytesWritten);

    public abstract void GetEncryptedSizeRange(
        int encryptedDataLength,
        out int minDecryptedLength,
        out int maxDecryptedLength);

    public byte[] Decrypt(
        ReadOnlySpan<byte> tag,
        ReadOnlySpan<byte> nonceOrIV,
        ReadOnlySpan<byte> encryptedData,
        ReadOnlySpan<byte> associatedData) => throw null;

    // some variant of the Dispose pattern here.
}

public sealed class AesGcmDecryptor : AuthenticatedDecryptor
{
    public AesGcmDecryptor(ReadOnlySpan<byte> key) => throw null;
}

public sealed class AesCcmDecryptor : AuthenticatedDecryptor
{
    public AesCcmDecryptor(ReadOnlySpan<byte> key) => throw null;
}

This proposal eliminates data streaming. We don't really have a lot of flexibility on that point. Real-world need (low) combined with the associated risks (extremely high for GCM) or impossibility thereof (CCM) means it's just gone.

This proposal uses an externalized source of nonce for encryption. We will not have any public implementations of this interface. Each application/protocol should make its own tying the key to the context so it can feed things in appropriately. While each call to TryEncrypt will only make one call to GetNextNonce there's no guarantee that that particular TryEncrypt will succeed, so it's still up to the application to understand if that means it should re-try the nonce. For CBC+HMAC we would create a new interface, IIVProvider, to avoid muddying the terminology. For SIV the IV is constructed, so there's no acceptable parameter; and based on the spec the nonce (when used) seems to just be considered as part of the associatedData. So SIV, at least, suggests that having nonceOrIV as a parameter to TryEncrypt is not generally applicable.

TryDecrypt most definitely throws on invalid tag. It only returns false if the destination is too small (per the rules of Try- methods)

Things that are definitely open for feedback:

• Should sizes be in bits (like significant parts of the specs) or bytes (since only %8 values are legal anyways, and we're always going to divide, and some of the parts of specs talk about things like the nonce size in bytes)?
• Parameter names and ordering.
• The LastTag/LastNonceOrIV properties. (Making them be (writable) Spans on a public TryEncrypt just means there are three buffers that could be too small, by making them sit on the side the "Try" is more clear; the base class can make the promise that it'll never offer a too-short buffer.).
• Offering up an AE algorithm for which this doesn't work.
• Should associatedData be moved to the end with a default of ReadOnlySpan<byte>.Empty?
  • Or overloads made which omit it?
• Does anyone want to assert love for, or hatred for, the byte[]-returning methods? (Low-allocation can be achieved by using the Span method, this is just for convenience)
• The size ranges methods were sort of bolted on at the end.
  • Their purpose is that
  • If the destination span is less than min, return false immediately.
  • The byte[]-returning methods will allocate a buffer of max, and then Array.Resize it as needed.
  • Yes, for GCM and CCM min=max=input.Length, but that's not true for CBC+HMAC or SIV
  • Is there an algorithm that would need to take the associatedData length into account?