Port S.S.C.ECDiffieHellman

[karelz]

@bartonjs is it part of netstandard2.0 effort?

[bartonjs]

@karelz I'm not sure. It's in System.Core.dll and from before 4.6, so... definitely maybe? But in order to be useful cross-platform it requires API changes which might not have been there until 4.6.2. @danmosemsft could say if it's something he's tracking.

[danmoseley]

Wes factored it out as it's not in Xamarin.

https://github.com/dotnet/standard/blob/6d04955a71f4789c7a10f5430d1802d1768460cc/extensions/crypto/crypto.cs

Note it's in standard/extensions and not in standard/netstandard/ref.

This is publicly visible at https://apisof.net/catalog/System.Security.Cryptography.ECDiffieHellman, which does not list ".NET Standard 2.0" as eg https://apisof.net/catalog/System.String does.

[danmoseley]

Change of plan, we now want all 4.6.2 and 4.6.3 API in NS2.0. So we do need to port these to Core. Here's the API:

M:System.Security.Cryptography.ECDiffieHellman.Create(System.Security.Cryptography.ECCurve)
M:System.Security.Cryptography.ECDiffieHellman.Create(System.Security.Cryptography.ECParameters)
M:System.Security.Cryptography.ECDiffieHellman.DeriveKeyFromHash(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Security.Cryptography.HashAlgorithmName)
M:System.Security.Cryptography.ECDiffieHellman.DeriveKeyFromHash(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Security.Cryptography.HashAlgorithmName,System.Byte[],System.Byte[])
M:System.Security.Cryptography.ECDiffieHellman.DeriveKeyFromHmac(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Security.Cryptography.HashAlgorithmName,System.Byte[])
M:System.Security.Cryptography.ECDiffieHellman.DeriveKeyFromHmac(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Security.Cryptography.HashAlgorithmName,System.Byte[],System.Byte[],System.Byte[])
M:System.Security.Cryptography.ECDiffieHellman.DeriveKeyTls(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Byte[],System.Byte[])
M:System.Security.Cryptography.ECDiffieHellman.ExportExplicitParameters(System.Boolean)
M:System.Security.Cryptography.ECDiffieHellman.ExportParameters(System.Boolean)
M:System.Security.Cryptography.ECDiffieHellman.GenerateKey(System.Security.Cryptography.ECCurve)
M:System.Security.Cryptography.ECDiffieHellman.ImportParameters(System.Security.Cryptography.ECParameters)
M:System.Security.Cryptography.ECDiffieHellmanCng.#ctor(System.Security.Cryptography.ECCurve)
M:System.Security.Cryptography.ECDiffieHellmanCng.DeriveKeyFromHash(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Security.Cryptography.HashAlgorithmName,System.Byte[],System.Byte[])
M:System.Security.Cryptography.ECDiffieHellmanCng.DeriveKeyFromHmac(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Security.Cryptography.HashAlgorithmName,System.Byte[],System.Byte[],System.Byte[])
M:System.Security.Cryptography.ECDiffieHellmanCng.DeriveKeyTls(System.Security.Cryptography.ECDiffieHellmanPublicKey,System.Byte[],System.Byte[])
M:System.Security.Cryptography.ECDiffieHellmanCng.ExportExplicitParameters(System.Boolean)
M:System.Security.Cryptography.ECDiffieHellmanCng.ExportParameters(System.Boolean)
M:System.Security.Cryptography.ECDiffieHellmanCng.GenerateKey(System.Security.Cryptography.ECCurve)
M:System.Security.Cryptography.ECDiffieHellmanCng.ImportParameters(System.Security.Cryptography.ECParameters)
M:System.Security.Cryptography.ECDiffieHellmanCngPublicKey.ExportExplicitParameters
M:System.Security.Cryptography.ECDiffieHellmanCngPublicKey.ExportParameters
M:System.Security.Cryptography.ECDiffieHellmanPublicKey.#ctor
M:System.Security.Cryptography.ECDiffieHellmanPublicKey.ExportExplicitParameters
M:System.Security.Cryptography.ECDiffieHellmanPublicKey.ExportParameters

[bartonjs]

For the OpenSSL-based implementation of this, here's a snippet of a quick tool that I wrote to make sure Windows and OpenSSL were on the same page.

static void* KDF1_SHA256(const void* in, size_t inlen, void* out, size_t* outlen)
{
    *outlen = SHA256_DIGEST_LENGTH;
    return SHA256(in, inlen, out);
}

int main(int argc, char** argv)
{
    BIO* bio_err = BIO_new_fp(stderr, BIO_NOCLOSE | BIO_FP_TEXT);
    BIO* in = BIO_new(BIO_s_file());
    EC_KEY* eckey;
    unsigned char kdfout[2048];
    int copied;

    ERR_load_crypto_strings();
    OpenSSL_add_all_algorithms();

    if (BIO_read_filename(in, "grr.pem") <= 0)
    {
        ERR_print_errors(bio_err);
        return 1;
    }

    eckey = PEM_read_bio_ECPrivateKey(in, NULL, NULL, "asdf");

    if (!eckey)
    {
        ERR_print_errors(bio_err);
        return 1;
    }

    copied = ECDH_compute_key(kdfout, sizeof(kdfout), EC_KEY_get0_public_key(eckey), eckey, KDF1_SHA256);

    if (copied <= 0)
    {
        ERR_print_errors(bio_err);
        return 1;
    }

    printf("%d\n", copied);
    rangedHexPrint(kdfout, copied);
    return 0;
}

Since KDF1_SHA256 produced the same answer as on Windows (DeriveKeyViaHash(HashAlgorithmName.SHA256)) clearly all the data alignment stuff is happy. DeriveKeyTls may be a bit harder, I don't seen an exported function for that offhand in OpenSSL, so it would mean either implementing the KDF or just PNSE (PNSE's certainly easier to start with).

[danmoseley]

My bad about NS2.0. These are not planend for it. Removing label.

[bartonjs]

Well, at least I put my notes somewhere easier to find because of this :)

[morganbr]

@sidshetye, you mentioned in dotnet/corefx#7023 that you were interested in ECDH. Can you please tell us more about what you'd like to do with it? We haven't heard much from users on interest in ECDH or DH and it would be helpful to better understand your usage.

[SidShetye]

@morganbr we're using ECDH's DeriveKeyMaterial to encrypt the media encryption key when encrypting a file between two entities who have their own EC certificate.

BTW, I think we had to fiddle with the CngKey in some way to key the EC keys to work (either for ECDsa or ECDH). Would be nice if the API didn't assume any particular EC Key usage since regardless of ECDSA or ECDH, the underlying key structures are same.

Are you folks exploring quantum resistant asymmetric ciphers? Anything to share ...?

[sdrapkin]

@morganbr ECDH is a crucial component of ECIES. I'm surprised MS hasn't heard much from users on interest in ECDH, given that modern TLS and practically all modern asymmetric cryptography runs on ECDH.

[Drawaes]

The reason is, most .net users use SSLStream for TLS support. As that doesn't use the .net crypto primatives but instead a native TLS implementation (SChannel, OpenSSL, SecureTransport) and they support DH and ECDH directly then most users don't need it. I guess the call for use cases is where someone would actually need it outside of TLS.

[sdrapkin]

TLS is online. There are equally as many scenarios for offline encryption. Additionally, those who can control both the Client and the Server, should not use TLS at all. Tarsnap does not use TLS, and neither do most modern messaging solutions (ex. WhatsApp) which are based on Noise-framework (which uses ECDH as a key building block).

[Drawaes]

(Incase you are mistaken I am all for it)

The issue here is MS need some real world use cases of people needing in .net today.... to push it to the top of the pile... unless I mistaken

[scalablecory]

We use ECDH in an ECIES implementation to provide stateless communication between services when the message must go through an untrusted party. This is one of the last few puzzle pieces in us getting fully on .NET Core.

[sdrapkin]

Real world use case: I cannot port Inferno to NetStandard/Core if ECDH is not available.

[karelz]

Incidentally we are just going through planning phase for next investments into crypto area. Would it help if we publish the ordered list draft in designs repo? (disclaimer: We will have to run the list and investment by our management before we can commit what we can implement when)

[SidShetye]

@karelz Yes, I think sharing the list of yes, maybe and no would help facilitate the broader discussion of where .net crypto should be heading. In the meantime, can you please comment if ECDH will make it to net standard 2.0 final? net core 2.0 preview 2 came out and very surprisingly - it's still not there. We will accordingly have to message all our customers to stay away from net standard / core 2.0 until further notice.

I really love what you guys are doing but want to say that the team needs to view ECDH/ECIES as an important foundational crypto block. There is crypto beyond TLS and today net core/standard has no asymmetric crypto to protect 256 bit entropy levels (RSA 4096 is a choke point dropping ~256bits to just ~128bits).

[karelz]

This issue is marked as Future, it will not make it into .NET Core 2.0. The .NET Standard 2.0 is frozen for couple of months.

want to say that the team needs to view ECDH/ECIES as an important foundational crypto block

How many customers are blocked? 100%, 10% or 1%? We have to look at things with wider context. There are many scenarios. Many customers with different needs. Everyone thinks that their scenario is the most important in the world :). You can also look at it the way -- if we had this, we would NOT have something else. The question is do we have useless crypto/other work in place? Also, .NET Standard is about .NET Framework, Xamarin and .NET Core commonalities. Xamarin didn't have this API, that is why it is not priority in .NET Standard itself (separate discussion from .NET Core priority of course).

[bartonjs]

@sidshetye Our telemetry (visible at https://apisof.net/catalog/System.Security.Cryptography.ECDiffieHellman) says that we've never seen anything actually use the ECDiffieHellman class (though there is epsilon usage of ECDHCng directly (https://apisof.net/catalog/System.Security.Cryptography.ECDiffieHellmanCng)).

That's not to say "no, we're not doing it", but it was part of why other things have traditionally gotten prioritized above it (DSA has 7 to 10 times as much usage, for example).

I agree that we should have it, and it's on my personal wish-list (after all, I'm the one who opened this issue, and potentially the one who would be doing the work :smile:). But we have to balance things.

As Karel said, the reason ECDH isn't in .NET Standard 2.0 is because Xamarin didn't have it (types were, for the most part, taken as the intersection of Xamarin and .NET Framework... members on those types were taken as the union of the ones that NetFX and CoreFX had (AFAIK Xamarin didn't define any new members on NetFX types)). .NET Core 2.0 doesn't have it because .NET Core 1.1 didn't have it (due to time constraints) and it wasn't part of .NET Standard 2.0 (which, along with the new macOS crypto stack, ate pretty much all of the crypto team's time).

Knowing that someone wants it is good, though. It helps justify doing it to my management for reasons other than "because NetFX has it, and it's cool."

[SidShetye]

@bartonjs @karelz I love the telemetry data but how is it being captured? I know for a fact that all workloads and public/private datacenters where we're using ECDH are typically quite locked down. I'm guessing the same might be true for other customer workloads requiring higher levels of info. assurance. Or is that telemetry off desktop apps. Our ECDH usage on servers >> desktops.

Anyway, looking forward to your team sharing the crypto roadmap with the community

[karelz]

@sidshetye we have public numbers from nuget.org and whoever used ApiPort and sent data. We have internal data from several other sources.

We also count all customers who tell us they use it in locked down environments. You are the first one to speak up.

[SidShetye]

@karelz : I missed answering your question a couple of comments above - a 100% of our Microsoft stack customers (FinTech, Healthcare, Govt) will be blocked from moving to .NET Core 2.0. I'm telling them to stay on .NET Framework. (Side note: Glad ASP.NET Core is supported on .NET Framework for that to be viable; had us worried about fragmenting the community like Python3 or Angular). We have folks on Java and Scala too but they'll be indirectly impacted as our APIs/services supporting them are on .NET Framework.

[karelz]

Crypto area investments plans are now published: https://github.com/dotnet/designs/issues/11

[sdrapkin]

Now that .NET Standard 2.0 is released, how does one translate the following net452 code to netstandard2?

var ecdh = new ECDiffieHellmanCng(privateDhmKey)
{
    HashAlgorithm = CngAlgorithm.Sha384,
    SecretAppend = [byte array],
    SecretPrepend = [byte array]
}
ecdh.DeriveKeyMaterial(publicDhmKey); // returns a byte array

Ie. how does one do a Diffie-Hellman key derivation with one private DH key and one public DH key?

[scalablecory]

@sdrapkin ECDH is not a part of .NET Standard 2.0.

[sdrapkin]

@scalablecory @bartonjs said here that ECDiffieHellmanCng will be ported. The NetStandard2 APIs are full of ECDiffieHelman* references.

Can someone/MSFT please confirm that DH shared-secret derivation over NIST P-curves is NOT supported by NetStandard2 APIs?

[Drawaes]

That roadmap is they will be ported in a future version. I believe 2 was pretty much baked in stone and on bug crushing by the time the roadmap was published.

[sdrapkin]

If ECDH shared-secret derivation is not available in NetStandard2, what is the purpose of all the ECDiffieHellman* APIs and properties?? The purpose of hash APIs is to hash. The purpose of ECDsa APIs is to sign. The purpose of ECDH APIs is to derive a shared-secret.

Cue every "You had one job..." meme here.

If ECDH derivation is still not possible in NetStandard2, the WTF is hard to process, and I doubt I would be the only one.

Can MSFT folks please weigh in?

[bartonjs]

@sdrapkin I'm not sure how ECDiffieHellmanPublicKey ended up in there (I wasn't involved in the issue or the PR, so it must have happened while I was on vacation). The other instances of "ECDiffieHellman' are for the CNG algorithm (and algorithm group) identifiers... and I'm not sure why those are in that doc, since they're not part of netstandard.dll (CNG is not part of .NET Standard).

On Windows you can create a CngKey object using the CngAlgorithm.ECDiffieHellmanP384 identifier, but no .NET Standard (or .NET Core) API exists to make use of that key.

Adding support for EC Diffie-Hellman is on the roadmap for (hopefully) the 2.1 release. As someone already pointed out, that roadmap document was describing what we're pursuing for after 2.0, not what we were hoping to squeeze in at the last minute.

[SidShetye]

@sdrapkin you're right that several folks are waiting for this fix (just see this thread). I too simply assumed something important like this would be a part of the long awaited 2.0 release. Hopefully in v2.0.0.0.0.1 ...

[henning-krause]

This is very unfortunate. We are using the Inferno library by @sdrapkin and this is blocking us from moving our application to .NET Core.

[danmoseley]

@karelz to see feedback above about blocking porting.

[karelz]

Feedback is noted - as mentioned in our security roadmap, this is not commonly used API (4-6 asks on this thread so far), but when needed, it is very important.

[ianhays]

I'm going to get started on this.

[NiclasOlofsson]

Add https://github.com/NiclasOlofsson/MiNET to the list of projects that await more security support to move to .NET core (ultimately Linux). It's sad how badly this is supported cross all .NET portal frameworks (mono, xamarin, standard, etc).

[mmaguigan]

Not to go too far back in time, but I want to address the comment made about telemetry data indicating no one has used ECDiffieHellman. The odds are, anyone who would use such a thing, would never allow telemetry. You're dealing with a part of the community that is naturally very averse to things like telemetry, and speaking on behalf of my customers - telemetry is always disabled unless it slips through by mistake. I don't know if we can rely on telemetry to guide cryptographic systems development, any data will be biased towards the sorts of systems used by less paranoid folks (eg. RSA would for example outweigh ECDSA). This is obviously very presumptuous, but I think it would have to be taken into account. Obviously the data isn't totally wrong, it does indicate that the majority use XYZ, but the other side of the inequality isn't being measured accurately and we can do nothing except guess at it's value 😞

EDIT: I see that @sidshetye, mentioned a similar comment. I'd like to mention that my customers in the government sector are affected as well.

[BravoTango86]

This is holding us back somewhat as it's making securing push notifications as per https://tools.ietf.org/html/draft-ietf-webpush-encryption-08 (https://developers.google.com/web/updates/2016/03/web-push-encryption, Firefox, etc) a pain.

[CIPop]

@ianhays @bartonjs was there any progress on this for .NET Core on Linux? /cc @amarochk @morganbr

I'm trying to implement TPMv2 support for .Net Core on Linux using Microsoft.TSS.NET and am currently missing the following:

CryptoAsym.cs(37,17): error CS0246: The type or namespace name 'ECDiffieHellmanCng' could not be found (are you missing a using directive or an assembly reference?) [/home/cristian/tss/TSS.Net/TSS.Net.Linux.csproj]
CryptoAsym.cs(38,17): error CS0246: The type or namespace name 'ECDsaCng' could not be found (are you missing a using directive or an assembly reference?) [/home/cristian/tss/TSS.Net/TSS.Net.Linux.csproj]
CryptoAsym.cs(643,23): error CS0246: The type or namespace name 'CngAlgorithm' could not be found (are you missing a using directive or an assembly reference?) [/home/cristian/tss/TSS.Net/TSS.Net.Linux.csproj]

I've tried both <TargetFramework>netstandard2.0</TargetFramework> and <TargetFramework>netcoreapp2.0</TargetFramework>.

[bartonjs]

It's on our 2.1 roadmap, but isn't ready yet.

Those types, of course, will be Windows-only (CNG), you'll want to use the abstract base classes instead.

[danmoseley]

@bartonjs estimates 2 weeks remaining

[CIPop]

With help from @amarochk from the Microsoft.TSS team we've worked around this issue for TPM on Linux. This is no longer blocking us.

[Troncek]

It's on our 2.1 roadmap, but isn't ready yet.

When you say 2.1, you mean netcoreapp2.1 or netstandard2.1?

[bartonjs]

When you say 2.1, you mean netcoreapp2.1 or netstandard2.1?

netcoreapp. But then probably also netstandard.

[Troncek]

Thanks.

Will these APIs be available also as a separate NuGet package initially? So they can be used with netstandard2.0 class libraries which can then be used by other .NET platforms beside .NET Core 2?

[karelz]

@Troncek We do not plan to ship every single API in its own nuget package / DLL - it would explode. Testing is nightmare, gazillion of dependencies is a problem in projects, etc. You can consider lightup (use the APIs when they are available on the platform) or cross-compilation for platforms you need to support. If there is strong demand from large set of customers, we can always consider alternatives. So far I see 6 votes on the top post, so it does not seem to be that impactful (compared to other APIs, bug fixes, etc.),

Also note that there is currently no timeline for next .NET Standard. It will for sure not ship with netcorepp2.1. The idea is that .NET Standard moves slowly when all platforms catch up. Adding just a few of additional APIs to it is not something we are considering at this moment.

[bartonjs]

@Troncek My understanding is that we can publish a new copy of the System.Security.Cryptography.Algorithms package which adds ECDH as an "on netstandard20" algorithm (deferring the rest of the netstandard20 types to the netstandard package). So it won't be as automagic as the things "in netstandard", but it should be available.

[Troncek]

@bartonjs

Yeah, that's what I meant.

Similar to System.Threading.Tasks.Dataflow namespace that is available with netcoreapp2.0, but not part of netstandard2.0, and can be added separately via System.Threading.Tasks.Dataflow NuGet package.

Thanks for the info.

[FrankSzendzielarz]

Working on Ethereum client implementation, I require various crypto , Keccak256 SHA-3, ECIES, not offered by .NET, and I think with the explosion of distributed ledger, the need is growing more urgent. Right now the go-to for some of that is BouncyCastle or roll-your-own. The uservoice /community feedback is probably not a reflective metric of the real need. How is the MS internal team doing BaaS without BouncyCastle? Why isn't all this stuff already in System.Security.Cryptography? I think MS will be doing itself a favour by throwing its weight behind one or the other (or both) and nailing down the direction wrt crypto on .NET

[SidShetye]

@bartonjs for an EC cert (e.g. secp-256 curve) that's in OS cert store (say OS=Windows, StoreLocation = LocalUser and StoreName = My), how does one load its private key into ECDiffieHellman in .net core 2.1 to actually use it?

[SidShetye]

@bartonjs @karelz : Any news on the comment above? We were blocked on this for our port to .net standard (or core).