[redbean] Expose more of mbedtls to Lua

mterron commented 5 months ago

I've recently found myself re-implementing quite a lot of mbedtls functionality within Lua. Is it possible to expose more mbedtls functions/operations (create keypairs, create and validate signatures, parse "pem" files, etc) to the Lua environment?

From a high level perspective, some "programs" in mbedtls (gen_key.c, keyapp.c, pk*.c, ecdsa.c and ecdh_curve25519.c) could be useful to expose as Lua functions.

pkulchenko commented 5 months ago

I'd say it's definitely possible and could be quite useful. I've been looking into this (see #966 and maybe #816), but it would be good to figure out which functions would need to be exposed, as there is a large number of them.

I'd like to be able to do a key exchange to support let's encrypt integration without resorting to running openssl. Something along the lines of https://github.com/alexpeattie/letsencrypt-fromscratch, but with mbedtls functions called from Lua instead of openssl commands. If you have the correct sequence (and may be C code or pointers to the mbedtls samples), I can probably integrate them into redbean by adding crypto.* namespace.

Couple of pointers that may be useful in that regard: https://github.com/srinskit/Crypt/blob/master/Crypt.cpp (C++ wrapper for mbedTLS) and https://mkottman.github.io/luacrypto/manual.html (for the API, although it's openssl-based; maybe crypt.h is enough).

mterron commented 5 months ago

The link I added to my original messages links to the C source code for a couple of useful mbedtls based programs. I can see each of those programs becoming a Lua function at a higher level of abstraction than mbedtls primitives.

mterron commented 5 months ago

I think the basics are:

Generate keypair (RSA, EC)
Encrypt and decrypt
Sign and verify signature

That'll cover the majority of the cases and give you a solid base to build on.

I'm not really concerned with the api at this stage, as long as those operations are exposed.

On Thu, 4 Apr 2024, 2:30 pm Paul Kulchenko, @.***> wrote:

I'd say it's definitely possible and could be quite useful. I've been looking into this (see #966 https://github.com/jart/cosmopolitan/issues/966 and maybe #816 https://github.com/jart/cosmopolitan/issues/816), but it would be good to figure out which functions would need to be exposed, as there is a large number of them.

I'd like to be able to do a key exchange to support let's encrypt integration without resorting to running openssl. Something along the lines of https://github.com/alexpeattie/letsencrypt-fromscratch, but with mbedtls functions called from Lua instead of openssl commands. If you have the correct sequence (and may be C code or pointers to the mbedtls samples), I can probably integrate them into redbean by adding crypto.* namespace.

Couple of pointers that may be useful in that regard: https://github.com/srinskit/Crypt/blob/master/Crypt.cpp (C++ wrapper for mbedTLS) and https://mkottman.github.io/luacrypto/manual.html (for the API, although it's openssl-based; maybe crypt.h https://github.com/srinskit/Crypt/blob/master/Crypt.h#L58-L110 is enough).

— Reply to this email directly, view it on GitHub https://github.com/jart/cosmopolitan/issues/1136#issuecomment-2035928251, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABB2XW43ANH65FK3JBHCLNDY3SUKNAVCNFSM6AAAAABFWHUWWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMZVHEZDQMRVGE . You are receiving this because you authored the thread.Message ID: @.***>

mterron commented 5 months ago

I found some of the stuff in redbean already, just not surfaces to the Lua system: https://github.com/jart/cosmopolitan/blob/cf9a1f7f339a9a0b680187595584be1c48bee296/tool/net/redbean.c#L2005 https://github.com/jart/cosmopolitan/blob/cf9a1f7f339a9a0b680187595584be1c48bee296/tool/net/redbean.c#L1989

mterron commented 5 months ago

@jart @pkulchenko what's the minimum functionality you expect from a crypto API based on mbedTLS? In my comment, I listed the 5 basic operations I can think of, is that sufficient or do you want to see something more comprehensive?

mterron commented 5 months ago

I took the liberty to look at LuaCrypto and try to trim to a proposal that seems doable (and add references to the mbedTLS implementation of each function where applicable). This is pretty much a copy-paste of LuaCrypto reference page, all credit to the authors.

The API tries to hide the mbedTLS internals, so in most cases it is not simply a pass-through to the existing mbedTLS API

Reference

Parameters

digest

A string naming the hashing algorithm to use for a digest operation. The list of supported algorithms may change with each version of the mbedTLS library.

The supported algorithms are:

MD5
SHA1
SHA224
SHA256
SHA384
SHA512
BLAKE2B256

The list of supported hashing algorithms can also be retrieved by using the crypto.list("digests").

keyType

A string representing public/private key type. Can be "rsa", "dsa", "Ed25519" or "ecdsa".

cipher

This parameter is a string naming the cipher algorithm used by encryption and decryption.

The list of supported hashing algorithms can also be retrieved by using the crypto.list("ciphers").

key

The string key/password used for encryption/decryption.

iv

An optional string initialization vector for encryption/decryption.

pad (Not sure this is needed, we could just hide it away at the Lua layer?)

An optional boolean flag whether padding should be used. The default is true, which means that input of any size can be provided. Returned date may be larger than input string due to the padding. If explicitly set to false, the padding is turned off and the input data size has to be multiple of block length.

Error handling

The functions throw an error when known invalid parameters are passed, such as non-existent digest/cipher and too large key or initialization vector. Otherwise, the functions return nil, error in case of runtime errors, such as incorrect input size when padding is enabled.

Message Digest (hash) - `crypto.hash`

Functions used to calculate cryptographic hashes of strings. Supported digest algorithms are returned by crypto.list("digests").

crypto.hash(digest, string [,key]) → string

This function generates the message digest of the input string and returns it. The hashing algorithm to use is specified by digest.

The existing functions Md5, Sha1, Sha224, Sha256, Sha384 and Sha512 are specialised versions of this function.

Encryption, decryption - `crypto.encrypt`, `crypto.decrypt`

A high-level API for encryption and decryption. Supported ciphers can be detected by calling crypto.list("ciphers").

crypto.encrypt(cipher, input, key [, iv[, pad]]) → string

This function encrypts the input string and returns the result. The encryption algorithm to use is specified by cipher. Encryption key is specified by the key parameter and is required. The optional iv parameter specifies an optional initialization vector. If boolean pad is specified after iv, it determines whether padding will be used (on by default).
See pk_encrypt.c

crypto.decrypt(cipher, input, key [, iv[, pad]]) → string

This function decrypts the the input string and returns the result. The decryption algorithm to use is specified by cipher. Decryption key is specified by the key parameter and is required. The optional iv parameter specifies an optional initialization vector. If boolean pad is specified after iv, it determines whether padding will be used (on by default).
See pk_decrypt.c

Keys - `crypto.key`

Functions to work with public and private keys.

crypto.key.generate(keyType, len) → table

Generates a new keyType ("rsa", "dsa", "ed25519", "ec") public/private key pair object of length len bits.
See gen_key.c

crypto.key.from_pem(pem) → table

Reads a key from PEM string pem.
See pem.c

crypto.key.to_pem(key) → string

Generates a PEM string representation of the key.
See pkwrite.c

Signing, verifying - `crypto.sign`, `crypto.verify`

A high-level interface to digital signatures. A digest algorithm is used to calculate a hash of the data, which is then signed using a private key into a signature. The signature can be used to verify a message using a public key.

crypto.sign(digest, string, privkey) → string

This function generates the message digest of the input string, signs it using the private key privkey and returns it as a raw binary string. The hashing algorithm to use is specified by digest.
See pk_sign.c

crypto.verify(digest, string, sig, pubkey) → boolean

This function generates the message digest of the input string using digest digest, and verifies it against signature sig using public key pubkey. Returns true if the message was verified correctly, false otherwise.
See pk_verify.c

HMAC - `crypto.hmac`

crypto.hmac.digest(digest, string, key) → string

This function returns the HMAC of the string. The hashing algorithm to use is specified by digest. The value provided in key will be used as the seed for the HMAC generation.

This is the same function as GetCryptoHash().

Misc functions - `crypto`

crypto.list(type) → table

Returns a Lua table array of supported digests and ciphers (strings), depending on the type argument:

"ciphers": returns list of ciphers supported by crypto.encrypt and crypto.decrypt
"digests": returns list of digests supported by crypto.digest and crypto.hmac.digest

jart commented 5 months ago

Have you seen our new Curve25519 API?

>: secret1 = "\1"
>: secret2 = "\2"
>: public1 = Curve25519(secret1, "\9")
>: public2 = Curve25519(secret2, "\9")
>: Curve25519(secret1, public2)
"\x93\xfe\xa2\xa7\xc1\xae\xb6,\xfddR\xff...
>: Curve25519(secret2, public1)
"\x93\xfe\xa2\xa7\xc1\xae\xb6,\xfddR\xff...

So now we have GetCryptoHash, Curve25519. and GetRandomBytes. What else do we need for a full cryptography stack other than a block cipher?

mterron commented 5 months ago

RSA is unavoidable to interoperate with basically anything else. The EC algorithms are nice too, you can build Ed25519 on top of Curve25519 but for interoperability you want the NIST curves too (P-256 and friends). The other thing with my proposed API is that it mirrors LuaCrypto to an extent that makes it easy to consume (namespaced, etc) as I saw some comment from Justine around having a more explicit and composable API vs point solutions (https://github.com/jart/cosmopolitan/pull/1122).

You are right around a block cipher, this aims to be a workable first step, not a 100% complete solution.

All that functionality is built into mbedTLS, what's missing is the Lua binding.

jart commented 5 months ago

MbedTLS has 16398 lines of header files. That's an enormous API surface area, considering it has only 78686 lines of source code. It's not a reasonable thing to ask that redbean wrap the MbedTLS API, because that's asking for literally thousands of interfaces. If you can clarify which specific algorithms you need, then I can write simple Lua APIs exporting them.

mterron commented 5 months ago

Maybe I'm not explaining myself well (English is not my native language).

In the api proposal document I linked to every mbedtls code that implements the proposed functions (all the See links) up to the specific line in some cases. Never suggested that all of mbedtls be exposed, I think that'd be a terrible idea actually.

Is that not what you mean with "clarify the algorithms"?

On Fri, 19 Apr 2024, 4:06 pm Justine Tunney, @.***> wrote:

MbedTLS has 16398 lines of header files. That's an enormous API surface area, considering it has only 78686 lines of source code. It's not a reasonable thing to ask that redbean wrap the MbedTLS API, because that's asking for literally thousands of interfaces. If you can clarify which specific algorithms you need, then I can write simple Lua APIs exporting them.

— Reply to this email directly, view it on GitHub https://github.com/jart/cosmopolitan/issues/1136#issuecomment-2065712865, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABB2XWZRZJRVEFGJESWD5PTY6CJ3FAVCNFSM6AAAAABFWHUWWCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANRVG4YTEOBWGU . You are receiving this because you authored the thread.Message ID: @.***>

jart / cosmopolitan