cryptography: (carefully) port well established crypto libraries

[Shamar]

Given we want to replace libsec, we need a modern well established crypto library that will replace it. Since it's required by the kernel (random number generation, devtls and so on), it's going to be part of our core system.

Thus we need

1. to native port it, with the long term goal of contributing the port to the upstream community
   • to native port its dependencies, with the long term goal of contributing the port the respective upstream communities
   • to keep control on the version we use, avoiding to automatically include the "latest" version
2. to minimize our impact on code that we do not really understand
3. to be able to add our own regression tests
4. to be able to run all tests provided by the upstream libraries that apply to Harvey

To minimize the risk of introducing bugs in the code we get, we adopt a convention based, automated process:

1. create a folder in source for each third party source we import
   1. create a .gitignore containing upstream/, port/* and any further temporary item required
   2. create a port/ folder that will contains the sources actually compiled.
   3. write the uri of the sources to download to port/UPSTREAM
2. create a common.json that define Pre commands to
   1. verify the downloaded tar sha256sum against a known valid checksum
   2. untar the sources in a newly created upstream/ folder (if not already present)
3. create a build.json in that folder that
   1. fetches a specific version of the sources
   2. includes one or more specific json Projects that will do the actual port
   3. on Post write the url of the downloaded sources in port/UPSTREAM
   4. fail if git reports that port/UPSTREAM has changed

Then in each custom project we can do the automatic port (carefully using tools and semantics available on Harvey user space). Any header added to /sys/include during the process must be added to $HARVEY/.gitignore and conforms to the Harvey conventions (that is, it should not contains macro guards or #ifdef unrelated to Harvey)

This way we

• keep control on the version included (with checksum and uri verifications that will be attributed to the proper author in the repo history)
• minimize bandwidth: upstream sources are downloaded only once in compressed form
• minimize repo size: external sources are gitignored (consider we are talking about libraries that can be as large and complex as our whole OS)
• can add as many regression test as we need
• are ready to contribute back to upstream
• can easily (but visibly) update the version imported
• can easily remove our hacks when upstream community will start to support Harvey

NOTE This process is specifically designed for cryptographic tools that are going to be part of the core native system. It does not impose or propose any rule or constraint on other kind of native or glibc based port.

[mennis]

Perhaps someone who knows what they are looking at could look at s2n for suitability. It does appears to at least some of what is in libsec. https://github.com/awslabs/s2n

[Shamar]

Thanks @mennis I'll give a look. For this issue, after a little study, I did considered LibreSSL's libcrypto the best candidate. But currently this issue is obsolete.

@rminnich and @hdonnay are working to provide an alternative approach. Indeed the majoirity of the Harvey community prefers to vendor the cryptography.

[Shamar]

Actually @mennis s2n seems a pretty good piece of code. But not suitable to replace libsec, as it delegates to other libraries (OpenSSL, LibreSSL and so on) the actual cryptography.

[mennis]

Ah, ok, I misunderstood a portion of a talk I saw.

[Shamar]

Replaced by #32.