Change libraries XBPS depends on

ericonr commented 4 years ago

Moving XBPS off of LibreSSL (or even OpenSSL, depending on the resolution here) is an interesting change, for reasons of API simplicity, bootstrapping, and others. Some potential options are BearSSL and mbedtls. Custom signing and verification code might need to be added as well.

Moving off of libarchive has also been brought up, but I don't believe there have been specific suggestions for libraries. We would probably need at least xz and zstd support. As a bonus, we could find a threaded compression and decompression library.

the-maldridge commented 4 years ago

I vote BearSSL, as well as dropping libarchive. Both have deeper than desirable bootstrapping chains in the chroot- set.

sgn commented 4 years ago

I also support BearSSL. mbedtls requires python3 in hostmakedepends, which will lengthen bootstrapping chain. Re: libarchive, if we gonna remove it, we'll add tar back into bootstrap chain, since bsdtar is built together with libarchive? And we'll need at least gzip, xz, bzip2, and zstd to de-compress inside chroot.

the-maldridge commented 4 years ago

Not sure what happened there, I lost the tab, most of my message, and the ticket closed. It wasn't important.

ericonr commented 4 years ago

Would moving to separate backends for libzstd and liblzma be worth it, then? They are the libraries used by libarchive itself.

@sgn Since bsdtar isn't code we have to maintain, I think it would be ok to keep it as a bootstrap package. However, gzip, xz, bzip2 and zstd are already bootstrap packages, since they are dependencies of libarchive, so removing libarchive from bootstrap is equally possible.

ericonr commented 4 years ago

It makes for a bigger bootstrapping/dependency chain, but another option includes using https://skarnet.org/software/s6-networking together with BearSSL. Some of the code, like for cert directories, will have to be present in XBPS, and I don't think depending on another library for some of it would be too bad (I might not have thought of all the potential issues).

Otherwise, it has some interesting code we can duplicate for interacting with things like certs.

pullmoll commented 4 years ago

If we don't have bsdtar in the base chroot, we will have to touch quite a few templates again to either add bsdtar to hostdepends or switch (back) to tar and add that, and fix the common/ scripts which handle archives. That's why I don't like the idea to remove libarchive from the bootstrap packages too much, if at all.

ericonr commented 4 years ago

That's why I don't like the idea to remove libarchive from the bootstrap packages too much, if at all.

I don't think it needs to be. It's a small dependency that sits above all the other archive libraries, that we'd need anyway if we were to switch to GNU tar. bsdtar as a one-size-fits-all tool is pretty good.

One reason we'd switch from libarchive for XBPS specifically would be to enable better access to library functionality, such as threaded compression and decompression, and, ideally, cleaner memory handling, once the libarchive layer is removed.

ericonr commented 4 years ago

I'm facing this problem in #324: https://github.com/earlephilhower/bearssl-esp8266/commit/0c27e4145af5886156c4dfb59292ffbbd041e6c0

BearSSL apparently can't natively decode an RSA key as stored in a PEM file. Using a program like below:

#include <stdio.h>
#include <sys/mman.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>

#include <bearssl.h>

void dummy(void *dest_ctx, const void *src, size_t len)
{
    br_x509_decoder_context *dest = dest_ctx;
    br_x509_decoder_push(dest, src, len);
}

int main(int argc, char **argv)
{
    if (argc < 2) {
        fputs("too few arguments", stderr);
        return 1;
    }

    int key_file = open(argv[1], O_RDONLY);
    if (key_file < 0) {
        fprintf(stderr, "error opening %s: %m\n", argv[1]);
        return 1;
    }
    unsigned char *buffer = mmap(NULL, 800, PROT_READ, MAP_PRIVATE, key_file, 0);
    if(buffer == NULL) {
        fputs("mmap error", stderr);
        return 1;
    }
    close(key_file);

    br_pem_decoder_context pem;
    br_x509_decoder_context x509;
    br_pem_decoder_init(&pem);
    br_x509_decoder_init(&x509, 0, 0);
    br_pem_decoder_setdest(&pem, dummy, &x509);
    size_t len = 800;
    while (len > 0) {
        size_t pushed = br_pem_decoder_push(&pem, buffer, len);
        printf("pushed %lu bytes\n", pushed);
        buffer += pushed;
        len -= pushed;

        switch(br_pem_decoder_event(&pem)) {
            const char *name;

            case 0:
                break;
            case BR_PEM_BEGIN_OBJ:
                name = br_pem_decoder_name(&pem);
                puts(name);

                break;
            case BR_PEM_ERROR:
                fputs("br_pem_error!", stderr);
                break;
            case BR_PEM_END_OBJ:
                if (len != 0) {
                    fputs("didn't finish file", stderr);
                } else {
                    fputs("finished file!", stderr);
                }
                break;
        }
    }

    fprintf(stderr, "err code: %d\n", br_x509_decoder_last_error(&x509));
}

on the following key, which is what LibreSSL decodes for us in the rsa_verify_hash function in lib/verifysig.c:

yields an error code of 36: https://www.bearssl.org/apidoc/bearssl__x509_8h.html#a5a1ef832c53c92ed31ea4ced41d830a1

ericonr commented 4 years ago

Also, BearSSL apparently doesn't do TSL 1.3, according to the front page.

ArsenArsen commented 4 years ago

Current version is 0.6. It is now considered beta-quality software: it successfully passes extensive test suites, and while not all intended features are present, new features should imply no breaking changes in API or ABI1. There is no such thing as bug-free code, and I won’t claim that there is none in BearSSL; only that I looked real hard. Use in production applications is, in any case, at your own risk.

I don't support moving to BearSSL. It might be better code, it might be a better API, but when it comes to an SSL library, it's more important that it's hard to break, this leaves two primary options:

OpenSSL: lots of eyes on it, especially after Heartbleed
LibreSSL: quite a competent set of people behind it, but it requires more patching, I've put it here also because it was already there.

aurieh commented 4 years ago

I have to agree with @ArsenArsen here. XBPS, being a critical system component, should not rely on a relatively little-known and beta-quality (according to its front page) TLS/crypto library.

CameronNemo commented 4 years ago

@sgn

mbedtls requires python3 in hostmakedepends, which will lengthen bootstrapping chain.

It only really needs GNU Make and a C99 toolchain to get a functional library. Python and Perl are required for generating and running tests, but it is fairly straightforward to disable them.

See here: https://github.com/void-linux/void-packages/pull/25033

ArsenArsen commented 4 years ago

We are planning on using xbps on a system we are working on, and I was looking into compatibility between Open- and LibreSSL in various software packages, and OpenSSL (unsurprisingly) requires a lot less patching, so I looked into whether xbps compiles with OpenSSL 1.1.1g, and bfad1afff3f938b0cf8f401e7346761c331dceb4 (./configure --enable-rpath --enable-fulldebug --enable-debug) compiles with no changes needed.
In my opinion, the best course of action, assuming Void switches to OpenSSL in its entirety, is to just switch to OpenSSL (or supporting both open- and libressl) for xbps.

I am not sure whether the benefits of OpenBSDs rigorous development process outweigh the downsides of it not being OpenSSL itself anymore.

EDIT: reconfigured and rebuilt xbps on master to run the tests, got these results:

===> Expected failures
libxbps/shell/obsoletefiles_test:multiple_obsoletes_with_alternatives_unordered  ->  expected_failure: not fixed yet: 1 != 0 (1 != 0)  [0.178s]
xbps-alternatives/main_test:cc_alternatives_removal  ->  expected_failure: https://github.com/void-linux/xbps/pull/253: 1 != 0 (1 != 0)  [0.164s]
xbps-alternatives/main_test:replace_file_with_alternative  ->  expected_failure: https://github.com/void-linux/xbps/pull/185: 1 != 0 (1 != 0)  [0.175s]
xbps-install/behaviour_tests:update_unpacked  ->  expected_failure: currently just configures the package: 'configure' != 'update'  [0.104s]
===> Failed tests
libxbps/shell/obsoletefiles_test:replace_package_same_files  ->  failed: 2 != 0 (2 != 0)  [0.154s]
xbps-install/behaviour_tests:reproducible  ->  failed: 2 != 0 (2 != 0)  [0.084s]
===> Summary
Results read from /home/arsen/xbps/result.db
Test cases: 244 total, 0 skipped, 4 expected failures, 0 broken, 2 failed
Start time: 2020-11-19T19:29:02.717288Z
End time:   2020-11-19T19:29:06.068760Z
Total time: 29.407s

void-linux / xbps

Change libraries XBPS depends on #323