search

genodelabs / genode

Genode OS Framework
https://genode.org/
Other
1.05k stars 248 forks source link

add openssl3 port #4992

Open trimpim opened 10 months ago

trimpim commented 10 months ago

As openssl version 1.1.1 reaches EOL on September 11. 2023 [1] , we do want to provide an update to a more recent version (3.1.1). See also my posting on the mailing list [2].

The original plan was to provide it as a separate port (openssl3) in parallel to the current one. That way every user of openssl would be able to choose when to update to the new version.

During the creation of a port for the TPM2-openssl [3] provider we discovered, that the original plan might not be realized so easy. The other dependencies (libcurl, libssh) of this component also depend on libcrypto/libssl. Which means, that we also need to update these and their transitive dependencies. This will probably be the case for most components that use openssl. In the end there would be a lot of components/libraries that change.

One way that we see to reduce the number of affected components is to just update the openssl port to use the new version. There will still be some components affected, as with the default -Werror flag the deprecated warnings of the new library will cause compilation errors.

We would be willing in adapt all components/libraries that depend on openssl. We know that at least the following repositories are affected by this:

Maybe some one sees an other solution, which we weren't able to see yet.

[1] https://www.openssl.org/blog/blog/2023/03/28/1.1.1-EOL [2] https://lists.genode.org/pipermail/users/2023-July/008777.html [3] https://github.com/tpm2-software/tpm2-openssl

trimpim commented 10 months ago

The commits above are my current state of work, which still uses the separate openssl3 port.

chelmuth commented 10 months ago

Thanks for creating this issue, @trimpim , despite my unresponsiveness via the ML - sorry for that!

Do I understand correctly that the version of openssl3 you ported is still API compatible with 1.1.1 and, thus, components not adapted yet just suffer from the deprecation warnings? From this background I'd much appreciate your work for review and merge. The transitive dependency issues remind me of my work in context of #3773. Back then I also started with an parallel openssl11 library but finally merged it into the regular library.

Please nudge me when you finished the mentioned adaption of components already in the source tree.

trimpim commented 10 months ago

This is still in testing.

The following tests work on x86_64:

The following run scripts fail with a page fault during initialization of vfs_tresor_trust_anchor

The page fault happens in gems/src/lib/vfs/tresor_trust_anchor/vfs.cc on line 1011.

SHA256((unsigned char const *)src.start, src.num_bytes,
            (unsigned char *)_passphrase_hash_buffer.base);

This could be, due to the new implementation of openssl causing a libc sleep in background. If I create a small posix application (libports/src/test/openssl/sha256) this call works.

Next I will run the tests on arm_v8a and x86_32 (qemu).

chelmuth commented 10 months ago

The page fault happens in gems/src/lib/vfs/tresor_trust_anchor/vfs.cc on line 1011.

SHA256((unsigned char const *)src.start, src.num_bytes,
            (unsigned char *)_passphrase_hash_buffer.base);

This could be, due to the new implementation of openssl causing a libc sleep in background. If I create a small posix application (libports/src/test/openssl/sha256) this call works.

This reads like something is misconfigured, or not? Why should SHA256 need sleep? Do you have a backtrace that ends up in libc sleep?

trimpim commented 10 months ago

@chelmuth you are ritght. I was wrong in assuming that is has something to do sleep().

Still it seems to be libc related. If I change test/openssl/sha256 to a libc component not using posix I get the error below:

[init -> test-openssl -> test-sha256_test] Error: libc suspend() called from non-user context (0x10f5c08d) - aborting
[init -> test-openssl] Warning: child "test-sha256_test" exited with exit value 1
[init] child "test-openssl" exited with exit value 1

(surrounding the call with Libc::with_libc() doesn't help (see 3ad57534fc3ed285ca8988cbbed0c2c3c27ea449 for the changed test).

It still could be, that the page fault happens because something is miss configured.

chelmuth commented 10 months ago

Your branch works for me with the following simple patch applied.

+++ b/repos/libports/src/test/openssl/sha256_test/test.cc
@@ -12,7 +12,7 @@
  */

 /* Genode includes */
-#include <libc/args.h>
+//#include <libc/args.h>
 #include <libc/component.h>

 /* openssl includes */
@@ -23,7 +23,7 @@
 #include <stdio.h>

-void Libc::Component::construct(Libc::Env &env)
+void Libc::Component::construct(Libc::Env &)
 {
    unsigned char const buf_in[]    { "This is a SHA256 test" };
    unsigned char       buf_out[32] { };
@@ -32,7 +32,6 @@ void Libc::Component::construct(Libc::Env &env)

    Libc::with_libc([&] () {
        result = SHA256(buf_in, sizeof(buf_in), buf_out);
-   });

    if (!result) {
        printf("nok sha256_test failed\n");
@@ -40,4 +39,5 @@ void Libc::Component::construct(Libc::Env &env)
    }

    printf("ok sha256_test\n");
+   });
 }

Please always make sure to wrap all (potentially blocking) calls of libc into with_libc().

trimpim commented 10 months ago

@chelmuth for me it also works, when I replace the printf() calls with Genode::log().

If I now understand it correctly, the problem boils down to, if it is possible to use Libc::with_libc in a vfs-plugin.

I'm by no means an expert on VFS related stuff and will probably need some help to solve this.

nfeske commented 10 months ago

[...] if it is possible to use Libc::with_libc in a vfs-plugin.

Since the VFS is logically part of the libc kernel, it must never call any I/O-related libc function.

chelmuth commented 10 months ago

Since the VFS is logically part of the libc kernel, it must never call any I/O-related libc function.

Please let me emphasize that this includes write(1,...) and write(2,...) resp. fprintf() to stdout and stderr.

trimpim commented 9 months ago

@chelmuth am I right in assuming, that in vfs plugins one shouldn't use malloc() from libc? This is what happens in the new version of openssl (see back-trace gathered using monitor below):

The old version of openssl implemented sha256() in a way that didn't use any allocation. The same will probably also apply to the other openssl functions that are used by tresor_trust_anchor.

What do you think would be a good way to continue with this?

Thread 2 "ep" received signal SIGSEGV, Segmentation fault.
Genode::Mutex::acquire (this=this@entry=0x2a8) at /data/genode/repos/base/src/lib/base/mutex.cc:21
21      if (_lock.lock_owner(myself))
(gdb) bt
#0  Genode::Mutex::acquire (this=this@entry=0x2a8) at /data/genode/repos/base/src/lib/base/mutex.cc:21
#1  0x0000000010e84c7e in Genode::Mutex::Guard::Guard (mutex=..., this=<synthetic pointer>) at /data/genode/repos/base/include/base/mutex.h:62
#2  Libc::Malloc::alloc (align=16, size=4, this=0x0) at /data/genode/repos/libports/src/lib/libc/malloc.cc:162
#3  malloc (size=4) at /data/genode/repos/libports/src/lib/libc/malloc.cc:245
#4  0x00000000012f1e5d in CRYPTO_zalloc (num=num@entry=4,
    file=file@entry=0x14aa390 "/data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/threads_none.c",
    line=line@entry=24) at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/mem.c:197
#5  0x000000000138dc4d in CRYPTO_THREAD_lock_new ()
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/threads_none.c:24
#6  0x00000000011e79aa in context_init (ctx=ctx@entry=0x15be5e0 <default_context_int>)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:79
#7  0x00000000011e7cc5 in default_context_do_init ()
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:342
#8  default_context_do_init_ossl_ () at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:337
#9  0x000000000138dcec in CRYPTO_THREAD_run_once (once=once@entry=0x15be5cc <default_context_init>, init=init@entry=0x11e7c60 <default_context_do_init_ossl_>)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/threads_none.c:70
#10 0x00000000011e80f3 in get_thread_default_context ()
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:365
#11 get_default_context () at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:373
#12 ossl_lib_ctx_get_concrete (ctx=0x0) at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:489
#13 ossl_lib_ctx_get_concrete (ctx=<optimized out>)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:485
#14 0x00000000011e856f in ossl_lib_ctx_get_data (ctx=<optimized out>, index=index@entry=0)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/context.c:516
#15 0x00000000012bddd5 in get_evp_method_store (libctx=<optimized out>)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/evp/evp_fetch.c:63
#16 inner_evp_generic_fetch (methdata=methdata@entry=0x401fe960, prov=<optimized out>, prov@entry=0x0, operation_id=operation_id@entry=1,
    name=name@entry=0x1498900 "SHA256", properties=properties@entry=0x0, new_method=new_method@entry=0x12a7000 <evp_md_from_algorithm>,
    up_ref_method=0x12a5ae0 <evp_md_up_ref>, free_method=0x12a6fd0 <evp_md_free>)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/evp/evp_fetch.c:249
#17 0x00000000012be3bd in evp_generic_fetch (libctx=libctx@entry=0x0, operation_id=operation_id@entry=1, name=name@entry=0x1498900 "SHA256",
    properties=properties@entry=0x0, new_method=new_method@entry=0x12a7000 <evp_md_from_algorithm>,
    up_ref_method=up_ref_method@entry=0x12a5ae0 <evp_md_up_ref>, free_method=0x12a6fd0 <evp_md_free>)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/evp/evp_fetch.c:364
#18 0x00000000012a5a5e in EVP_MD_fetch (ctx=ctx@entry=0x0, algorithm=algorithm@entry=0x1498900 "SHA256", properties=properties@entry=0x0)
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/evp/digest.c:1097
#19 0x00000000012a6f5f in EVP_Q_digest (libctx=libctx@entry=0x0, name=name@entry=0x1498900 "SHA256", propq=propq@entry=0x0, data=0x1a0c40, datalen=7,
    md=md@entry=0x1d1e8 "", mdlen=0x0) at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/evp/digest.c:692
#20 0x0000000001379ad1 in SHA256 (d=<optimized out>, n=<optimized out>, md=0x1d1e8 "")
    at /data/genode/contrib/openssl-2d094f9e741c560214b55a3b2ab7f69a9d3a4c88/src/lib/openssl/crypto/sha/sha1_one.c:62
#21 0x0000000010f38f8e in Trust_anchor::queue_initialize (src=..., this=0x1cd88) at /data/genode/repos/gems/src/lib/vfs/tresor_trust_anchor/vfs.cc:1011
#22 Trust_anchor::queue_initialize (src=..., this=0x1cd88) at /data/genode/repos/gems/src/lib/vfs/tresor_trust_anchor/vfs.cc:1001
#23 Vfs_tresor_trust_anchor::Initialize_file_system::Initialize_handle::write (out_count=@0x401feae0: 0, src=..., this=0x29180)
    at /data/genode/repos/gems/src/lib/vfs/tresor_trust_anchor/vfs.cc:1854
#24 Vfs_tresor_trust_anchor::Initialize_file_system::Initialize_handle::write (this=0x29180, src=..., out_count=@0x401feae0: 0)
    at /data/genode/repos/gems/src/lib/vfs/tresor_trust_anchor/vfs.cc:1845
#25 0x000000000100bb4a in Vfs_server::Io_node::_execute_write (write_pos=<optimized out>, src=..., this=0x28e60)
    at /data/genode/repos/os/src/server/vfs/node.h:412
#26 Vfs_server::File::execute_job (this=0x28e60) at /data/genode/repos/os/src/server/vfs/node.h:809
#27 0x000000000100c980 in Vfs_server::Session_component::_execute_jobs()::{lambda(Vfs_server::Node&)#1}::operator()(Vfs_server::Node&) const (node=...,
    __closure=<optimized out>) at /data/genode/repos/os/src/server/vfs/main.cc:255
#28 Genode::Fifo<Vfs_server::Node>::dequeue<Vfs_server::Session_component::_execute_jobs()::{lambda(Vfs_server::Node&)#1}>(Vfs_server::Session_component::_exec
ute_jobs()::{lambda(Vfs_server::Node&)#1} const&) (func=..., this=0x22368) at /data/genode/repos/base/include/util/fifo.h:168
#29 Genode::Fifo<Vfs_server::Node>::dequeue_all<Vfs_server::Session_component::_execute_jobs()::{lambda(Vfs_server::Node&)#1}>(Vfs_server::Session_component::_execute_jobs()::{lambda(Vfs_server::Node&)#1} const&) (func=..., this=0x22368) at /data/genode/repos/base/include/util/fifo.h:182
#30 Vfs_server::Session_component::_execute_jobs (this=0x20018) at /data/genode/repos/os/src/server/vfs/main.cc:252
#31 Vfs_server::Session_component::process_packets (this=this@entry=0x20018) at /data/genode/repos/os/src/server/vfs/main.cc:330
#32 0x000000000100d1b5 in Vfs_server::Session_component::_handle_packet_stream (this=0x20018) at /data/genode/repos/os/src/server/vfs/main.cc:367
#33 0x0000000000096698 in Genode::Entrypoint::Signal_proxy_component::signal (
    this=this@entry=0x13d7d8 <Genode::bootstrap_component(Genode::Platform&)::startup+3352>) at /data/genode/repos/base/src/lib/base/entrypoint.cc:61
#34 0x000000000009504f in Genode::Meta::call_member<Genode::Meta::Empty, Genode::Entrypoint::Signal_proxy_component, Genode::Meta::Empty> (
    func=(void (Genode::Entrypoint::Signal_proxy_component::*)(Genode::Entrypoint::Signal_proxy_component * const)) 0x96600 <Genode::Entrypoint::Signal_proxy_c
omponent::signal()>, server=...) at /data/genode/repos/base/include/util/meta.h:462
#35 Genode::Entrypoint::Signal_proxy::Rpc_signal::serve<Genode::Entrypoint::Signal_proxy_component, Genode::Meta::Empty> (args=<synthetic pointer>...,
    server=...) at /data/genode/repos/base/include/base/entrypoint.h:54
#36 Genode::Rpc_dispatcher<Genode::Entrypoint::Signal_proxy, Genode::Entrypoint::Signal_proxy_component>::_do_serve<Genode::Entrypoint::Signal_proxy::Rpc_signal> (args=<synthetic pointer>..., this=0x13d860 <Genode::bootstrap_component(Genode::Platform&)::startup+3488>)
    at /data/genode/repos/base/include/base/rpc_server.h:147
#37 Genode::Rpc_dispatcher<Genode::Entrypoint::Signal_proxy, Genode::Entrypoint::Signal_proxy_component>::_do_dispatch<Genode::Meta::Type_list<Genode::Entrypoint::Signal_proxy::Rpc_signal> > (in=..., out=..., opcode=..., this=0x13d860 <Genode::bootstrap_component(Genode::Platform&)::startup+3488>)
    at /data/genode/repos/base/include/base/rpc_server.h:181
#38 Genode::Rpc_dispatcher<Genode::Entrypoint::Signal_proxy, Genode::Entrypoint::Signal_proxy_component>::dispatch (out=..., in=..., opcode=...,
    this=0x13d860 <Genode::bootstrap_component(Genode::Platform&)::startup+3488>) at /data/genode/repos/base/include/base/rpc_server.h:235
#39 Genode::Rpc_object<Genode::Entrypoint::Signal_proxy, Genode::Entrypoint::Signal_proxy_component>::dispatch (
    this=0x13d7d8 <Genode::bootstrap_component(Genode::Platform&)::startup+3352>, opcode=..., in=..., out=...)
    at /data/genode/repos/base/include/base/rpc_server.h:273
#40 0x0000000000080dd5 in operator() (obj=0x13d7d8 <Genode::bootstrap_component(Genode::Platform&)::startup+3352>, __closure=<synthetic pointer>)
    at /data/genode/repos/base-nova/src/lib/base/rpc_entrypoint.cc:169
#41 Genode::Object_pool<Genode::Rpc_object_base>::apply<Genode::Rpc_entrypoint::_activation_entry()::<lambda(Genode::Rpc_object_base*)> > (func=...,
    capid=<optimized out>, this=0x13cd00 <Genode::bootstrap_component(Genode::Platform&)::startup+576>)
    at /data/genode/repos/base/include/base/object_pool.h:147
#42 Genode::Rpc_entrypoint::_activation_entry () at /data/genode/repos/base-nova/src/lib/base/rpc_entrypoint.cc:171
#43 0x0000000000000000 in ?? ()
chelmuth commented 9 months ago

That looks like a nice Kuddelmuddel and sinister initialization mess. As the static Malloc *mallocator seems to be NULL in malloc.cc, the libc appears to be not yet initialized. From my perspective, this is to be expected as the VFS server isn't a Libc::Component. Thus, the libc can't hook into Component::construct and call the appropriate initializers. In the end, this boils down to the following.

  1. Usage of libc in VFS plugins is quite limited (especially no I/O and no malloc).
  2. Quite some third-party libraries that depend on libc cannot be used in plugins.
  3. Rule 2 seems to apply to OpenSSL 3 libcrypto, which is pretty sad. Isn't SHA256 just maths?
trimpim commented 9 months ago

Looking at the old implementation it is definitively only maths for SHA256. The other openssl functions that I have seen are aes256 related.

A quick grep for C++ implementations for this returned the following.

If you would be okay with it, I could check if they fit the needs of vfs_tresor_trust_anchor.

chelmuth commented 9 months ago

Before talking about alternatives I'd like to hear @m-stein's opinion on this matter. Note, we currently have a (dated) port of libsodium in genode-world that seems to limit libc dependency more effectively and implements SHA256 among other modern algorithms like BLAKE2 ans Salsa20/ChaCha.

chelmuth commented 9 months ago

After some offline discussion, I consider the creation of a simple crypto suite (static library) from libcrypto sources of the OpenSSL port as a tempting option. The library would only provide AES and SHA in modes required by our native components (e.g., tresor) via a traditional low-level API. We need some time to investigate this option, though.

gstrauss commented 9 months ago

For porting to openssl 3.x: You can check out https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master/src/sys-crypto-md.h#L584 to see how lighttpd defines some inline functions to reimplement SHA256_CTX, SHA256_Init(), SHA256_Update(), SHA256_Final() for openssl 3.x. (You can combine them to create SHA256())

As an option separate from openssl 3.x, have a look at Nettle, which implements low-level crypto algorithms and does not allocate memory. https://github.com/gnutls/nettle https://github.com/gnutls/nettle/blob/master/README http://www.lysator.liu.se/~nisse/nettle/

chelmuth commented 9 months ago

For porting to openssl 3.x: You can check out https://git.lighttpd.net/lighttpd/lighttpd1.4/src/branch/master/src/sys-crypto-md.h#L584 to see how lighttpd defines some inline functions to reimplement SHA256_CTX, SHA256_Init(), SHA256_Update(), SHA256_Final() for openssl 3.x. (You can combine them to create SHA256())

Thanks for the pointer. Unfortunately, this approach did not help as SHA256_Init() uses EVP_MD_CTX_new() which uses OPENSSL_zalloc()...