search

edgelesssys / edgelessdb

EdgelessDB is a MySQL-compatible database for confidential computing. It runs entirely inside a secure enclave and comes with advanced features for collaboration, recovery, and access control.
https://edgeless.systems/products/edgelessdb
GNU General Public License v2.0
170 stars 17 forks source link

no performance_schema feature? #116

Closed water5-cmd closed 1 year ago

water5-cmd commented 2 years ago

Hi, @thomasten,

I found that edgelssdb does not have performance_schema and when I try to use the performance_schema variable, the following error is reported during the database startup:

******
2022-10-10  9:42:57 0 [ERROR] edb: unknown variable 'performance_schema=ON'
******

I learn performance_schema from Performance Schema Overview, but there is no information about this variable in edgelessdb and the logs are as follows.

root@daier:/home/daier# mysql -P3307 -h127.0.0.1 -utest -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 123
Server version: 5.5.5-10.5.11-MariaDB-debug-log Source distribution

Copyright (c) 2000, 2022, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> SHOW VARIABLES LIKE 'performance_schema';
Empty set (0.00 sec)

mysql>

So, edgelessdb doesn't implement and use the performance_schema feature? Is there any way to set the value of performance_schema to OFF?

Nirusu commented 2 years ago

Hi @lijun0314,

Yes, EdgelessDB is not built with Performance Schema enabled. We are indecisive about whether we want to include it or not, given that enabling it by default also does not seem like a good idea as it can impact performance and also other parts of the code that we would need to test.

However, I see that you seem to be using a self-built version anyway. If you want to try it by yourself, you can apply the changes from the following branch and you should be able to enable performance schema afterwards with the way you tried before: https://github.com/edgelesssys/edgelessdb/tree/feat/perf-schema

I briefly tested this yesterday and it seems to work, but I haven't tested it in-depth.

In case you try it out, it would be great to hear back from you if everything works as expected, or if there are still any issues left. Maybe we can try to fix any arising issues from our side. We will likely upstream the changes from mysqld_edb.cc anyway since they could be impacting other parts in MariaDB, so even if we decide not to enable Performance Schema upstream by default, we could still apply fixes so it becomes easier to enable the plugin (or potentially other plugins) for users who are willing to build EdgelessDB by themselves.

water5-cmd commented 2 years ago

Hi, @Nirusu,

Thank you for your help! I highly approve of what you said about helping users build EdgelessDB with different plugins enabled.

I have enabled PERFSCHEMA on EdgelessDB v0.2.1 and it works as expected (I will use the latest version of EdgelessDB as soon as possible). I will bring it up again if I run into problems in subsequent tests.

One more question, does PLUGIN_*=NO/STATIC located in CMakeLists.txt serve the purpose of disabling and enabling? I saw that you also modified mysqld_edb.cc when you enabled performance_schema, do I need to modify the code of other files as well if I enable other plugins?

Nirusu commented 2 years ago

In theory, yes, enabling a plugin should work by setting:

  1. the CMake build option to STATIC
  2. adding the path of the built static library (*.afile) to be linked into the enclave. That's what the ${MARIADB}/storage/perfschema/libperfschema.a addition in my branch does.

Not all plugins support STATIC if I recall correctly. These likely won't work.

In practise, however, your mileage may vary greatly. There's a medium to high chance of things breaking I would say, since MariaDB enables multiple different build options that change the behaviour of the code and thus might break things, due to different reasons.

The PERFSCHEMA is an example of that, and that's where also the changes from mysqld_edb.cc came from. We patched the network listening in MariaDB to also include listening on an internal socket so the initialisation via the EdgelessDB manifest can happen completely in the enclave. After this, we call mysql_socket_accept, which in the current release mainly seems to be a wrapper around accept(2), which explicitly allows setting addr and addrlen to be NULL:

The argument addr is a pointer to a sockaddr structure. This structure is filled in with the address of the peer socket, as known to the communications layer. The exact format of the address returned addr is determined by the socket's address family (see socket(2) and the respective protocol man pages). When addr is NULL, nothing is filled in; in this case, addrlen is not used, and should also be NULL.

Now, enabling PERFSCHEMA changes the underlying implementation of sockets in MariaDB to use something called a "PSI socket". This is not completely unreasonable, given that Performance Schema also tracks socket performance. I haven't analysed this too much and honestly don't know what it exactly does, but from what I could tell, the change to the PSI socket (or Performance Schema specifically, not sure) does actually expect addr and addrlen to exist, and crashes when it's NULL due to trying to access a null pointer.

In this case, it's a relatively easy fix: We supply an empty data structure and the PSI socket/Performance Schema can read/write to it if it wants to. However, for other plugins that might actually be not as easy to fix, since like virtually all SGX runtimes, we do not support every possible Linux syscall or flag to a syscall. This might break things or expectations the original MariaDB code has. This is something you will often encounter when using ported applications on SGX, since the LibOSs / runtimes need to reimplement the Linux ABI in a way that makes sense in the SGX context and cannot just redirect everything to the underlying host system without breaking confidentiality.

And confidentiality is an important consideration. Even if a plugin technically works, it could accidentally expose data to the host system. A good example for this are plugins involving disk I/O such as storage engines, where e.g. enabling a different storage engine without any changes will likely write unencrypted data to the disk on the host filesystem outside of the enclave, which kinda breaks the point of running a database on SGX if the host can read or manipulate your data via its disk.

So if you're planning to enable something like InnoDB, you don't need to try, it'll go wrong without further changes. ;)

TL;DR: You can try enabling other plugins. It can likely break stuff under the hood, but if it works, please check that it does not break confidentiality.

If you encounter any bugs with any plugins, you can let us know (with a good description on how to reproduce, please!). We might be able to help you and we can check see if it's an easy fix or not that we could carry over to upstream.

Nirusu commented 2 years ago

By the way, the changes to mysqld_edb.cc have been merged to the current main branch. So if you want to rebuild from the most recent main later on, you can skip the changes on this file.

water5-cmd commented 2 years ago

Thanks a lot for the explanation, it was very useful for me! When I enable the new plugin later, I will double-check it based on your suggestions.

water5-cmd commented 2 years ago

Hi, @Nirusu @thomasten,

When I enabled the PERFSCHEMA plugin and started the binlog mode, an error occurred during the initialization of EdgelessDB. Here is the details. EdgelessDB v0.2.1, without PERFSCHEMA & EDG_EDB_LOG_DIR enabled, build and run successfully, log is:

root@adminroot:~/edgelessdb/build# ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:23:01 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:23:01 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:23:01 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:23:16 initializing ...
2022-10-25  2:23:16 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2272393 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:24:05 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:24:05 starting up ...
2022-10-25  2:24:05 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2272393 ...
2022-10-25  2:24:05 0 [Warning] You need to use --log-bin to make --binlog-format work.
2022-10-25  2:24:05 0 [Note] Initializing built-in plugins
2022-10-25  2:24:05 0 [Note] Initializing plugins specified on the command line
2022-10-25  2:24:05 0 [Note] RocksDB: 4 column families found
2022-10-25  2:24:05 0 [Note] RocksDB: Column Families at start:
2022-10-25  2:24:05 0 [Note]   cf=default
2022-10-25  2:24:05 0 [Note]     write_buffer_size=67108864
2022-10-25  2:24:05 0 [Note]     target_file_size_base=67108864
2022-10-25  2:24:05 0 [Note]   cf=__system__
2022-10-25  2:24:05 0 [Note]     write_buffer_size=67108864
2022-10-25  2:24:05 0 [Note]     target_file_size_base=67108864
2022-10-25  2:24:05 0 [Note]   cf=edg_db_cf
2022-10-25  2:24:05 0 [Note]     write_buffer_size=67108864
2022-10-25  2:24:05 0 [Note]     target_file_size_base=67108864
2022-10-25  2:24:05 0 [Note]   cf=edg_frm_cf
2022-10-25  2:24:05 0 [Note]     write_buffer_size=67108864
2022-10-25  2:24:05 0 [Note]     target_file_size_base=67108864
2022-10-25  2:24:06 0 [Note] RocksDB: Table_store: loaded DDL data for 109 tables
2022-10-25  2:24:06 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2022-10-25  2:24:06 0 [Note] RocksDB: sched_getcpu() failed - global statistics will use thread_id_indexer_t instead
2022-10-25  2:24:06 0 [Note] MyRocks storage engine plugin has been successfully initialized.
2022-10-25  2:24:07 0 [Note] Initializing installed plugins
2022-10-25  2:24:07 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-10-25  2:24:07 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'root' host: '' (This connection closed normally)
2022-10-25  2:24:07 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '255.0.0.1' (This connection closed normally without authentication)
2022-10-25  2:24:07 0 [Note] Reading of all Master_info entries succeeded
2022-10-25  2:24:07 0 [Note] Added new Master_info '' to hash table
2022-10-25  2:24:07 0 [Note] edb: ready for connections.
Version: '10.5.11-MariaDB-debug-log'  socket: ''  port: 3306  Source distribution
internalPath:/tmp/edb
internalAdress:255.0.0.1
externalPath:/data
externalAdress:
[EDB] 2022/10/25 02:24:07 DB is running.
[EDB] 2022/10/25 02:24:07 HTTP REST API listening on :8080

EdgelessDB v0.2.1, without PERFSCHEMA enabled, with EDG_EDB_LOG_DIR enabled, build and run successfully, log is:

root@adminroot:~/edgelessdb/build# EDG_EDB_LOG_DIR=edblogs ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:28:05 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:28:05 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:28:05 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:28:09 initializing ...
2022-10-25  2:28:09 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2273455 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:28:57 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:28:57 starting up ...
2022-10-25  2:28:57 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2273455 ...

The content of edblogs is:

root@adminroot:~/edgelessdb/build# ls edblogs/
data_LOG                       mariadb-binary.000001  mariadb.err  mariadb-slow.log
data_LOG.old.1666664938380665  mariadb-binary.index   mariadb.log

EdgelessDB v0.2.1, with PERFSCHEMA enabled, with or without EDG_EDB_LOG_DIR enabled, build successfully, but there is an error runing, log is:

PERFSCHEMA enabled, EDG_EDB_LOG_DIR disenabled

root@adminroot:~/edgelessdb/build# ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:36:16 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:36:16 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:36:17 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:36:19 initializing ...
2022-10-25  2:36:20 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2284529 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:37:08 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:37:08 starting up ...
2022-10-25  2:37:08 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2284529 ...
2022-10-25  2:37:08 0 [Warning] You need to use --log-bin to make --binlog-format work.
2022-10-25  2:37:08 0 [Note] Initializing built-in plugins
2022-10-25  2:37:08 0 [Note] Initializing plugins specified on the command line
2022-10-25  2:37:08 0 [Note] RocksDB: 4 column families found
2022-10-25  2:37:08 0 [Note] RocksDB: Column Families at start:
2022-10-25  2:37:08 0 [Note]   cf=default
2022-10-25  2:37:08 0 [Note]     write_buffer_size=67108864
2022-10-25  2:37:08 0 [Note]     target_file_size_base=67108864
2022-10-25  2:37:08 0 [Note]   cf=__system__
2022-10-25  2:37:08 0 [Note]     write_buffer_size=67108864
2022-10-25  2:37:08 0 [Note]     target_file_size_base=67108864
2022-10-25  2:37:08 0 [Note]   cf=edg_db_cf
2022-10-25  2:37:08 0 [Note]     write_buffer_size=67108864
2022-10-25  2:37:08 0 [Note]     target_file_size_base=67108864
2022-10-25  2:37:08 0 [Note]   cf=edg_frm_cf
2022-10-25  2:37:08 0 [Note]     write_buffer_size=67108864
2022-10-25  2:37:08 0 [Note]     target_file_size_base=67108864
2022-10-25  2:37:09 0 [Note] RocksDB: Table_store: loaded DDL data for 109 tables
2022-10-25  2:37:09 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2022-10-25  2:37:09 0 [Note] RocksDB: sched_getcpu() failed - global statistics will use thread_id_indexer_t instead
2022-10-25  2:37:09 0 [Note] MyRocks storage engine plugin has been successfully initialized.
2022-10-25  2:37:10 0 [Note] Initializing installed plugins
2022-10-25  2:37:10 0 [Note] Server socket created on IP: '0.0.0.0'.
./edb: line 3: 2284529 Segmentation fault      (core dumped) erthost "$DIR/edb-enclave.signed" "$@"

PERFSCHEMA enabled, EDG_EDB_LOG_DIR enabled

root@adminroot:~/edgelessdb/build# EDG_EDB_LOG_DIR=edblogs ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:38:18 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:38:18 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:38:19 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:38:22 initializing ...
2022-10-25  2:38:22 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2285095 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:39:14 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:39:14 starting up ...
2022-10-25  2:39:14 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2285095 ...
./edb: line 3: 2285095 Segmentation fault      (core dumped) erthost "$DIR/edb-enclave.signed" "$@"

I'm working on this problem, but haven't figured out why yet. Can you give me some advice? BTW, I'm not sure if EdgelessDB v0.3.1 has this problem, you can try it.

thomasten commented 2 years ago

Hi @lijun0314,

Sorry, I'm confused. You say

with PERFSCHEMA enabled, with or without EDG_EDB_LOG_DIR enabled, build successfully, but there is an error runing

So the crash is only related to PERFSCHEMA and not to binlog (or to EDG_EDB_LOG_DIR, respectively) or am I missing something?

Previously, you said

I have enabled PERFSCHEMA on EdgelessDB v0.2.1 and it works as expected

Have you changed anything such that it has stopped working since then?

water5-cmd commented 2 years ago

Hi, @thomasten,

Yes, EdgelessDB crashed because of an incorrect code change, so sorry about that!