Closed water5-cmd closed 1 year ago
Hi @lijun0314,
Yes, EdgelessDB is not built with Performance Schema enabled. We are indecisive about whether we want to include it or not, given that enabling it by default also does not seem like a good idea as it can impact performance and also other parts of the code that we would need to test.
However, I see that you seem to be using a self-built version anyway. If you want to try it by yourself, you can apply the changes from the following branch and you should be able to enable performance schema afterwards with the way you tried before: https://github.com/edgelesssys/edgelessdb/tree/feat/perf-schema
I briefly tested this yesterday and it seems to work, but I haven't tested it in-depth.
In case you try it out, it would be great to hear back from you if everything works as expected, or if there are still any issues left. Maybe we can try to fix any arising issues from our side. We will likely upstream the changes from mysqld_edb.cc
anyway since they could be impacting other parts in MariaDB, so even if we decide not to enable Performance Schema upstream by default, we could still apply fixes so it becomes easier to enable the plugin (or potentially other plugins) for users who are willing to build EdgelessDB by themselves.
Hi, @Nirusu,
Thank you for your help! I highly approve of what you said about helping users build EdgelessDB with different plugins enabled.
I have enabled PERFSCHEMA
on EdgelessDB v0.2.1 and it works as expected (I will use the latest version of EdgelessDB as soon as possible). I will bring it up again if I run into problems in subsequent tests.
One more question, does PLUGIN_*=NO/STATIC
located in CMakeLists.txt
serve the purpose of disabling and enabling? I saw that you also modified mysqld_edb.cc
when you enabled performance_schema
, do I need to modify the code of other files as well if I enable other plugins?
In theory, yes, enabling a plugin should work by setting:
STATIC
*.a
file) to be linked into the enclave. That's what the ${MARIADB}/storage/perfschema/libperfschema.a
addition in my branch does.Not all plugins support STATIC
if I recall correctly. These likely won't work.
In practise, however, your mileage may vary greatly. There's a medium to high chance of things breaking I would say, since MariaDB enables multiple different build options that change the behaviour of the code and thus might break things, due to different reasons.
The PERFSCHEMA
is an example of that, and that's where also the changes from mysqld_edb.cc
came from. We patched the network listening in MariaDB to also include listening on an internal socket so the initialisation via the EdgelessDB manifest can happen completely in the enclave. After this, we call mysql_socket_accept
, which in the current release mainly seems to be a wrapper around accept(2), which explicitly allows setting addr
and addrlen
to be NULL:
The argument addr is a pointer to a sockaddr structure. This structure is filled in with the address of the peer socket, as known to the communications layer. The exact format of the address returned addr is determined by the socket's address family (see socket(2) and the respective protocol man pages). When addr is NULL, nothing is filled in; in this case, addrlen is not used, and should also be NULL.
Now, enabling PERFSCHEMA
changes the underlying implementation of sockets in MariaDB to use something called a "PSI socket". This is not completely unreasonable, given that Performance Schema also tracks socket performance. I haven't analysed this too much and honestly don't know what it exactly does, but from what I could tell, the change to the PSI socket (or Performance Schema specifically, not sure) does actually expect addr
and addrlen
to exist, and crashes when it's NULL due to trying to access a null pointer.
In this case, it's a relatively easy fix: We supply an empty data structure and the PSI socket/Performance Schema can read/write to it if it wants to. However, for other plugins that might actually be not as easy to fix, since like virtually all SGX runtimes, we do not support every possible Linux syscall or flag to a syscall. This might break things or expectations the original MariaDB code has. This is something you will often encounter when using ported applications on SGX, since the LibOSs / runtimes need to reimplement the Linux ABI in a way that makes sense in the SGX context and cannot just redirect everything to the underlying host system without breaking confidentiality.
And confidentiality is an important consideration. Even if a plugin technically works, it could accidentally expose data to the host system. A good example for this are plugins involving disk I/O such as storage engines, where e.g. enabling a different storage engine without any changes will likely write unencrypted data to the disk on the host filesystem outside of the enclave, which kinda breaks the point of running a database on SGX if the host can read or manipulate your data via its disk.
So if you're planning to enable something like InnoDB, you don't need to try, it'll go wrong without further changes. ;)
TL;DR: You can try enabling other plugins. It can likely break stuff under the hood, but if it works, please check that it does not break confidentiality.
If you encounter any bugs with any plugins, you can let us know (with a good description on how to reproduce, please!). We might be able to help you and we can check see if it's an easy fix or not that we could carry over to upstream.
By the way, the changes to mysqld_edb.cc
have been merged to the current main branch. So if you want to rebuild from the most recent main later on, you can skip the changes on this file.
Thanks a lot for the explanation, it was very useful for me! When I enable the new plugin later, I will double-check it based on your suggestions.
Hi, @Nirusu @thomasten,
When I enabled the PERFSCHEMA plugin and started the binlog mode, an error occurred during the initialization of EdgelessDB. Here is the details. EdgelessDB v0.2.1, without PERFSCHEMA & EDG_EDB_LOG_DIR enabled, build and run successfully, log is:
root@adminroot:~/edgelessdb/build# ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:23:01 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:23:01 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:23:01 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:23:16 initializing ...
2022-10-25 2:23:16 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2272393 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:24:05 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:24:05 starting up ...
2022-10-25 2:24:05 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2272393 ...
2022-10-25 2:24:05 0 [Warning] You need to use --log-bin to make --binlog-format work.
2022-10-25 2:24:05 0 [Note] Initializing built-in plugins
2022-10-25 2:24:05 0 [Note] Initializing plugins specified on the command line
2022-10-25 2:24:05 0 [Note] RocksDB: 4 column families found
2022-10-25 2:24:05 0 [Note] RocksDB: Column Families at start:
2022-10-25 2:24:05 0 [Note] cf=default
2022-10-25 2:24:05 0 [Note] write_buffer_size=67108864
2022-10-25 2:24:05 0 [Note] target_file_size_base=67108864
2022-10-25 2:24:05 0 [Note] cf=__system__
2022-10-25 2:24:05 0 [Note] write_buffer_size=67108864
2022-10-25 2:24:05 0 [Note] target_file_size_base=67108864
2022-10-25 2:24:05 0 [Note] cf=edg_db_cf
2022-10-25 2:24:05 0 [Note] write_buffer_size=67108864
2022-10-25 2:24:05 0 [Note] target_file_size_base=67108864
2022-10-25 2:24:05 0 [Note] cf=edg_frm_cf
2022-10-25 2:24:05 0 [Note] write_buffer_size=67108864
2022-10-25 2:24:05 0 [Note] target_file_size_base=67108864
2022-10-25 2:24:06 0 [Note] RocksDB: Table_store: loaded DDL data for 109 tables
2022-10-25 2:24:06 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2022-10-25 2:24:06 0 [Note] RocksDB: sched_getcpu() failed - global statistics will use thread_id_indexer_t instead
2022-10-25 2:24:06 0 [Note] MyRocks storage engine plugin has been successfully initialized.
2022-10-25 2:24:07 0 [Note] Initializing installed plugins
2022-10-25 2:24:07 0 [Note] Server socket created on IP: '0.0.0.0'.
2022-10-25 2:24:07 1 [Warning] Aborted connection 1 to db: 'unconnected' user: 'root' host: '' (This connection closed normally)
2022-10-25 2:24:07 2 [Warning] Aborted connection 2 to db: 'unconnected' user: 'unauthenticated' host: '255.0.0.1' (This connection closed normally without authentication)
2022-10-25 2:24:07 0 [Note] Reading of all Master_info entries succeeded
2022-10-25 2:24:07 0 [Note] Added new Master_info '' to hash table
2022-10-25 2:24:07 0 [Note] edb: ready for connections.
Version: '10.5.11-MariaDB-debug-log' socket: '' port: 3306 Source distribution
internalPath:/tmp/edb
internalAdress:255.0.0.1
externalPath:/data
externalAdress:
[EDB] 2022/10/25 02:24:07 DB is running.
[EDB] 2022/10/25 02:24:07 HTTP REST API listening on :8080
EdgelessDB v0.2.1, without PERFSCHEMA enabled, with EDG_EDB_LOG_DIR enabled, build and run successfully, log is:
root@adminroot:~/edgelessdb/build# EDG_EDB_LOG_DIR=edblogs ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:28:05 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:28:05 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:28:05 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:28:09 initializing ...
2022-10-25 2:28:09 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2273455 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:28:57 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:28:57 starting up ...
2022-10-25 2:28:57 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2273455 ...
The content of edblogs
is:
root@adminroot:~/edgelessdb/build# ls edblogs/
data_LOG mariadb-binary.000001 mariadb.err mariadb-slow.log
data_LOG.old.1666664938380665 mariadb-binary.index mariadb.log
EdgelessDB v0.2.1, with PERFSCHEMA enabled, with or without EDG_EDB_LOG_DIR enabled, build successfully, but there is an error runing, log is:
root@adminroot:~/edgelessdb/build# ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:36:16 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:36:16 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:36:17 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:36:19 initializing ...
2022-10-25 2:36:20 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2284529 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:37:08 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:37:08 starting up ...
2022-10-25 2:37:08 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2284529 ...
2022-10-25 2:37:08 0 [Warning] You need to use --log-bin to make --binlog-format work.
2022-10-25 2:37:08 0 [Note] Initializing built-in plugins
2022-10-25 2:37:08 0 [Note] Initializing plugins specified on the command line
2022-10-25 2:37:08 0 [Note] RocksDB: 4 column families found
2022-10-25 2:37:08 0 [Note] RocksDB: Column Families at start:
2022-10-25 2:37:08 0 [Note] cf=default
2022-10-25 2:37:08 0 [Note] write_buffer_size=67108864
2022-10-25 2:37:08 0 [Note] target_file_size_base=67108864
2022-10-25 2:37:08 0 [Note] cf=__system__
2022-10-25 2:37:08 0 [Note] write_buffer_size=67108864
2022-10-25 2:37:08 0 [Note] target_file_size_base=67108864
2022-10-25 2:37:08 0 [Note] cf=edg_db_cf
2022-10-25 2:37:08 0 [Note] write_buffer_size=67108864
2022-10-25 2:37:08 0 [Note] target_file_size_base=67108864
2022-10-25 2:37:08 0 [Note] cf=edg_frm_cf
2022-10-25 2:37:08 0 [Note] write_buffer_size=67108864
2022-10-25 2:37:08 0 [Note] target_file_size_base=67108864
2022-10-25 2:37:09 0 [Note] RocksDB: Table_store: loaded DDL data for 109 tables
2022-10-25 2:37:09 0 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2022-10-25 2:37:09 0 [Note] RocksDB: sched_getcpu() failed - global statistics will use thread_id_indexer_t instead
2022-10-25 2:37:09 0 [Note] MyRocks storage engine plugin has been successfully initialized.
2022-10-25 2:37:10 0 [Note] Initializing installed plugins
2022-10-25 2:37:10 0 [Note] Server socket created on IP: '0.0.0.0'.
./edb: line 3: 2284529 Segmentation fault (core dumped) erthost "$DIR/edb-enclave.signed" "$@"
root@adminroot:~/edgelessdb/build# EDG_EDB_LOG_DIR=edblogs ./edb
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:38:18 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
[EDB] 2022/10/25 02:38:18 DB has not been initialized, waiting for manifest.
[EDB] 2022/10/25 02:38:19 HTTP REST API listening on :8080
[EDB] 2022/10/25 02:38:22 initializing ...
2022-10-25 2:38:22 0 [Note] edb (mysqld 10.5.11-MariaDB-debug) starting as process 2285095 ...
restarting ...
[erthost] loading enclave ...
[erthost] entering enclave ...
[EDB] 2022/10/25 02:39:14 EdgelessDB v0.2.1 (8c1df2066513153ac9b8e129a0924abe431e3191)
host is: 0.0.0.0[EDB] 2022/10/25 02:39:14 starting up ...
2022-10-25 2:39:14 0 [Note] edb (mysqld 10.5.11-MariaDB-debug-log) starting as process 2285095 ...
./edb: line 3: 2285095 Segmentation fault (core dumped) erthost "$DIR/edb-enclave.signed" "$@"
I'm working on this problem, but haven't figured out why yet. Can you give me some advice? BTW, I'm not sure if EdgelessDB v0.3.1 has this problem, you can try it.
Hi @lijun0314,
Sorry, I'm confused. You say
with PERFSCHEMA enabled, with or without EDG_EDB_LOG_DIR enabled, build successfully, but there is an error runing
So the crash is only related to PERFSCHEMA and not to binlog (or to EDG_EDB_LOG_DIR, respectively) or am I missing something?
Previously, you said
I have enabled PERFSCHEMA on EdgelessDB v0.2.1 and it works as expected
Have you changed anything such that it has stopped working since then?
Hi, @thomasten,
Yes, EdgelessDB crashed because of an incorrect code change, so sorry about that!
Hi, @thomasten,
I found that edgelssdb does not have performance_schema and when I try to use the performance_schema variable, the following error is reported during the database startup:
I learn performance_schema from Performance Schema Overview, but there is no information about this variable in edgelessdb and the logs are as follows.
So, edgelessdb doesn't implement and use the performance_schema feature? Is there any way to set the value of performance_schema to OFF?