Exporting backup is broken

[Jikstra]

Trying to export a backup on desktop results in this error:

DC_EVENT_ERROR 0 failed to backup plaintext database to PathBuf { inner: "/home/x/Downloads/delta-chat-backup-2022-02-23-00.db" }: failed to export to attached backup database: no such function: sqlcipher_export: Error code 1: SQL error or missing database

core version: v1.75.0

[hpk42]

i've done some backups from desktop successfully recently. Today, i retried with recent master desktop/node/master compile and it exports fine, just tested. Little more info from my compile:

`

@.*** build:core:rust node scripts/rebuild-core.js

Compiling deltachat_ffi v1.76.0 (/home/hpk/p/delta/core-rust2/deltachat-ffi) ... `

[link2xt]

Make sure you are using "bundled/vendored" version, i.e. no --no-default-features when running cargo. You can also uninstall system sqlcipher headers. There is also an issue deltachat/deltachat-core-rust#3080, check that you are using the Cargo.lock lockfile as commited to the deltachat-core-rust repo so you are building exactly the same version of rusqlite and libsqlite3-sys.

[link2xt]

Can confirm this on Arch Linux when building desktop.

[link2xt]

deltachat.a static library is an ar archive which contains object files, among them is the sqlite3.o file. My guess is that in the linking step of the node module there is another library involved which also contains sqlite3.o. When there are multiple libraries with the same .o file inside, the first one is picked silently and the rest are ignored.

I tried as an experiment to compile the following test.c file:

#include <stdio.h>
const char *sqlite3_libversion(void);

int main () {
        printf("%s\n", sqlite3_libversion());
        return 0;
}

Then created a library consisting of a single sqlite3.o file compiled from this source

const char *sqlite3_libversion(void){
        return "foo";
}

The result is:

$ ar rc libmylib.a sqlite3.o
$ gcc test.c libdeltachat.a libmylib.a -lm && ./a.out 
3.36.0
$ gcc test.c libmylib.a libdeltachat.a -lm && ./a.out
foo

[link2xt]

Update: actual command make is running when executing node-gyp build is

g++ -shared -pthread -rdynamic -m64  -Wl,-soname=deltachat.node -o Release/obj.target/deltachat.node -Wl,--start-group Release/obj.target/deltachat/src/module.o -Wl,--end-group -lpthread ../deltachat-core-rust/target/release/libdeltachat.a -ldl -lm -lrt

So i'm not sure what is going on.

[link2xt]

@Jikstra could you add a test to deltachat-node for exporting of backup? So we isolate this problem to either node.js or electron. Export is definitely working in the core itself because it's tested in python and it's working in deltachat-android.

[link2xt]

I have added

[patch.crates-io]
rusqlite = { path = "/home/user/src/rusqlite" }

to Cargo.toml of the checked out submodule of deltachat-node. Then opened libsqlite3-sys/sqlcipher/sqlite3.c inside rusqlite and inserted exit(1) there into openDatabase (which also calls sqlite3RegisterPerConnectionBuiltinFunctions to register sqlcipher_export). This change causes libsqlite3-sys to rebuild and npm run test in deltachat-node now exits before finishing, yet deltachat-desktop runs just as well as before.

Turns out electron in deltachat-desktop somehow hijacks sqlite completely and makes the node module use its own vanilla sqlite3 implementation.

[link2xt]

This seems related: https://www.electronjs.org/docs/latest/tutorial/using-native-node-modules https://github.com/electron/electron-rebuild

Electron has something I didin't have time to understand yet to rebuild already compiled native node modules on the fly to match the node version used in electron, which seems to involve replacing sqlite implementation. This is probably what is going on, but I don't know if it's electron-rebuild or something else used in deltachat-desktop, and how to disable it.

[link2xt]

I have applied the following patch to rusqlite:

rusqlite]$ git diff --staged
diff --git a/libsqlite3-sys/sqlcipher/sqlite3.c b/libsqlite3-sys/sqlcipher/sqlite3.c
index 286600f..4c96c4d 100644
--- a/libsqlite3-sys/sqlcipher/sqlite3.c
+++ b/libsqlite3-sys/sqlcipher/sqlite3.c
@@ -173528,6 +173528,8 @@ SQLITE_API int sqlite3_open_v2(
   int flags,              /* Flags */
   const char *zVfs        /* Name of VFS module to use */
 ){
+  fprintf(stderr, "EXITING 3\n");
+  exit(3);
   return openDatabase(filename, ppDb, (unsigned int)flags, zVfs);
 }

Then added this to deltachat-core-rust/Cargo.toml:

[patch.crates-io]
rusqlite = { path = "/home/user/src/rusqlite" }

Then built deltachat-node exits when running tests:

$ npm run test

> deltachat-node@1.75.1 test
> npm run test:lint && npm run test:mocha

> deltachat-node@1.75.1 test:lint
> npm run lint

> deltachat-node@1.75.1 lint
> prettier --check "lib/**/*.{ts,tsx}"

Checking formatting...
All matched files use Prettier code style!

> deltachat-node@1.75.1 test:mocha
> mocha -r esm test/test.js --growl --reporter=spec

  static tests
    ✓ reverse lookup of events
    ✓ event constants are consistent
    ✓ static method maybeValidAddr()
EXITING 3

deltachat-desktop still works and uses its own sqlite3 version.

Turns out sqlite3_open_v2 has relocation entry and as a result can still be hijacked even after stripping, this is what electron does I think:

[deltachat-node]$ strip  build/Release/deltachat.node
[deltachat-node]$ objdump -R build/Release/deltachat.node | grep sqlite3_open_v2
000000000129b978 R_X86_64_64       sqlite3_open_v2@@Base
00000000012a85c8 R_X86_64_GLOB_DAT  sqlite3_open_v2@@Base

[link2xt]

This seems to be what we want to do with the deltachat.node binary - hide the symbols of sqlite3.o: https://stackoverflow.com/questions/393980/restricting-symbols-in-a-linux-static-library

SQLite has an easy option for C programs, but it's not applicable to rusqlite: https://stackoverflow.com/a/29951687

[link2xt]

I have managed to make electron run my own version of sqlite3.

I opened the Makefile generated by node-gyp in deltachat-node/build/Makefile and modified it as follows to see the linking command:

#quiet_cmd_solink_module = SOLINK_MODULE($(TOOLSET)) $@                      
quiet_cmd_solink_module = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -o $@ -Wl,--start-group $(filter-out FORCE_DO_CMD, $^) -Wl,--end-group $(LIBS)
cmd_solink_module = $(LINK.$(TOOLSET)) -shared $(GYP_LDFLAGS) $(LDFLAGS.$(TOOLSET)) -Wl,-soname=$(@F) -o $@ -Wl,--start-group $(filter-out FORCE_DO_CMD, $^) -Wl,--end-group $(LIBS)

Then I ran the Makefile:

$ rm Release/obj.target/deltachat.node 
$ make
  g++ -shared -pthread -rdynamic -m64  -Wl,-soname=deltachat.node -o Release/obj.target/deltachat.node -Wl,--start-group Release/obj.target/deltachat/src/module.o -Wl,--end-group -lpthread ../deltachat-core-rust/target/release/libdeltachat.a -ldl -lm -lrt
  COPY Release/deltachat.node

Now I modified the command with -Wl,-Bsymbolic and recompiled:

$ g++ -shared -pthread -rdynamic -m64  -Wl,-soname=deltachat.node -o Release/obj.target/deltachat.node -Wl,--start-group Release/obj.target/deltachat/src/module.o -Wl,--end-group -lpthread -Wl,-Bsymbolic ../deltachat-core-rust/target/release/libdeltachat.a -Wl,-Bdynamic -ldl -lm -lrt

Running Makefile again copied the deltachat.node from Release/obj.target/deltachat.node to Release/deltachat.node and then running electron killed the app immediately with the exit I inserted earlier.

The following patch should probably work, going to try it:

diff --git a/binding.gyp b/binding.gyp
index 199f4d3..0c04fda 100644
--- a/binding.gyp
+++ b/binding.gyp
@@ -42,7 +42,9 @@
                 {
                   "include_dirs": ["deltachat-core-rust/deltachat-ffi"],
                   "libraries": [
+                   "-Wl,-Bsymbolic",
                     "../deltachat-core-rust/target/release/libdeltachat.a",
+                   "-Wl,-Bdynamic",
                     "-ldl",
                   ],
                   "conditions": [],