Simplify configuration for custom builds

[JoshuaWise]

better-sqlite3 supports the ability to be compiled with custom versions of SQLite3. When doing this, any desired compile-time configuration must be prepended directly into the sqlite3.c source file.

For example, to compile with SQLCipher, the typical process is this:

# First build the SQLCipher files
git clone https://github.com/sqlcipher/sqlcipher.git
cd sqlcipher
./configure --enable-tempstore=yes CFLAGS="-DSQLITE_HAS_CODEC"
make sqlite3.c

# Then prepend your custom configuration
echo "#define SQLITE_HAS_CODEC" | tee -a sqlite3.c_temp sqlite3.h_temp
echo "#define SQLITE_TEMP_STORE 2" | tee -a sqlite3.c_temp sqlite3.h_temp
cat sqlite3.c >> sqlite3.c_temp
cat sqlite3.h >> sqlite3.h_temp
mv -f sqlite3.c_temp sqlite3.c
mv -f sqlite3.h_temp sqlite3.h

# Then install better-sqlite3 with your custom amalgamation
cd /path/to/my/nodejs/project
npm install --sqlite3=/absolute/path/to/sqlcipher

If better-sqlite3 adopted the _HAVE_SQLITE_CONFIG_H compile-time option, custom configuration could more easily be applied by simply creating a config.h file. However, the disadvantage is that people will need to create config.h even if custom configuration is not desired.

[zoubingwu]

like their docs said, when build SQLCipher files it will have to link libcrypto I think?

./configure --enable-tempstore=yes CFLAGS="-DSQLITE_HAS_CODEC" \
    LDFLAGS="/your/path/to/libcrypto.a"

I am trying to use better-sqlite3 with sqlcipher build for our electron app and since it is a monorepo, so we are using yarn as pakcakge manager.

1. git clone sqlcipher
2. build a custom amalgamation file(remember to have openssl in your include path since sqlcipher use openssl as default cypto provider otherwise it can't find the proper header file for #include <openssl/rand.h>)
3. cd to our own project, and run yarn install to install all dependencies for every workspace.
4. rebuild for electron. I was using node-gyp directly cus electron is very likely to use a different V8 version so we have to recompile for it, npm install --sqlite3=xx or npm rebuild --sqlite3=xx won't be enough. Instead I was running

   node-gyp --target=your_electron_version \
   --dist-url=https://electronjs.org/headers \
   --arch=x64 \
   --sqlite3=path/to/sqlcipher \
   --directory=./node_modules/better-sqlite3

This will actually work for the original better-sqlite3 build but when --sqlite3=path/to/sqlcipher was added, it throws error saying some system calls function could not be found in the sqlcipher/sqlite3.c file, not sure this is some dependency issue on c files or what. I'll paste the error message here tomorrow.

@JoshuaWise any thoughts on this? much appreciated.

[JoshuaWise]

@shadeofgod I haven't personally delved too much into all the different possible ways of compiling with SQLCipher, because I don't actually use it in my daily life. Sorry I can't help much. All I know is that the instructions I listed above work on my machine, and on various linux boxes I've used.

SQLCipher is not officially supported by better-sqlite3. The ability to compile with other sqlite3-like sources is more of a "you're on your own" kind of thing. Only the official SQLite3 is actually supported.

[zoubingwu]

SQLCipher is not officially supported by better-sqlite3. The ability to compile with other sqlite3-like sources is more of a "you're on your own" kind of thing. Only the official SQLite3 is actually supported.

@JoshuaWise Yeah I totally understand this and your work is really amazing.

As for sqlite3, I'm not sure how many people use it in their node application since it was mainly for building web servers. But I can see there are heavy needs in electron apps as an embedded database. So just thought it will be good if there is an easy-to-use out-of-box solution in the open source community and I'm happy to contribute for it.

I finally managed to get things done 😊. After rebuild for electron like I said before, when I run my electron app,

it throws error saying some system calls function could not be found

this was actually due to the SQLITE_HAS_CODEC define was missing so I fixed it after recompile.

and then it shows errors when connect to the database:

dyld: lazy symbol binding failed: Symbol not found: _EVP_get_cipherbyname
  Referenced from: /path/to/project/node_modules/better-sqlite3/build/better_sqlite3.node
  Expected in: flat namespace

dyld: Symbol not found: _EVP_get_cipherbyname
  Referenced from: /path/to/project/node_modules/better-sqlite3/build/better_sqlite3.node
  Expected in: flat namespace

I've spent hours digging into this, as a javascript developer this is really a PITA to solving those c/c++ compiling issues. And it turns out that it could not find the correct openssl lib. I tweaked a little bit in the binding.gyp in node_modules/better-sqlite3, this line was added to add dependence to static library:

"libraries": [ "/my/own/path/to/libcrypto.a" ]

this path was the same when trying to build SQLCipher from source:

./configure --enable-tempstore=yes CFLAGS="-DSQLITE_HAS_CODEC" \
    LDFLAGS="/my/own/path/to/libcrypto.a"

there are still warnings like object files was built for newer OSX version (10.13) than being linked (10.7) but it worked!

So I was thinking, since sqlcipher depends on openssl it would be better to link to node or electron's own built-in openssl But I have no idea how to achieve that.

or is it possible to use some other flags to make better-sqlite3 could be built with statically linked libsqlcipher.a?

or do you have any better idea? my knowledge on c/c++ is really limited.

[JoshuaWise]

It's weird that you had this trouble with libcrypto.a in the first place. I wonder why my machine doesn't run into that error. It could be that I have some kind of global LD configuration set up. I'll need to do some digging to get to the bottom of this.

[zoubingwu]

@JoshuaWise now I've running into another issue. I think this might have something to do with this lib, pls correct me if I'm wrong.

generally in a node app, we simply write our code and use node entry.js to run it, all the code including dependencies runs on our own machine. But for client side app, to reduce the file size, we usually use bundlers like webpack or parcel to bundle everything together. for example we use webpack to bundle all the files, uglify it and then use electron-builder to pack it to .dmg for end users. It reduced the size from 260mb to 60mb after bundled.

It was totally ok in js world cus the bundler will statically analyze js files and map the relative path to correct denpendent after bundled.

but for a native module, if the native code depends on some js code (I found it depends on the sqlite-error.js in this cpp file), it might cause some problem when trying to bundle it since the bundler can do nothing on a compiled binary like .node file so it makes better-sqlite3 not "webpack-friendly".

[zoubingwu]

sorry, let me elaborate on this, for example we have:

// a.js
const hi = require('./b.js');
hi()

// b.js
module.exports = function() {console.log('hi')}

after bundled, it generates a bundle.js file, each module was isolated in function scope and they have access to other module through module object by invoking scoped require function.

not 100% accurate but something like this:

// bundle.js, it has a giant IIFE to hold everything
(function(modules) {
...
  // module object holds refs to each module and was stored in closure
  var module = { exports: {} };

  function require(id) {
    ...
  }

...
})({
  './a.js': function(module, require) { var hi = require('./b.js'); hi(); },
  './b.js': function(module, require) { module.exports = function() {console.log('hi')} },
})

you can see that all the js files was bundled into a bundle.js and if native code depends on some ./c.js, it simply couldn't find it.

[zoubingwu]

so just wondering is there any other way to handle the sqlite-error in c++? I think it might be better to not depends on other js code in the native part.

[zoubingwu]

found it also requires integer module in native code, it gonna be tough to solve this webpack issue.

[JoshuaWise]

We might be able to move sqlite-error into C++ with some fairly substantial refactoring, but moving integer is not really an option. Perhaps you can tell webpack to ignore integer and sqlite-error.js, or put them in specific locations, and then symlink to them so better-sqlite3 can find them.

[zoubingwu]

was kinda busy recently, there is a way to do so in webpack I think, I will try it later

[JoshuaWise]

Closing due to lack of interest