Include `better-sqlite` within each release

VishwaJay commented 1 year ago

Pre-flight checklist

[X] I have read the contribution documentation for this project.
[X] I agree to follow the code of conduct that this project uses.
[X] I have searched the issue tracker for a feature request that matches the one I want to file, without success.

Problem description

Lots of people keep wanting to add the functionality of sqlite3 to their electron and electron-forge builds.

The top error I keep finding mentioned is that the prebuilt executables were created with a different version of nodejs and/or webpack (which is several point releases ahead of the version included with electron-forge). Attempts at rebuilding the various sqlite3 packages will fail because of this unresolvable disparity, and any successes that anyone seems to have are either short-lived (breaking with the next release of node or webpack or electron-forge) or they are platform-dependent (and also short-lived, breaking with the next release, etc.).

This complex issue has no great solution that continues to work for any length of time.

Proposed solution

This isn't the fault of electron-forge at all. But a potential solution is to include some flavour of SQLite sources and rebuild against the installed release from source, so that it has some basic included functionality and doesn't continually run into the version disparity that plagues the current electron ecosphere as a whole.

I think perhaps there may even need to be @electron-forge/sqlite3 or some such package, based on source from an existing project (such as better-sqlite3 which seems to have the best maintenance of the packages to date). This would keep the source code on a slower release than the official package, but it would offer some degree of consistency with regard to prebuilt executables and inclusion in packages and makes.

And it might even be simply a snapshot of that existing package/source, so that the devs don't have to fiddle with it too much and can simply help project developers make use of the functionality as needed.

This is, of course, entirely in the purview of the devs.

Alternatives considered

I have given thought to several alternatives:

A collaborative effort, while making a lot of logical sense, is likely a logistical nightmare.
A clear set of instructions for integration seems like it should be possible, but everyone's thought that for at least 6 years that I can find. Every result that works will break when ONE of the packages no longer matches what the other two need. It's not merely a case of dependencies, but rather one of specific versioning (in the same way that webpack needs its own template).
Getting devs from the other sqlite packages to conform to the needs of electron-forge seems like a non-starter (i.e., that it would never move forward and at least one side of it would fail to participate).

In all cases, the real solution seems to be that the electron-forge team is in the best position to do anything about it.

Additional information

However, rather than racing to the finish line, this is something that's going to take a great deal of study and time.

I mean, I certainly don't have the skill set (never mind the time or salient understand of the inner workings of electron-forge) needed to do this kind of integration.

It will need to be updated for the LTS versions of nodejs, rather than for every major release, just for the sake of sanity.

It will need to be updated with major releases of webpack and should at least be rebuilt annually.

But the sheer complexity of the electron-forge package (rather than the resulting ease-of-use that is a major hallmark of the project) means that at best this should be on a lower priority development tangent if it's even pursued.

VishwaJay commented 1 year ago

To be clear (as it doesn't seem so after hitting "open"), I am requesting a new feature in the long term.

kilbot commented 1 year ago

I hate to think how many developer hours have been wasted trying to get a standard electron + sqlite3 build. It's an endless loop of NODE_MODULE_VERSION, code signing issues etc etc. get it work and then update one package thing and it all breaks again. There has to be a better way.

VishwaJay commented 1 year ago

So... I checked again over at the better-sqlite Electron Help Thread, where none of their suggestions were working with the last release. But suddenly, After trying it after updating electron (and not the whole forge project), I found it working!

Tested on an empty install. Fully working on vanilla electron 26.0.0.

Used npm (instead of yarn) to install better-sqlite3. Added a reference (no methods, other than an init line:

const sqlite3 = require('better-sqlite3');

Triggered the expected crash (reproduction).

Added the following line to my scripts in the electron-forge project and ran it:

"rebuild-sqlite": "npx electron-rebuild -f -w -t dev better-sqlite3"

Ran it as a script, though npm didn't like how I held my mouth whilst I typed it out on the command line. Which is to say: it failed on the command line, and yet worked perfectly as the above script.

Fully working on Node.js LTS (currently 18.12.0) on Windows 11.

Can anyone else confirm this as working on other platforms? Seems to be working on my Ubuntu VBox.

If we can get a confirmation on at least 2 other kernel flavours (Ubuntu's core is GNU/Linux) and a Mac, should we request a pull of that line into the package.json file in the scripting section?

malept commented 1 year ago

Unfortunately, the Electron Forge maintainers don't have the bandwidth to maintain a sqlite3 native module. Ultimately, it's up to the community to either enhance the existing solutions or create a new solution.

VishwaJay commented 1 year ago

The beauty of this script is that if better-sqlite@latest is added as a dependency, the above script seems to recompile it for the current platform/release.

electron / forge