search

TryGhost / node-sqlite3

SQLite3 bindings for Node.js
BSD 3-Clause "New" or "Revised" License
6.21k stars 815 forks source link

Unable to import sqlite3 without critical compile warnings + runtime error #1029

Closed j-berman closed 5 years ago

j-berman commented 6 years ago

I've been trying to get sqlite3 imported into a create-react-app for the past couple days. Any help would be much appreciated. When I initially install and run the app, I get the following compile error:

Failed to compile.

./node_modules/node-pre-gyp/lib/info.js
Module not found: Can't resolve 'aws-sdk'

After I install aws-sdk (either by installing inside ./node_modules/node-pre-gyp or on the project's root), I get the following output in my CLI when I try to run the app:

./node_modules/node-pre-gyp/lib/util/versioning.js
17:20-67 Critical dependency: the request of a dependency is an expression

./node_modules/node-pre-gyp/lib/pre-binding.js
20:22-48 Critical dependency: the request of a dependency is an expression

./node_modules/sqlite3/lib/sqlite3.js
4:14-35 Critical dependency: the request of a dependency is an expression

./node_modules/node-pre-gyp/lib/util/compile.js
Module not found: Can't resolve 'node-gyp' in '/Users/justin/testcode/node_test/test_app/node_modules/node-pre-gyp/lib/util'

./node_modules/node-pre-gyp/lib/util/compile.js
Module not found: Can't resolve 'npm' in '/Users/justin/testcode/node_test/test_app/node_modules/node-pre-gyp/lib/util'

./node_modules/node-pre-gyp/lib/util/nw-pre-gyp/index.html
Module parse failed: Unexpected token (1:0)
You may need an appropriate loader to handle this file type.
| <!doctype html>
| <html>
| <head>

And I get this runtime error:

TypeError: Cannot read property '_handle' of undefined
(anonymous function)
node_modules/set-blocking/index.js:3
  1 | module.exports = function (blocking) {
  2 |   [process.stdout, process.stderr].forEach(function (stream) {
> 3 |     if (stream._handle && stream.isTTY && typeof stream._handle.setBlocking === 'function') {
  4 |       stream._handle.setBlocking(blocking)
  5 |     }
  6 |   })
View compiled
./node_modules/set-blocking/index.js.module.exports
node_modules/set-blocking/index.js:2
  1 | module.exports = function (blocking) {
> 2 |   [process.stdout, process.stderr].forEach(function (stream) {
  3 |     if (stream._handle && stream.isTTY && typeof stream._handle.setBlocking === 'function') {
  4 |       stream._handle.setBlocking(blocking)
  5 |     }
View compiled
(anonymous function)
node_modules/npmlog/log.js:11
   8 | var setBlocking = require('set-blocking')
   9 | var consoleControl = require('console-control-strings')
  10 | 
> 11 | setBlocking(true)
  12 | var stream = process.stderr
  13 | Object.defineProperty(log, 'stream', {
  14 |   set: function (newStream) {
View compiled
./node_modules/npmlog/log.js
http://localhost:3000/static/js/bundle.js:72472:30
__webpack_require__
/Users/justin/testcode/node_test/test_app/webpack/bootstrap 66935c31b13be3dd3e9e:678
  675 | };
  676 | 
  677 | // Execute the module function
> 678 | modules[moduleId].call(module.exports, module, module.exports, hotCreateRequire(moduleId));
  679 | 
  680 | // Flag the module as loaded
  681 | module.l = true;
View compiled
fn
/Users/justin/testcode/node_test/test_app/webpack/bootstrap 66935c31b13be3dd3e9e:88
  85 |      console.warn("[HMR] unexpected require(" + request + ") from disposed module " + moduleId);
  86 |      hotCurrentParents = [];
  87 |  }
> 88 |  return __webpack_require__(request);
  89 | };
  90 | var ObjectFactory = function ObjectFactory(name) {
  91 |  return {
View compiled
(anonymous function)
node_modules/node-pre-gyp/lib/node-pre-gyp.js:16
  13 | var fs = require('fs');
  14 | var path = require('path');
  15 | var nopt = require('nopt');
> 16 | var log = require('npmlog');
  17 | log.disableProgress();
  18 | var napi = require('./util/napi.js');
  19 | 
View compiled
./node_modules/node-pre-gyp/lib/node-pre-gyp.js
http://localhost:3000/static/js/bundle.js:69853:30
__webpack_require__
/Users/justin/testcode/node_test/test_app/webpack/bootstrap 66935c31b13be3dd3e9e:678
  675 | };
  676 | 
  677 | // Execute the module function
> 678 | modules[moduleId].call(module.exports, module, module.exports, hotCreateRequire(moduleId));
  679 | 
  680 | // Flag the module as loaded
  681 | module.l = true;
View compiled
fn
/Users/justin/testcode/node_test/test_app/webpack/bootstrap 66935c31b13be3dd3e9e:88
  85 |      console.warn("[HMR] unexpected require(" + request + ") from disposed module " + moduleId);
  86 |      hotCurrentParents = [];
  87 |  }
> 88 |  return __webpack_require__(request);
  89 | };
  90 | var ObjectFactory = function ObjectFactory(name) {
  91 |  return {
View compiled
(anonymous function)
node_modules/sqlite3/lib/sqlite3.js:1
> 1 | var binary = require('node-pre-gyp');
  2 | var path = require('path');
  3 | var binding_path = binary.find(path.resolve(path.join(__dirname,'../package.json')));
  4 | var binding = require(binding_path);
View compiled
./node_modules/sqlite3/lib/sqlite3.js
http://localhost:3000/static/js/bundle.js:123571:30
__webpack_require__
/Users/justin/testcode/node_test/test_app/webpack/bootstrap 66935c31b13be3dd3e9e:678
  675 | };
  676 | 
  677 | // Execute the module function
> 678 | modules[moduleId].call(module.exports, module, module.exports, hotCreateRequire(moduleId));
  679 | 
  680 | // Flag the module as loaded
  681 | module.l = true;
View compiled
fn
/Users/justin/testcode/node_test/test_app/webpack/bootstrap 66935c31b13be3dd3e9e:88
  85 |      console.warn("[HMR] unexpected require(" + request + ") from disposed module " + moduleId);
  86 |      hotCurrentParents = [];
  87 |  }
> 88 |  return __webpack_require__(request);
  89 | };
  90 | var ObjectFactory = function ObjectFactory(name) {
  91 |  return {
View compiled
./src/App.js
src/App.js:4
  1 | import React, { Component } from 'react';
  2 | import logo from './logo.svg';
  3 | import './App.css';
> 4 | var sqlite3 = require('sqlite3').verbose();
  5 | 
  6 | 
  7 | class App extends Component {

I've tried installing from source, installing past versions, using a different node version, and looking into similar issues (#909 , this issue, this issue, and some others) with no luck.

From what I've seen, it appears the issue may stem from how webpack is configured, but my familiarity with webpack is pretty lacking. For reference, create react app automatically configures webpack with this config file.

Also for reference, here's the log when I install regularly with npm install sqlite3 --loglevel=info:

npm info it worked if it ends with ok
npm info using npm@5.6.0
npm info using node@v10.1.0
npm http fetch GET 304 https://registry.npmjs.org/sqlite3 171ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/node-pre-gyp 81ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/npm-packlist 85ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/needle 185ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/detect-libc 249ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/nopt 269ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/tar 280ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/npmlog 283ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/abbrev 67ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/osenv 75ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/ignore-walk 65ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/npm-bundled 78ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/are-we-there-yet 77ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/gauge 82ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/console-control-strings 92ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/delegates 74ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/aproba 63ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/wide-align 77ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/has-unicode 87ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/minizlib 55ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/chownr 57ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/fs-minipass 85ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/minipass 94ms (from cache)
npm http fetch GET 304 https://registry.npmjs.org/yallist 96ms (from cache)
npm info lifecycle abbrev@1.1.1~preinstall: abbrev@1.1.1
npm info lifecycle aproba@1.2.0~preinstall: aproba@1.2.0
npm info lifecycle chownr@1.0.1~preinstall: chownr@1.0.1
npm info lifecycle console-control-strings@1.1.0~preinstall: console-control-strings@1.1.0
npm info lifecycle delegates@1.0.0~preinstall: delegates@1.0.0
npm info lifecycle are-we-there-yet@1.1.5~preinstall: are-we-there-yet@1.1.5
npm info lifecycle detect-libc@1.0.3~preinstall: detect-libc@1.0.3
npm info lifecycle has-unicode@2.0.1~preinstall: has-unicode@2.0.1
npm info lifecycle ignore-walk@3.0.1~preinstall: ignore-walk@3.0.1
npm info lifecycle needle@2.2.1~preinstall: needle@2.2.1
npm info lifecycle npm-bundled@1.0.3~preinstall: npm-bundled@1.0.3
npm info lifecycle npm-packlist@1.1.10~preinstall: npm-packlist@1.1.10
npm info lifecycle osenv@0.1.5~preinstall: osenv@0.1.5
npm info lifecycle nopt@4.0.1~preinstall: nopt@4.0.1
npm info lifecycle wide-align@1.1.3~preinstall: wide-align@1.1.3
npm info lifecycle gauge@2.7.4~preinstall: gauge@2.7.4
npm info lifecycle npmlog@4.1.2~preinstall: npmlog@4.1.2
npm info lifecycle yallist@3.0.2~preinstall: yallist@3.0.2
npm info lifecycle minipass@2.3.3~preinstall: minipass@2.3.3
npm info lifecycle fs-minipass@1.2.5~preinstall: fs-minipass@1.2.5
npm info lifecycle minizlib@1.1.0~preinstall: minizlib@1.1.0
npm info lifecycle tar@4.4.4~preinstall: tar@4.4.4
npm info lifecycle node-pre-gyp@0.10.3~preinstall: node-pre-gyp@0.10.3
npm info lifecycle sqlite3@4.0.2~preinstall: sqlite3@4.0.2
npm info linkStuff abbrev@1.1.1
npm info linkStuff aproba@1.2.0
npm info linkStuff chownr@1.0.1
npm info linkStuff console-control-strings@1.1.0
npm info linkStuff delegates@1.0.0
npm info linkStuff are-we-there-yet@1.1.5
npm info linkStuff detect-libc@1.0.3
npm info linkStuff has-unicode@2.0.1
npm info linkStuff ignore-walk@3.0.1
npm info linkStuff needle@2.2.1
npm info linkStuff npm-bundled@1.0.3
npm info linkStuff npm-packlist@1.1.10
npm info linkStuff osenv@0.1.5
npm info linkStuff nopt@4.0.1
npm info linkStuff wide-align@1.1.3
npm info linkStuff gauge@2.7.4
npm info linkStuff npmlog@4.1.2
npm info linkStuff yallist@3.0.2
npm info linkStuff minipass@2.3.3
npm info linkStuff fs-minipass@1.2.5
npm info linkStuff minizlib@1.1.0
npm info linkStuff tar@4.4.4
npm info linkStuff node-pre-gyp@0.10.3
npm info linkStuff sqlite3@4.0.2
npm info lifecycle abbrev@1.1.1~install: abbrev@1.1.1
npm info lifecycle aproba@1.2.0~install: aproba@1.2.0
npm info lifecycle chownr@1.0.1~install: chownr@1.0.1
npm info lifecycle console-control-strings@1.1.0~install: console-control-strings@1.1.0
npm info lifecycle delegates@1.0.0~install: delegates@1.0.0
npm info lifecycle are-we-there-yet@1.1.5~install: are-we-there-yet@1.1.5
npm info lifecycle detect-libc@1.0.3~install: detect-libc@1.0.3
npm info lifecycle has-unicode@2.0.1~install: has-unicode@2.0.1
npm info lifecycle ignore-walk@3.0.1~install: ignore-walk@3.0.1
npm info lifecycle needle@2.2.1~install: needle@2.2.1
npm info lifecycle npm-bundled@1.0.3~install: npm-bundled@1.0.3
npm info lifecycle npm-packlist@1.1.10~install: npm-packlist@1.1.10
npm info lifecycle osenv@0.1.5~install: osenv@0.1.5
npm info lifecycle nopt@4.0.1~install: nopt@4.0.1
npm info lifecycle wide-align@1.1.3~install: wide-align@1.1.3
npm info lifecycle gauge@2.7.4~install: gauge@2.7.4
npm info lifecycle npmlog@4.1.2~install: npmlog@4.1.2
npm info lifecycle yallist@3.0.2~install: yallist@3.0.2
npm info lifecycle minipass@2.3.3~install: minipass@2.3.3
npm info lifecycle fs-minipass@1.2.5~install: fs-minipass@1.2.5
npm info lifecycle minizlib@1.1.0~install: minizlib@1.1.0
npm info lifecycle tar@4.4.4~install: tar@4.4.4
npm info lifecycle node-pre-gyp@0.10.3~install: node-pre-gyp@0.10.3
npm info lifecycle sqlite3@4.0.2~install: sqlite3@4.0.2

> sqlite3@4.0.2 install /Users/justin/testcode/node_test/test_app/node_modules/sqlite3
> node-pre-gyp install --fallback-to-build

node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using node-pre-gyp@0.10.3
node-pre-gyp info using node@10.1.0 | darwin | x64
node-pre-gyp WARN Using request for node-pre-gyp https download 
node-pre-gyp info check checked for "/Users/justin/testcode/node_test/test_app/node_modules/sqlite3/lib/binding/node-v64-darwin-x64/node_sqlite3.node" (not found)
node-pre-gyp http GET https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v4.0.2/node-v64-darwin-x64.tar.gz
node-pre-gyp http 200 https://mapbox-node-binary.s3.amazonaws.com/sqlite3/v4.0.2/node-v64-darwin-x64.tar.gz
node-pre-gyp info install unpacking node-v64-darwin-x64/node_sqlite3.node
node-pre-gyp info tarball done parsing tarball
[sqlite3] Success: "/Users/justin/testcode/node_test/test_app/node_modules/sqlite3/lib/binding/node-v64-darwin-x64/node_sqlite3.node" is installed via remote
node-pre-gyp info ok 
npm info lifecycle abbrev@1.1.1~postinstall: abbrev@1.1.1
npm info lifecycle aproba@1.2.0~postinstall: aproba@1.2.0
npm info lifecycle chownr@1.0.1~postinstall: chownr@1.0.1
npm info lifecycle console-control-strings@1.1.0~postinstall: console-control-strings@1.1.0
npm info lifecycle delegates@1.0.0~postinstall: delegates@1.0.0
npm info lifecycle are-we-there-yet@1.1.5~postinstall: are-we-there-yet@1.1.5
npm info lifecycle detect-libc@1.0.3~postinstall: detect-libc@1.0.3
npm info lifecycle has-unicode@2.0.1~postinstall: has-unicode@2.0.1
npm info lifecycle ignore-walk@3.0.1~postinstall: ignore-walk@3.0.1
npm info lifecycle needle@2.2.1~postinstall: needle@2.2.1
npm info lifecycle npm-bundled@1.0.3~postinstall: npm-bundled@1.0.3
npm info lifecycle npm-packlist@1.1.10~postinstall: npm-packlist@1.1.10
npm info lifecycle osenv@0.1.5~postinstall: osenv@0.1.5
npm info lifecycle nopt@4.0.1~postinstall: nopt@4.0.1
npm info lifecycle wide-align@1.1.3~postinstall: wide-align@1.1.3
npm info lifecycle gauge@2.7.4~postinstall: gauge@2.7.4
npm info lifecycle npmlog@4.1.2~postinstall: npmlog@4.1.2
npm info lifecycle yallist@3.0.2~postinstall: yallist@3.0.2
npm info lifecycle minipass@2.3.3~postinstall: minipass@2.3.3
npm info lifecycle fs-minipass@1.2.5~postinstall: fs-minipass@1.2.5
npm info lifecycle minizlib@1.1.0~postinstall: minizlib@1.1.0
npm info lifecycle tar@4.4.4~postinstall: tar@4.4.4
npm info lifecycle node-pre-gyp@0.10.3~postinstall: node-pre-gyp@0.10.3
npm info lifecycle sqlite3@4.0.2~postinstall: sqlite3@4.0.2
npm info lifecycle undefined~preshrinkwrap: undefined
npm info lifecycle undefined~shrinkwrap: undefined
npm info lifecycle undefined~postshrinkwrap: undefined
npm WARN ajv-keywords@3.2.0 requires a peer of ajv@^6.0.0 but none is installed. You must install peer dependencies yourself.

+ sqlite3@4.0.2
added 24 packages in 14.92s
npm info ok
janglada commented 6 years ago

I solved it by adding

externals: { 'sqlite3':'commonjs sqlite3', }

to the webpack config file

j-berman commented 6 years ago

@janglada unfortunately I'm unable to get this to work, but I'm working on it. I saw this solution in issue #698 as well.

When I attempt to add externals: { sqlite3: 'commonjs sqlite3' } to my webpack config file and start the node server via npm start, the server appears to run perfectly fine with no errors in the command line. However, when I open up the browser to interact with the app, I get the following error in the browser console:

external "sqlite3":1 Uncaught ReferenceError: require is not defined
    at Object.sqlite3 (external "sqlite3":1)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at fn (bootstrap c8470ec94022a2bb9e9f:88)
    at Object../src/App.js (App.js:4)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at fn (bootstrap c8470ec94022a2bb9e9f:88)
    at Object../src/index.js (index.css?f255:26)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at fn (bootstrap c8470ec94022a2bb9e9f:88)
    at Object.0 (registerServiceWorker.js:117)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at ./node_modules/ansi-regex/index.js.module.exports (bootstrap c8470ec94022a2bb9e9f:724)
    at bootstrap c8470ec94022a2bb9e9f:724

I could be getting this wrong, but in looking around to understand this issue, it's because when you add externals to the webpack file, it tells webpack to ignore bundling sqlite3 prior to running the app and to load the dependency at run-time. But, that also means sqlite3 won't get loaded within node's execution environment -- the browser attempts to load it. And the browser doesn't have a built-in require function like node does, thus the undefined reference error. I've found some potential solutions to this that I'm currently in the process of exploring

Zuzon commented 6 years ago

have same issue, and what's interesting. In runtime it fails on sqlite3's package.json require

converge commented 6 years ago

@janglada solution solved my issue

qaziwaqar commented 5 years ago

I am having an exact same issue as @j-berman but I am not still able to resolve it. does anyone got any solution/fix ? Below is my error.


[0] Starting the development server...
[0] 
[0] Compiled with warnings.
[0] 
[0] ./node_modules/sqlite3/lib/sqlite3.js
[0] Critical dependency: the request of a dependency is an expression
[0] 
[0] ./node_modules/sqlite3/node_modules/node-pre-gyp/lib/pre-binding.js
[0] Critical dependency: the request of a dependency is an expression
[0] 
[0] ./node_modules/sqlite3/node_modules/node-pre-gyp/lib/util/versioning.js
[0] Critical dependency: the request of a dependency is an expression
[0] 
[0] ./src/App.js
[0]   Line 19:   'electron_process' is assigned a value but never used  no-unused-vars
[0]   Line 20:   'electron_app' is assigned a value but never used      no-unused-vars
[0]   Line 37:  Expected '===' and instead saw '=='                    eqeqeq
[0] 
[0] ./src/Login.js
[0]   Line 5:    'db' is assigned a value but never used                                                                    no-unused-vars
[0]   Line 63:  img elements must have an alt prop, either with meaningful text, or an empty string for decorative images  jsx-a11y/alt-text
[0] 
[0] ./node_modules/sqlite3/node_modules/node-pre-gyp/lib/util/compile.js
[0] Module not found: Can't resolve 'npm' in '/Volumes/Example/node_modules/sqlite3/node_modules/node-pre-gyp/lib/util'
[0] 
[0] ./node_modules/sqlite3/node_modules/node-pre-gyp/lib/util/nw-pre-gyp/index.html 1:0
[0] Module parse failed: Unexpected token (1:0)
[0] You may need an appropriate loader to handle this file type.
[0] > <!doctype html>
[0] | <html>
[0] | <head>
[0] 
[0] Search for the keywords to learn more about each warning.
[0] To ignore, add // eslint-disable-next-line to the line before.
[0]
converge commented 5 years ago

I think node-pre-gyp requires python 2.7

qaziwaqar commented 5 years ago

I already have python 2.7.10 installed

FengYangLiu commented 5 years ago

There are prerequisites for using sqlite3, and it must be in a node environment，I tried the above solution. electron can be used and the browser will report an error. such as : require('sqlite3')

j-berman commented 5 years ago

@FengYangLiu that makes sense. Unfortunately I've been super busy lately and haven't gotten around to digging much deeper. But, essentially I think the root of this boils down to the sandboxed nature of the browser: the local filesystem isn't inherently available to a web page. I think sqlite3 is designed to run within a node environment with access to the local filesystem (i.e. not from within the browser).

I think the architecture I was aiming for (a browser that uses a local sql db file as the persistent data store) is only achievable like this:

  1. Request a sandboxed filesystem and do some hacking to get sqlite3 to interact with it
  2. Import the db file from the user's local filesystem into memory using sql.js

Actually implementing 1 seems too fragile because it would only work in Chrome and Chrome may drop support for it in the future. I didn't look much into this route at all.

I don't like 2 because the process of importing/exporting the db file (and the size limitation of the db) feels pretty terrible.

I think the best/closest solution to my desired architecture is IndexedDB (and using one of those libraries noted that uses IndexedDB.. Lovefield is also a sql-based library for reference if anyone is looking for that -- I saw someone mentioned using that in issue 698).

Overall, I don't think this is an issue with sqlite3 so I'm going to close the issue

FrancescoBorzi commented 4 years ago

I have the same issue, however, I couldn't solve it. I'm using Angular + Electron, I described my issue here:

https://stackoverflow.com/questions/60363339/using-sqlite-in-an-angular-electron-app-critical-dependency-the-request-of-a

any help would be much appreciated.

UPDATE: I've found a SOLUTION for my issue, you find it in the link above.

TheLazyHatGuy commented 4 years ago

I don't think @FrancescoBorzi method works with the most recent angular-electron boilerplate

FrancescoBorzi commented 4 years ago

@TheLazyHatGuy you can add a new answer to my original question :)

TheLazyHatGuy commented 4 years ago

@FrancescoBorzi I haven't figured out the answer yet. Tried to apply your suggestion from Stack Overflow but it didn't work

Javadebi commented 4 years ago

Is there any solution? i'm still having this problem.

sjfkai commented 4 years ago

I solved it by adding

externals: { 'sqlite3':'commonjs sqlite3', }

to the webpack config file

Works for me with electron. But why?

tim-rohrer commented 3 years ago

I really appreciate this more detailed response. I was experiencing the same error with a CRA-based app I'm working on, and now I realize that I will need to shift the sqlite code to a mini-service rather than continuing to try and resolve the error.

Thank you!

@FengYangLiu that makes sense. Unfortunately I've been super busy lately and haven't gotten around to digging much deeper. But, essentially I think the root of this boils down to the sandboxed nature of the browser: the local filesystem isn't inherently available to a web page. I think sqlite3 is designed to run within a node environment with access to the local filesystem (i.e. not from within the browser).

j-berman commented 3 years ago

No problem @tim-rohrer :)

Side note, excuse this shameless self-promotion, but I helped make a tool to satisfy my requirements on this and figure it may be useful. I'm planning to release it in the next week. It's end-to-end encrypted sql.js -- an encrypted sql.js database gets loaded from the server and decrypted client-side. All writes to the database are encrypted.

I talked about it a bit here, and made a CRA-based app with it here (can also check the console in that app to see the SQL statements in action)

j-berman commented 3 years ago

@tim-rohrer

Here's the end-to-end encrypted sql.js tool I mentioned above! Hope others arriving here may find it useful too.

eyea commented 3 years ago

@janglada unfortunately I'm unable to get this to work, but I'm working on it. I saw this solution in issue #698 as well.

When I attempt to add externals: { sqlite3: 'commonjs sqlite3' } to my webpack config file and start the node server via npm start, the server appears to run perfectly fine with no errors in the command line. However, when I open up the browser to interact with the app, I get the following error in the browser console:

external "sqlite3":1 Uncaught ReferenceError: require is not defined
    at Object.sqlite3 (external "sqlite3":1)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at fn (bootstrap c8470ec94022a2bb9e9f:88)
    at Object../src/App.js (App.js:4)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at fn (bootstrap c8470ec94022a2bb9e9f:88)
    at Object../src/index.js (index.css?f255:26)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at fn (bootstrap c8470ec94022a2bb9e9f:88)
    at Object.0 (registerServiceWorker.js:117)
    at __webpack_require__ (bootstrap c8470ec94022a2bb9e9f:678)
    at ./node_modules/ansi-regex/index.js.module.exports (bootstrap c8470ec94022a2bb9e9f:724)
    at bootstrap c8470ec94022a2bb9e9f:724

I could be getting this wrong, but in looking around to understand this issue, it's because when you add externals to the webpack file, it tells webpack to ignore bundling sqlite3 prior to running the app and to load the dependency at run-time. But, that also means sqlite3 won't get loaded within node's execution environment -- the browser attempts to load it. And the browser doesn't have a built-in require function like node does, thus the undefined reference error. I've found some potential solutions to this that I'm currently in the process of exploring

so, do you have a better way to solve this problem, I'm in the same case :(

j-berman commented 3 years ago

@eyea

I think this is the main limiting issue:

node-sqlite3 is designed to run within a node environment with access to the local filesystem (i.e. not from within the browser)

And the local filesystem isn't available to JS running in the browser.

But, I did just come across this package called BrowserFS that may help you workaround that. Check out their webpack config. Haven't tried it, I'm not optimistic :/ I bet with some serious hacking this could eventually be made to work, but would be pretty challenging

Edit: this require script may come in handy too

entwwwicklung commented 3 years ago

This https://github.com/mapbox/node-sqlite3/issues/1029#issuecomment-414116796 worked for me.

For those using vue3 cli and they don't have webpack config file - just edit or create vue.config.js file in project root directory and add following: module.exports = { configureWebpack: { externals: { 'sqlite3': 'commonjs sqlite3', } } }

Reference: https://stackoverflow.com/a/50304636/4642868