search

databricks / databricks-sql-nodejs

Databricks SQL Connector for Node.js
Apache License 2.0
24 stars 32 forks source link

lz4 dependency is causing install issues for our users #245

Closed archiewood closed 5 months ago

archiewood commented 5 months ago

A recent version of this connector, 1.8.0 introduced lz4 compression https://github.com/databricks/databricks-sql-nodejs/pull/216

We recently upgraded from <1.8.0 to 1.8.3 and we have since then had several reports from users about being unable to install our library.

Here is an example, but i'd say we've had c.10-20 reports of this in the last 3 weeks: https://github.com/evidence-dev/evidence/issues/1885

The crux of the issue seems to be that lz4 is building from source, and this requires a number of pieces of software to be installed on a client's machine that they do not always have, and didn't previously need: make, visual studio build tools etc.

For the moment, I'm intending on resolving this by downgrading our Databricks dependency to 1.7.x but I thought it might be useful for your team to know.

kravets-levko commented 5 months ago

@archiewood thank you for reporting. I'd also prefer any native LZ4 library, but unfortunately there's basically no choice: lz4 is the only library that supports LZ4 frame API. I think we can make lz4 dependency optional, so at least the connector will remain usable when lz4 cannot be installed

archiewood commented 5 months ago

Thanks for the fast response. I think this approach would work for us

kravets-levko commented 5 months ago

Hi @archiewood! We just released v1.8.4 where lz4 became optional. Please give it a try, and let me know if it helped (if not - please also re-open this issue)

szsori commented 3 months ago

Hi @archiewood! We just released v1.8.4 where lz4 became optional. Please give it a try, and let me know if it helped (if not - please also re-open this issue)

Have you had anyone report issues with this? I'm receiving this error when running 1.8.4:

 ⚠ ./node_modules/@databricks/sql/dist/utils/lz4.js
Module not found: Can't resolve 'lz4' in '[redacted]'
Did you mean './lz4'?
Requests that should resolve in the current directory need to start with './'.
Requests that start with a name are treated as module requests and resolve within module directories (node_modules).
If changing the source code is not an option there is also a resolve options called 'preferRelative' which tries to resolve these kind of requests in the current directory too.

Import trace for requested module:
./node_modules/@databricks/sql/dist/utils/lz4.js
./node_modules/@databricks/sql/dist/utils/index.js
./node_modules/@databricks/sql/dist/index.js
kravets-levko commented 3 months ago

Hi @szsori! This doesn't look like a NPM or Yarn error - do you use any bundler maybe? If yes - please share its configuration to help me to reproduce your issue

szsori commented 3 months ago

@kravets-levko We're on NextJS 14, which uses webpack 5 (we haven't switched to turbo yet). Here is the relevant area of our config:

webpack: (config) => {
    config.externals.push({
      "utf-8-validate": "commonjs utf-8-validate",
      bufferutil: "commonjs bufferutil",
    });
    return config;
  }

Our package.json looks like:

{
  "name": "raptor",
  "version": "0.5.1",
  "private": true,
  "scripts": {
    "dev": "next dev",
    "build": "next build",
    "start": "next start",
    "lint": "next lint"
  },
  "dependencies": {
    "@databricks/sql": "^1.7.0",
    "@faker-js/faker": "^8.4.1",
    "@fortawesome/fontawesome-svg-core": "^6.4.2",
    "@fortawesome/free-solid-svg-icons": "^6.4.2",
    "@fortawesome/react-fontawesome": "^0.2.0",
    "@progress/kendo-drawing": "^1.20.1",
    "@progress/kendo-file-saver": "^1.1.1",
    "@progress/kendo-licensing": "^1.3.5",
    "@progress/kendo-react-common": "^7.4.0",
    "@progress/kendo-react-pdf": "^7.4.0",
    "@reduxjs/toolkit": "^2.0.1",
    "@rehooks/component-size": "^1.0.3",
    "algoliasearch": "^4.22.0",
    "axios": "^1.6.5",
    "bootstrap": "^5.3.2",
    "echarts": "^5.4.3",
    "express": "^4.19.2",
    "moment": "^2.30.1",
    "next": "^14.2.3",
    "papaparse": "^5.4.1",
    "react": "^18",
    "react-bootstrap": "^2.9.1",
    "react-dom": "^18",
    "react-native-segmented-control-tab": "^4.0.0",
    "react-redux": "^9.0.4",
    "react-segmented-control": "^0.2.2",
    "react-select": "^5.8.0",
    "redux-persist": "^6.0.0",
    "sharp": "^0.33.2"
  },
  "devDependencies": {
    "@types/node": "^20",
    "@types/papaparse": "^5.3.13",
    "@types/react": "^18",
    "@types/react-dom": "^18",
    "eslint": "^8",
    "eslint-config-next": "14.0.3",
    "typescript": "^5"
  }
}
kravets-levko commented 3 months ago

Thank you @szsori! I think this is enough to reproduce your issue. I will try to do it ASAP, and will get back to you. If you (by any chance) will figure it out yourself - please let me know so we can fix the library, or even submit a PR 🙂

szsori commented 3 months ago

@kravets-levko This is in our next.config.js as well, but removing it doesn't resolve the issue. This was to resolve a separate nextjs issue detailed here: https://github.com/vercel/next.js/issues/62193. Just to be clear, everything still works... we're just getting errors in the console.

experimental: {
    serverComponentsExternalPackages: ["lz4"],
  },
haggholm commented 1 week ago

Even on hosts that do have all dependencies installed, LZ4 sometimes fails to compile for no obvious reason. We have a CI pipeline that usually passes but occasionally fails. We have not yet invested any time into investigating this, but I'm sharing it as a kind of warning sign.

It's worth noting that the LZ4 repository

95.47 npm ERR! code 1
95.47 npm ERR! path /home/node/app/node_modules/lz4
95.48 npm ERR! command failed
95.48 npm ERR! command sh -c node-gyp rebuild
95.48 npm ERR! make: Entering directory '/home/node/app/node_modules/lz4/build'
95.48 npm ERR!   CXX(target) Release/obj.target/lz4/lib/binding/lz4_binding.o
95.48 npm ERR! make: Leaving directory '/home/node/app/node_modules/lz4/build'
95.48 npm ERR! gyp info it worked if it ends with ok
95.48 npm ERR! gyp info using node-gyp@10.0.1
95.48 npm ERR! gyp info using node@20.12.2 | linux | x64
95.48 npm ERR! gyp info find Python using Python version 3.7.3 found at "/usr/bin/python3"
95.48 npm ERR! gyp http GET https://nodejs.org/download/release/v20.12.2/node-v20.12.2-headers.tar.gz
95.48 npm ERR! gyp http 200 https://nodejs.org/download/release/v20.12.2/node-v20.12.2-headers.tar.gz
95.48 npm ERR! gyp http GET https://nodejs.org/download/release/v20.12.2/SHASUMS256.txt
95.48 npm ERR! gyp http 200 https://nodejs.org/download/release/v20.12.2/SHASUMS256.txt
95.48 npm ERR! gyp info spawn /usr/bin/python3
95.48 npm ERR! gyp info spawn args [
95.48 npm ERR! gyp info spawn args '/home/node/app/node_modules/node-gyp/gyp/gyp_main.py',
95.48 npm ERR! gyp info spawn args 'binding.gyp',
95.48 npm ERR! gyp info spawn args '-f',
95.48 npm ERR! gyp info spawn args 'make',
95.48 npm ERR! gyp info spawn args '-I',
95.48 npm ERR! gyp info spawn args '/home/node/app/node_modules/lz4/build/config.gypi',
95.48 npm ERR! gyp info spawn args '-I',
95.48 npm ERR! gyp info spawn args '/home/node/app/node_modules/node-gyp/addon.gypi',
95.48 npm ERR! gyp info spawn args '-I',
95.48 npm ERR! gyp info spawn args '/root/.cache/node-gyp/20.12.2/include/node/common.gypi',
95.48 npm ERR! gyp info spawn args '-Dlibrary=shared_library',
95.48 npm ERR! gyp info spawn args '-Dvisibility=default',
95.48 npm ERR! gyp info spawn args '-Dnode_root_dir=/root/.cache/node-gyp/20.12.2',
95.48 npm ERR! gyp info spawn args '-Dnode_gyp_dir=/home/node/app/node_modules/node-gyp',
95.48 npm ERR! gyp info spawn args '-Dnode_lib_file=/root/.cache/node-gyp/20.12.2/<(target_arch)/node.lib',
95.48 npm ERR! gyp info spawn args '-Dmodule_root_dir=/home/node/app/node_modules/lz4',
95.48 npm ERR! gyp info spawn args '-Dnode_engine=v8',
95.48 npm ERR! gyp info spawn args '--depth=.',
95.48 npm ERR! gyp info spawn args '--no-parallel',
95.48 npm ERR! gyp info spawn args '--generator-output',
95.48 npm ERR! gyp info spawn args 'build',
95.48 npm ERR! gyp info spawn args '-Goutput_dir=.'
95.48 npm ERR! gyp info spawn args ]
95.48 npm ERR! gyp info spawn make
95.48 npm ERR! gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
95.48 npm ERR! In file included from /root/.cache/node-gyp/20.12.2/include/node/v8.h:29,
95.48 npm ERR!                  from /root/.cache/node-gyp/20.12.2/include/node/node.h:73,
95.48 npm ERR!                  from ../lib/binding/lz4_binding.cc:4:
95.48 npm ERR! /root/.cache/node-gyp/20.12.2/include/node/v8-debug.h:26:3: error: 'Location' does not name a type; did you mean 'Function'?
95.48 npm ERR!    Location GetLocation() const;
95.48 npm ERR!    ^~~~~~~~
95.48 npm ERR!    Function
95.48 npm ERR! /root/.cache/node-gyp/20.12.2/include/node/v8-debug.h: In member function 'int v8::StackFrame::GetLineNumber() const':
95.48 npm ERR! /root/.cache/node-gyp/20.12.2/include/node/v8-debug.h:34:38: error: 'GetLocation' was not declared in this scope
95.48 npm ERR!    int GetLineNumber() const { return GetLocation().GetLineNumber() + 1; }
95.48 npm ERR!                                       ^~~~~~~~~~~
95.48 npm ERR! /root/.cache/node-gyp/20.12.2/include/node/v8-debug.h: In member function 'int v8::StackFrame::GetColumn() const':
95.48 npm ERR! /root/.cache/node-gyp/20.12.2/include/node/v8-debug.h:43:34: error: 'GetLocation' was not declared in this scope
95.48 npm ERR!    int GetColumn() const { return GetLocation().GetColumnNumber() + 1; }
95.48 npm ERR!                                   ^~~~~~~~~~~
95.48 npm ERR! ../lib/binding/lz4_binding.cc: In function 'Nan::NAN_METHOD_RETURN_TYPE LZ4Uncompress_fast(Nan::NAN_METHOD_ARGS_TYPE)':
95.48 npm ERR! ../lib/binding/lz4_binding.cc:364:85: warning: 'int LZ4_decompress_fast(const char*, char*, int)' is deprecated: This function is deprecated and unsafe. Consider using LZ4_decompress_safe() instead [-Wdeprecated-declarations]
95.48 npm ERR!                                                                Buffer::Length(output))
95.48 npm ERR!                                                                                      ^
95.48 npm ERR! In file included from ../lib/binding/lz4_binding.cc:8:
95.48 npm ERR! ../lib/binding/../../deps/lz4/lib/lz4.h:744:16: note: declared here
95.48 npm ERR!  LZ4LIB_API int LZ4_decompress_fast (const char* src, char* dst, int originalSize);
95.48 npm ERR!                 ^~~~~~~~~~~~~~~~~~~
95.48 npm ERR! ../lib/binding/lz4_binding.cc:364:85: warning: 'int LZ4_decompress_fast(const char*, char*, int)' is deprecated: This function is deprecated and unsafe. Consider using LZ4_decompress_safe() instead [-Wdeprecated-declarations]
95.48 npm ERR!                                                                Buffer::Length(output))
95.48 npm ERR!                                                                                      ^
95.48 npm ERR! In file included from ../lib/binding/lz4_binding.cc:8:
95.48 npm ERR! ../lib/binding/../../deps/lz4/lib/lz4.h:744:16: note: declared here
95.48 npm ERR!  LZ4LIB_API int LZ4_decompress_fast (const char* src, char* dst, int originalSize);
95.48 npm ERR!                 ^~~~~~~~~~~~~~~~~~~
95.48 npm ERR! ../lib/binding/lz4_binding.cc: In function 'Nan::NAN_METHOD_RETURN_TYPE LZ4Uncompress(Nan::NAN_METHOD_ARGS_TYPE)':
95.48 npm ERR! ../lib/binding/lz4_binding.cc:333:67: warning: this statement may fall through [-Wimplicit-fallthrough=]
95.48 npm ERR!      sIdx = info[2]->Uint32Value(Nan::GetCurrentContext()).FromJust();
95.48 npm ERR!             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~
95.48 npm ERR! ../lib/binding/lz4_binding.cc:334:3: note: here
95.48 npm ERR!    case 2:
95.48 npm ERR!    ^~~~
95.48 npm ERR! make: *** [lz4.target.mk:112: Release/obj.target/lz4/lib/binding/lz4_binding.o] Error 1
95.48 npm ERR! gyp ERR! build error 
95.48 npm ERR! gyp ERR! stack Error: `make` failed with exit code: 2
95.48 npm ERR! gyp ERR! stack at ChildProcess.<anonymous> (/home/node/app/node_modules/node-gyp/lib/build.js:209:23)
95.48 npm ERR! gyp ERR! System Linux 6.2.0-1018-aws
95.48 npm ERR! gyp ERR! command "/usr/local/bin/node" "/home/node/app/node_modules/.bin/node-gyp" "rebuild"
95.48 npm ERR! gyp ERR! cwd /home/node/app/node_modules/lz4
95.48 npm ERR! gyp ERR! node -v v20.12.2
95.48 npm ERR! gyp ERR! node-gyp -v v10.0.1
95.48 npm ERR! gyp ERR! not ok