search

confluentinc / confluent-kafka-javascript

Confluent's Apache Kafka JavaScript client
https://www.npmjs.com/package/@confluentinc/kafka-javascript
MIT License
80 stars 7 forks source link

Install Script stalls indefinitely, shows logs after terminating process #103

Open FlacoJones opened 1 month ago

FlacoJones commented 1 month ago

Environment Information

Steps to Reproduce

  1. Add "@confluentinc/kafka-javascript": "^0.2.0" to package.json.
  2. Run pnpm install in the

confluent-kafka-javascript Configuration Settings Only added to package.json.

Additional context

We are using PNPM and Turbo for our monorepo. When I run pnpm install in the workspace root, it stalls indefinitely on "Running install script.

Upon Ctrl+C to terminate the install, it shows these logs all at once:

Logs

nonode_modules/.pnpm/@confluentinc+kafka-javascript@0.2.0/node_modules/@confluentinc/kafka-javascript: Running install script, failed in 3m 47.3s
.../@confluentinc/kafka-javascript install$ node-pre-gyp install --fallback-to-build
│ node-pre-gyp info it worked if it ends with ok
│ node-pre-gyp info using node-pre-gyp@1.0.11
│ node-pre-gyp info using node@20.15.0 | darwin | x64
│ node-pre-gyp info check checked for "/Users/alo/OpenQ-Fullstack/OpenQ-DRM/node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.0/node_modules/@con…
│ node-pre-gyp http GET https://github.com/confluentinc/confluent-kafka-javascript/releases/download/v0.2.0/confluent-kafka-javascript-v0.2.0-node-v1…
│ node-pre-gyp ERR! install response status 404 Not Found on https://github.com/confluentinc/confluent-kafka-javascript/releases/download/v0.2.0/conf…
│ node-pre-gyp WARN Pre-built binaries not installable for @confluentinc/kafka-javascript@v0.2.0 and node@20.15.0 (node-v115 ABI, unknown) (falling b…
│ node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/confluentinc/confluent-kafka-javascript/releases/download/v0.2.0/co…
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@9.4.1
│ gyp info using node@20.15.0 | darwin | x64
│ gyp info ok 
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@9.4.1
│ gyp info using node@20.15.0 | darwin | x64
│ gyp info find Python using Python version 3.7.9 found at "/usr/local/bin/python3"
│ gyp info spawn /usr/local/bin/python3
│ gyp info spawn args [
│ gyp info spawn args   '/usr/local/Cellar/pnpm/8.13.1/libexec/dist/node_modules/node-gyp/gyp/gyp_main.py',
│ gyp info spawn args   'binding.gyp',
│ gyp info spawn args   '-f',
│ gyp info spawn args   'make',
│ gyp info spawn args   '-I',
│ gyp info spawn args   '/Users/alo/OpenQ-Fullstack/OpenQ-DRM/node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.0/node_modules/@confluentinc/kafk…
│ gyp info spawn args   '-I',
│ gyp info spawn args   '/usr/local/Cellar/pnpm/8.13.1/libexec/dist/node_modules/node-gyp/addon.gypi',
│ gyp info spawn args   '-I',
│ gyp info spawn args   '/Users/alo/Library/Caches/node-gyp/20.15.0/include/node/common.gypi',
│ gyp info spawn args   '-Dlibrary=shared_library',
│ gyp info spawn args   '-Dvisibility=default',
│ gyp info spawn args   '-Dnode_root_dir=/Users/alo/Library/Caches/node-gyp/20.15.0',
│ gyp info spawn args   '-Dnode_gyp_dir=/usr/local/Cellar/pnpm/8.13.1/libexec/dist/node_modules/node-gyp',
│ gyp info spawn args   '-Dnode_lib_file=/Users/alo/Library/Caches/node-gyp/20.15.0/<(target_arch)/node.lib',
│ gyp info spawn args   '-Dmodule_root_dir=/Users/alo/OpenQ-Fullstack/OpenQ-DRM/node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.0/node_modules/…
│ gyp info spawn args   '-Dnode_engine=v8',
│ gyp info spawn args   '--depth=.',
│ gyp info spawn args   '--no-parallel',
│ gyp info spawn args   '--generator-output',
│ gyp info spawn args   'build',
│ gyp info spawn args   '-Goutput_dir=.'
│ gyp info spawn args ]
│ gyp info ok 
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@9.4.1
│ gyp info using node@20.15.0 | darwin | x64
│ gyp info spawn make
│ gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
│   ACTION deps_librdkafka_gyp_librdkafka_target_configure deps/librdkafka/config.h
│ checking for OS or distribution... ok (osx)
│ checking for C compiler from CC env... failed
│ checking for gcc (by command)... ok
│ checking for C++ compiler from CXX env... failed
│ checking for C++ compiler (g++)... ok
│ checking executable ld... ok
│ checking executable nm... ok
│ checking executable objdump... ok
│ checking executable strip... ok
│ checking executable libtool... ok
│ checking executable ranlib... ok
│ checking for debug symbols compiler flag (-g...)... ok
│ checking for pkgconfig (by command)... ok
│ checking for install (by command)... failed
│ checking for GNU ar... failed (disable)
│ checking for PIC (by compile)... ok
│ checking for GNU-compatible linker options... failed
│ checking for OSX linker options... ok
│ checking for GNU linker-script ld flag... failed
│ checking for Solaris linker-script ld flag... failed (ignore)
│ checking for __atomic_32 (by compile)... ok
│ checking for __atomic_64 (by compile)... ok
│ checking for socket (by compile)... ok
│ parsing version '0x020503ff'... ok (2.5.3)
│ checking for librt (by pkg-config)... failed
│ checking for librt (by compile)... failed
│ checking for libpthread (by pkg-config)... failed
│ checking for libpthread (by compile)... ok
│ checking for c11threads (by pkg-config)... failed
│ checking for c11threads (by compile)... failed (disable)
│ checking for libdl (by pkg-config)... failed
│ checking for libdl (by compile)... ok
│ building dependency zlib... ok (from source)
│ checking for zlib (by pkg-config)... failed (ignore)
│ checking for zlib (by compile)... ok
│ make: *** [deps/librdkafka/config.h] Interrupt: 2
│ building dependency libcrypto...
└─ Failed in 3m 47.3s at /Users/alo/OpenQ-Fullstack/OpenQ-DRM/node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.0/node_modules/@confluentinc/kafka-javascript

It appears to fail on building dependency libcrypto... when i hit deps/librdkafka/config.h

milindl commented 1 month ago

At a glance, it seems like you're using a non-M1 mac, is that correct?

If so, we don't provide pre-built packages for that (for macOS, we only provide it for M1), so in this case, npm install is trying to build librdkafka from scratch, which is not only time consuming, but a bunch of dependencies are required for that.

I can suggest doing this:

Let me know if that helps, or if you're getting issues.

FlacoJones commented 1 month ago

That worked! Thank you very much.

dqiubread commented 3 weeks ago

Im still getting an error when i do npm install. I followed the steps but i get this error now.

npm verbose stack Error: command failed
npm verbose stack     at promiseSpawn (/Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/@npmcli/promise-spawn/lib/index.js:22:22)
npm verbose stack     at spawnWithShell (/Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/@npmcli/promise-spawn/lib/index.js:124:10)
npm verbose stack     at promiseSpawn (/Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/@npmcli/promise-spawn/lib/index.js:12:12)
npm verbose stack     at runScriptPkg (/Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/@npmcli/run-script/lib/run-script-pkg.js:77:13)
npm verbose stack     at runScript (/Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/@npmcli/run-script/lib/run-script.js:9:12)
npm verbose stack     at /Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/@npmcli/arborist/lib/arborist/rebuild.js:329:17
npm verbose stack     at run (/Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/promise-call-limit/dist/commonjs/index.js:67:22)
npm verbose stack     at /Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/promise-call-limit/dist/commonjs/index.js:84:9
npm verbose stack     at new Promise (<anonymous>)
npm verbose stack     at callLimit (/Users/davidqiu/.volta/tools/image/npm/10.8.1/node_modules/promise-call-limit/dist/commonjs/index.js:35:69)
npm verbose pkgid @confluentinc/kafka-javascript@0.3.0
npm error code 1
.../@confluentinc/kafka-javascript install$ node-pre-gyp install --fallback-to-build
│ node-pre-gyp info it worked if it ends with ok
│ node-pre-gyp info using node-pre-gyp@1.0.11
│ node-pre-gyp info using node@20.16.0 | darwin | x64
│ node-pre-gyp info check checked for "/node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.1/node_modules/@co…
│ node-pre-gyp http GET https://github.com/confluentinc/confluent-kafka-javascript/releases/download/v0.2.1/confluent-kafka-javascript-v0.2.1-node-v115-darwin-unknown-x64.tar.gz
│ node-pre-gyp ERR! install response status 404 Not Found on https://github.com/confluentinc/confluent-kafka-javascript/releases/download/v0.2.1/confluent-kafka-javascript-v0.2.1-…
│ node-pre-gyp WARN Pre-built binaries not installable for @confluentinc/kafka-javascript@v0.2.1 and node@20.16.0 (node-v115 ABI, unknown) (falling back to source compile with nod…
│ node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/confluentinc/confluent-kafka-javascript/releases/download/v0.2.1/confluent-kafka-javascript-v0.2.…
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@10.2.0
│ gyp info using node@20.16.0 | darwin | x64
│ gyp info ok 
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@10.2.0
│ gyp info using node@20.16.0 | darwin | x64
│ gyp info find Python using Python version 3.11.3 found at "/usr/local/opt/python@3.11/bin/python3.11"
│ gyp info spawn /usr/local/opt/python@3.11/bin/python3.11
│ gyp info spawn args [
│ gyp info spawn args '/Users/davidqiu/.volta/tools/image/packages/pnpm/lib/node_modules/pnpm/dist/node_modules/node-gyp/gyp/gyp_main.py',
│ gyp info spawn args 'binding.gyp',
│ gyp info spawn args '-f',
│ gyp info spawn args 'make',
│ gyp info spawn args '-I',
│ gyp info spawn args '/node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.1/node_modules/@confluentinc/kafka…
│ gyp info spawn args '-I',
│ gyp info spawn args '/Users/davidqiu/.volta/tools/image/packages/pnpm/lib/node_modules/pnpm/dist/node_modules/node-gyp/addon.gypi',
│ gyp info spawn args '-I',
│ gyp info spawn args '/Users/davidqiu/Library/Caches/node-gyp/20.16.0/include/node/common.gypi',
│ gyp info spawn args '-Dlibrary=shared_library',
│ gyp info spawn args '-Dvisibility=default',
│ gyp info spawn args '-Dnode_root_dir=/Users/davidqiu/Library/Caches/node-gyp/20.16.0',
│ gyp info spawn args '-Dnode_gyp_dir=/Users/davidqiu/.volta/tools/image/packages/pnpm/lib/node_modules/pnpm/dist/node_modules/node-gyp',
│ gyp info spawn args '-Dnode_lib_file=/Users/davidqiu/Library/Caches/node-gyp/20.16.0/<(target_arch)/node.lib',
│ gyp info spawn args '-Dmodule_root_dir=/node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.1/node_modules/@…
│ gyp info spawn args '-Dnode_engine=v8',
│ gyp info spawn args '--depth=.',
│ gyp info spawn args '--no-parallel',
│ gyp info spawn args '--generator-output',
│ gyp info spawn args 'build',
│ gyp info spawn args '-Goutput_dir=.'
│ gyp info spawn args ]
│ gyp info ok 
│ gyp info it worked if it ends with ok
│ gyp info using node-gyp@10.2.0
│ gyp info using node@20.16.0 | darwin | x64
│ gyp info spawn make
│ gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
│   CXX(target) Release/obj.target/confluent-kafka-javascript/src/binding.o
│ In file included from ../src/binding.cc:13:
│ /node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.1/node_modules/@confluentinc/kafka-javascript/src/bindi…
│ #include "rdkafkacpp.h" // NOLINT
│          ^~~~~~~~~~~~~~
│ 1 error generated.
│ make: *** [Release/obj.target/confluent-kafka-javascript/src/binding.o] Error 1
│ gyp ERR! build error 
│ gyp ERR! stack Error: `make` failed with exit code: 2
│ gyp ERR! stack at ChildProcess.<anonymous> (/Users/davidqiu/.volta/tools/image/packages/pnpm/lib/node_modules/pnpm/dist/node_modules/node-gyp/lib/build.js:216:23)
│ gyp ERR! System Darwin 22.6.0
│ gyp ERR! command "/Users/davidqiu/.volta/tools/image/node/20.16.0/bin/node" "/Users/davidqiu/.volta/tools/image/packages/pnpm/lib/node_modules/pnpm/dist/node_modules/node-gyp/bi…
│ gyp ERR! cwd /node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.1/node_modules/@confluentinc/kafka-javascr…
│ gyp ERR! node -v v20.16.0
│ gyp ERR! node-gyp -v v10.2.0
│ gyp ERR! not ok 
│ node-pre-gyp ERR! build error 
│ node-pre-gyp ERR! stack Error: Failed to execute '/Users/davidqiu/.volta/tools/image/node/20.16.0/bin/node /Users/davidqiu/.volta/tools/image/packages/pnpm/lib/node_modules/pnpm…
│ node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/node_modules/.pnpm/@mapbox+node-pre-gyp@1.0.11/node_…
│ node-pre-gyp ERR! stack     at ChildProcess.emit (node:events:519:28)
│ node-pre-gyp ERR! stack     at maybeClose (node:internal/child_process:1105:16)
│ node-pre-gyp ERR! stack     at ChildProcess._handle.onexit (node:internal/child_process:305:5)
│ node-pre-gyp ERR! System Darwin 22.6.0
│ node-pre-gyp ERR! command "/Users/davidqiu/.volta/tools/image/node/20.16.0/bin/node" "/node_modules/.pnpm/@map…
│ node-pre-gyp ERR! cwd /node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.1/node_modules/@confluentinc/kafk…
│ node-pre-gyp ERR! node -v v20.16.0
│ node-pre-gyp ERR! node-pre-gyp -v v1.0.11
│ node-pre-gyp ERR! not ok 
│ Failed to execute '/Users/davidqiu/.volta/tools/image/node/20.16.0/bin/node /Users/davidqiu/.volta/tools/image/packages/pnpm/lib/node_modules/pnpm/dist/node_modules/node-gyp/bin…
└─ Failed in 3.5s at /node_modules/.pnpm/@confluentinc+kafka-javascript@0.2.1/node_modules/@confluentinc/kafka-javascript
dqiubread commented 3 weeks ago

At a glance, it seems like you're using a non-M1 mac, is that correct?

If so, we don't provide pre-built packages for that (for macOS, we only provide it for M1), so in this case, npm install is trying to build librdkafka from scratch, which is not only time consuming, but a bunch of dependencies are required for that.

I can suggest doing this:

  • install librdkafka through brew: https://formulae.brew.sh/formula/librdkafka
  • set the environment variables CKJS_LINKING=dynamic and BUILD_LIBRDKAFKA=0
  • npm install as usual - the process shouldn't take more than a couple minutes.

Let me know if that helps, or if you're getting issues.

@milindl any ideas on the reason why its failing now. I followed your steps and set the env variables too but im now getting the above error.

milindl commented 3 weeks ago

At a very basic glance it seems like it can't find the header files for librdkafka. Could you try doing the following steps to verify this?

Make a program in any directory, check.c

#include <stdio.h>
#include <string.h>
#include <librdkafka/rdkafka.h>

int main() {
        rd_kafka_conf_t *conf;
        char buf[512];
        size_t sz = sizeof(buf);
        rd_kafka_conf_res_t res;

        printf("Version string: %s\n", rd_kafka_version_str());
                conf = rd_kafka_conf_new();
        res  = rd_kafka_conf_get(conf, "builtin.features", buf, &sz);

        if (res != RD_KAFKA_CONF_OK) {
                printf("ERROR: conf_get failed: %d\n", res);
                return 1;
        }

        printf("builtin.features: %s\n", buf);
        return 0;
}

And try compiling and running the program (you're using mac too right? I think clang should be present)

 clang check.c -o check -lrdkafka
 ./check

Does that work without any further issues?

wspurgin commented 6 days ago

@milindl thanks for helping to debug the apple silicon hiccups with running rdkafka.

I've installed librdkafka from brew (which has a check not unlike the one you have which one can run with brew test librdkafka).

The problem is that the lib and include locations also need to be set. So clang check.c -o check -lrdkafka won't work unless these are set via the -L and -I flags or in ENV variables. e.g.,

/usr/bin/clang check.c -L$(brew --prefix librdkafka)/lib -I$(brew --prefix librdkafka)/include -lrdkafka -o check

Theoretically, moving those into the ENVs LIBRARY_PATH and CPATH respectively, should allow you to install e.g.,:

CKJS_LINKING=dynamic BUILD_LIBRDKAFKA=0 LIBRARY_PATH=$(brew --prefix librdkafka)/lib CPATH=$(brew --prefix librdkafka)/include pnpm add '@confluentinc/kafka-javascript'

☝️ but that still fails for me with the same error (not finding the librdkafka headers).

EDIT: Also I just realized that it's complaining about #include "rdkafkacpp.h" specifically... not "librdkafka/rdkafkacpp.h".

Which made me realize that this build expects you include the specific librdkafka dir - not just the general include path. So

BUILD_LIBRDKAFKA=0 LIBRARY_PATH=$(brew --prefix)/lib CPATH=$(brew --prefix)/include/librdkafka pnpm add '@confluentinc/kafka-javascript'

Was what it was after. 🚀

milindl commented 6 days ago

Thanks for documenting your process @wspurgin , that's very informative for other users.

What node version and library version are you using? With the latest one, we've added pre-built binaries for both Intel and Apple Silicon Macs, so ideally you should be able to avoid doing the build entirely.

wspurgin commented 5 days ago

@milindl - here were my specs

dqiubread commented 5 days ago

At a very basic glance it seems like it can't find the header files for librdkafka. Could you try doing the following steps to verify this?

Make a program in any directory, check.c

#include <stdio.h>
#include <string.h>
#include <librdkafka/rdkafka.h>

int main() {
        rd_kafka_conf_t *conf;
        char buf[512];
        size_t sz = sizeof(buf);
        rd_kafka_conf_res_t res;

        printf("Version string: %s\n", rd_kafka_version_str());
                conf = rd_kafka_conf_new();
        res  = rd_kafka_conf_get(conf, "builtin.features", buf, &sz);

        if (res != RD_KAFKA_CONF_OK) {
                printf("ERROR: conf_get failed: %d\n", res);
                return 1;
        }

        printf("builtin.features: %s\n", buf);
        return 0;
}

And try compiling and running the program (you're using mac too right? I think clang should be present)

 clang check.c -o check -lrdkafka
 ./check

Does that work without any further issues?

So i did this to get it installed.

## NOTE: If you have an intel mac and your `npm install` hangs please try these steps
1.If you have a nodemodules folder remove it. 
2.Remove package-lock.json
3.Install librdkafka through brew: https://formulae.brew.sh/formula/librdkafka
4.Set the environment variables CPLUS_INCLUDE_PATH, LIBRARY_PATH, BUILD_LIBRDKAFKA=0
  1a. export CPLUS_INCLUDE_PATH="$(brew --prefix librdkafka)/include/librdkafka"
  2a. export LIBRARY_PATH="$(brew --prefix librdkafka)/lib"
  3a. export BUILD_LIBRDKAFKA=0
5. npm install -verbose
milindl commented 3 days ago

Hey @dqiubread , thanks for documenting your steps too. Did you not need to set CKJS_LINKING=dynamic as well?

dqiubread commented 3 days ago

Hey @dqiubread , thanks for documenting your steps too. Did you not need to set CKJS_LINKING=dynamic as well?

I did not need to set CKJS_LINKING. My issue was on an intel mac