Install ignores existing librdkafka

[iftheshoefritz]

This gem appears to try building librdkafka regardless of its presence already. Unfortunately the librdkafka build process it uses seems to assume /bin/bash and breaks without it.

If the gem build process could check for the presence of librdkafka, it wouldn't be necessary to force users to install bash on their servers. In my case there is already a librdkafka package available for Alpine linux and I'd rather use that than install bash just for this.

[thijsc]

We don't have plans to detect librdkafka on the machine. Would using /bin/sh work better for you?

[iftheshoefritz]

/bin/sh would work for me and is more universal, I think.

[leandrofinger]

Is there a way to skip librdkafka build?

I'm on a M1 Pro and it fails when running gem install rdkafka but I could install the librdkafka through Homebrew

[mikesnare]

Same issue here. Can't install this gem on M1. Any fix available?

[thijsc]

@mikesnare what error do you get?

[leandrofinger]

@mikesnare what error do you get?

This is what I get:

Running 'configure' for librdkafka 1.8.2... OK
Running 'compile' for librdkafka 1.8.2... ERROR, review '/Users/finger/.rvm/gems/ruby-2.7.4/gems/rdkafka-0.11.1/ext/tmp/arm64-apple-darwin21.4.0/ports/librdkafka/1.8.2/compile.log' to see what happened. Last lines are:
========================================================================
      _rd_kafka_ssl_ctx_init in rdkafka_ssl.o
      _rd_kafka_transport_ssl_cert_verify_cb in rdkafka_ssl.o
  "_X509_new", referenced from:
      _rd_kafka_ssl_ctx_init in rdkafka_ssl.o
  "_X509_verify_cert_error_string", referenced from:
      _rd_kafka_transport_ssl_handshake in rdkafka_ssl.o
  "_d2i_PKCS12_bio", referenced from:
      _rd_kafka_conf_set_ssl_cert in rdkafka_cert.o
  "_d2i_PKCS12_fp", referenced from:
      _rd_kafka_ssl_ctx_init in rdkafka_ssl.o
  "_d2i_PrivateKey_bio", referenced from:
      _rd_kafka_conf_set_ssl_cert in rdkafka_cert.o
  "_d2i_X509_bio", referenced from:
      _rd_kafka_conf_set_ssl_cert in rdkafka_cert.o
  "_i2d_X509", referenced from:
      _rd_kafka_transport_ssl_cert_verify_cb in rdkafka_ssl.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [librdkafka.1.dylib] Error 1
make: *** [libs] Error 2
========================================================================
rake aborted!
Failed to complete compile task
/Users/finger/.rvm/gems/ruby-2.7.4/gems/mini_portile2-2.8.0/lib/mini_portile2/mini_portile.rb:460:in `block in execute'
/Users/finger/.rvm/gems/ruby-2.7.4/gems/mini_portile2-2.8.0/lib/mini_portile2/mini_portile.rb:426:in `chdir'
/Users/finger/.rvm/gems/ruby-2.7.4/gems/mini_portile2-2.8.0/lib/mini_portile2/mini_portile.rb:426:in `execute'
/Users/finger/.rvm/gems/ruby-2.7.4/gems/mini_portile2-2.8.0/lib/mini_portile2/mini_portile.rb:144:in `compile'
/Users/finger/.rvm/gems/ruby-2.7.4/gems/mini_portile2-2.8.0/lib/mini_portile2/mini_portile.rb:187:in `cook'
/Users/finger/.rvm/gems/ruby-2.7.4/gems/rdkafka-0.11.1/ext/Rakefile:22:in `block in <top (required)>'
/Users/finger/.rvm/gems/ruby-2.7.4/gems/rake-13.0.6/exe/rake:27:in `<main>'
Tasks: TOP => default
(See full trace by running task with --trace)

rake failed, exit code 1

[mikesnare]

@thijsc Here's the error:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/msnare/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/rdkafka-0.10.0/ext
/Users/msnare/.rbenv/versions/2.7.5/bin/ruby -I/Users/msnare/.rbenv/versions/2.7.5/lib/ruby/2.7.0/rubygems -rrubygems /Users/msnare/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake
RUBYARCHDIR\=/Users/msnare/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/extensions/arm64-darwin-21/2.7.0/rdkafka-0.10.0 RUBYLIBDIR\=/Users/msnare/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/extensions/arm64-darwin-21/2.7.0/rdkafka-0.10.0

Extracting v1.5.0 into tmp/arm64-apple-darwin21.4.0/ports/librdkafka/1.5.0... OK
Running 'configure' for librdkafka 1.5.0... OK
Running 'compile' for librdkafka 1.5.0... ERROR, review '/Users/msnare/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/rdkafka-0.10.0/ext/tmp/arm64-apple-darwin21.4.0/ports/librdkafka/1.5.0/compile.log' to see what happened. Last lines are:
========================================================================
      _rd_kafka_txn_coord_query in librdkafka.a(rdkafka_txnmgr.o)
      _rd_kafka_txn_coord_set in librdkafka.a(rdkafka_txnmgr.o)
      _rd_kafka_txn_handle_AddPartitionsToTxn in librdkafka.a(rdkafka_txnmgr.o)
      _rd_kafka_txn_handle_FindCoordinator in librdkafka.a(rdkafka_txnmgr.o)
      _rd_kafka_txns_term in librdkafka.a(rdkafka_txnmgr.o)
      _rd_kafka_toppar_dump.constprop.0 in librdkafka.a(rdkafka.o)
      ...
  "___aarch64_ldadd8_acq_rel", referenced from:
      _rd_kafka_toppar_dump.constprop.0 in librdkafka.a(rdkafka.o)
      _rd_kafka_broker_dump in librdkafka.a(rdkafka.o)
      _rd_kafka_stats_emit_toppar in librdkafka.a(rdkafka.o)
      _rd_kafka_stats_emit_tmr_cb in librdkafka.a(rdkafka.o)
      _rd_kafka_msg_partitioner in librdkafka.a(rdkafka_msg.o)
      _rd_kafka_broker_conn_closed in librdkafka.a(rdkafka_broker.o)
      _rd_kafka_broker_timeout_scan in librdkafka.a(rdkafka_broker.o)
      ...
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[1]: *** [rdkafka_example_cpp] Error 1
make: *** [examples] Error 2
========================================================================
rake aborted!
Failed to complete compile task

Tasks: TOP => default
(See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in /Users/msnare/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/gems/rdkafka-0.10.0 for inspection.
Results logged to /Users/msnare/.rbenv/versions/2.7.5/lib/ruby/gems/2.7.0/extensions/arm64-darwin-21/2.7.0/rdkafka-0.10.0/gem_make.out

[trickleup]

Does anyone have a workaround that allows them to install the gem on an M1 mac?

[clupprich]

I don't remember how I did it, but I have it installed. Must have been rather uneventful, otherwise I would probably remember. Let me see if I can refresh my memory today.

[mikesnare]

Still no resolution to this after 6 months?

[thijsc]

We don't have a use case for using a system version of the library ourselves. If you want to take on the work to make that happen I'm happy to help it get shipped!

[mikesnare]

@thijsc If any of us posting here knew what needed to be done I don't think we'd be posting here. What you're saying, I think, is that this library isn't usable to people on M1 macs? Meaning racecar -- which requires this -- can't be used by us?

[thijsc]

This issue is not about rdkafka not being usable on M1, I regularly use it on my M1 air myself by the way so afaik that should work. If there is a problem with that we should open an issue to track that. The issue is about using a system version of the C lib. I'm not planning to work on that specifically.

[mikesnare]

@thijsc Fair enough, but 3 of the comments in this post are specifically about people on M1s not being able to install this gem on M1s. I can open a separate issue for that.

[thijsc]

Also fair enough from my end then :-). I think this fell of my radar because I wasn't planning to work on the issue as it was originally opened. I'm actually going to close this one since I don't think anybody is going to push on that. Feel free to open an issue about M1 problems!