search

karafka / rdkafka-ruby

Modern and performant Kafka client library for Ruby based on librdkafka
https://karafka.io
Other
349 stars 120 forks source link

Native extensions fail to build in official AWS Lambda ruby environment #270

Closed wjensen-godaddy closed 9 months ago

wjensen-godaddy commented 1 year ago

We are deploying our app to AWS lambda, and we bundle our gems using the officially provided Docker image. However, I haven't been able to solve the following error: 

Fetching gem metadata from https://rubygems.org/.....
Resolving dependencies...
Using rake 13.0.6
Using bundler 2.4.15
Fetching ffi 1.15.5
Fetching mini_portile2 2.8.2
Installing mini_portile2 2.8.2
Installing ffi 1.15.5 with native extensions
Fetching rdkafka 0.12.0
Installing rdkafka 0.12.0 with native extensions
Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext
/var/lang/bin/ruby -rrubygems /var/lang/lib/ruby/gems/3.2.0/gems/rake-13.0.6/exe/rake RUBYARCHDIR\=/var/lang/lib/ruby/gems/3.2.0/extensions/x86_64-linux/3.2.0/rdkafka-0.12.0
RUBYLIBDIR\=/var/lang/lib/ruby/gems/3.2.0/extensions/x86_64-linux/3.2.0/rdkafka-0.12.0
Downloading v1.9.0 (100%) 
Extracting v1.9.0 into tmp/x86_64-redhat-linux/ports/librdkafka/1.9.0... OK
Running 'configure' for librdkafka 1.9.0... OK
Running 'compile' for librdkafka 1.9.0... OK
Running 'install' for librdkafka 1.9.0... ERROR. Please review logs to see what happened:
----- contents of '/var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext/tmp/x86_64-redhat-linux/ports/librdkafka/1.9.0/install.log' -----
make[1]: Entering directory `/var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext/tmp/x86_64-redhat-linux/ports/librdkafka/1.9.0/librdkafka-1.9.0/src'
Install librdkafka to /var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext/ports/x86_64-redhat-linux/librdkafka/1.9.0
d $DESTDIR/var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext/ports/x86_64-redhat-linux/librdkafka/1.9.0/include/librdkafka
/bin/sh: d: command not found
make[1]: [lib-install] Error 127 (ignored)
d $DESTDIR/var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext/ports/x86_64-redhat-linux/librdkafka/1.9.0/lib
/bin/sh: d: command not found
make[1]: [lib-install] Error 127 (ignored)
rdkafka.h rdkafka_mock.h $DESTDIR/var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext/ports/x86_64-redhat-linux/librdkafka/1.9.0/include/librdkafka
/bin/sh: rdkafka.h: command not found
make[1]: *** [lib-install] Error 127
make[1]: Leaving directory `/var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0/ext/tmp/x86_64-redhat-linux/ports/librdkafka/1.9.0/librdkafka-1.9.0/src'
make: *** [install-subdirs] Error 2
----- end of file -----
rake aborted!
Failed to complete install task

Tasks: TOP => default
(See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in /var/lang/lib/ruby/gems/3.2.0/gems/rdkafka-0.12.0 for inspection.
Results logged to /var/lang/lib/ruby/gems/3.2.0/extensions/x86_64-linux/3.2.0/rdkafka-0.12.0/gem_make.out

  /var/lang/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:118:in `run'
  /var/lang/lib/ruby/site_ruby/3.2.0/rubygems/ext/rake_builder.rb:30:in `build'
  /var/lang/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:186:in `build_extension'
  /var/lang/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:220:in `block in build_extensions'
  /var/lang/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:217:in `each'
  /var/lang/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:217:in `build_extensions'
  /var/lang/lib/ruby/site_ruby/3.2.0/rubygems/installer.rb:845:in `build_extensions'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/rubygems_gem_installer.rb:72:in `build_extensions'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/rubygems_gem_installer.rb:28:in `install'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/source/rubygems.rb:201:in `install'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/installer/gem_installer.rb:54:in `install'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/installer/gem_installer.rb:16:in `install_from_spec'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/installer/parallel_installer.rb:156:in `do_install'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/installer/parallel_installer.rb:147:in `block in worker_pool'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/worker.rb:62:in `apply_func'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/worker.rb:57:in `block in process_queue'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/worker.rb:54:in `loop'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/worker.rb:54:in `process_queue'
  /var/lang/lib/ruby/site_ruby/3.2.0/bundler/worker.rb:90:in `block (2 levels) in create_threads'

An error occurred while installing rdkafka (0.12.0), and Bundler cannot continue.

In Gemfile:
  rdkafka

Example Dockerfile to reproduce the issue: 

FROM public.ecr.aws/lambda/ruby:3.2

RUN yum -y groupinstall 'Development Tools'
RUN yum -y install librdkafka librdkafka-devel

RUN gem install rdkafka

EDIT

I confirmed the header file mentioned in the error exists:

sh-4.2# rpm -ql librdkafka-devel
/usr/include/librdkafka
/usr/include/librdkafka/rdkafka.h
/usr/include/librdkafka/rdkafkacpp.h
/usr/lib64/librdkafka++.so
/usr/lib64/librdkafka.so
/usr/lib64/pkgconfig/rdkafka++-static.pc
/usr/lib64/pkgconfig/rdkafka++.pc
/usr/lib64/pkgconfig/rdkafka-static.pc
/usr/lib64/pkgconfig/rdkafka.pc

I even tried running the following:

export C_INCLUDE_PATH=/usr/include
export LIBRARY_PATH=/usr/lib64
mensfeld commented 1 year ago

@wjensen-godaddy can you try https://rubygems.org/gems/rdkafka/versions/0.13.0.beta.8?

We should release stable soon but just to check if this solves the problem. As far as I recall there was some work around building it.

wjensen-godaddy commented 1 year ago

@mensfeld I just tried the beta and got the same error. I also tried a different base docker image provided by Amazon for their SAM framework. 

FROM public.ecr.aws/sam/build-ruby3.2

RUN yum -y groupinstall 'Development Tools'
RUN yum -y install librdkafka librdkafka-devel

RUN gem install rdkafka --pre
mensfeld commented 1 year ago

Ok. Will take care of this but I won't be able to fix this that week as I am on holiday.

mensfeld commented 11 months ago

@wjensen-godaddy can you try it with 0.14.0? :sweat_smile: if still failing, will look into it

mensfeld commented 9 months ago

Closing due to lack of reply. Feel free to reopen if still a problem in 0.15.0 or higher.

vestedpr-dev commented 8 months ago

Hello, I have the exact same exception, except using OSX Monterey on Intel. Long and short, regardless of which version I try installing etc of karafka or its supporting librdkafka installation gems, I end up with this build failure:

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10/ext
/Users/username/.asdf/installs/ruby/3.2.3/bin/ruby -rrubygems /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/rake-13.1.0/exe/rake
RUBYARCHDIR\=/Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/extensions/x86_64-darwin-21/3.2.0/karafka-rdkafka-0.14.10 RUBYLIBDIR\=/Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/extensions/x86_64-darwin-21/3.2.0/karafka-rdkafka-0.14.10
Downloading v2.3.0 (  0%) 
<snip>
Downloading v2.3.0 (100%) 
Extracting v2.3.0 into tmp/x86_64-apple-darwin21.6.0/ports/librdkafka/2.3.0... OK
Running 'configure' for librdkafka 2.3.0... OK
Running 'compile' for librdkafka 2.3.0... OK
Running 'install' for librdkafka 2.3.0... ERROR. Please review logs to see what happened:
----- contents of '/Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10/ext/tmp/x86_64-apple-darwin21.6.0/ports/librdkafka/2.3.0/install.log' -----
Install librdkafka to /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10/ext/ports/x86_64-apple-darwin21.6.0/librdkafka/2.3.0
d $DESTDIR/Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10/ext/ports/x86_64-apple-darwin21.6.0/librdkafka/2.3.0/include/librdkafka
/bin/sh: d: command not found
make[1]: [lib-install] Error 127 (ignored)
d $DESTDIR/Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10/ext/ports/x86_64-apple-darwin21.6.0/librdkafka/2.3.0/lib
/bin/sh: d: command not found
make[1]: [lib-install] Error 127 (ignored)
rdkafka.h rdkafka_mock.h $DESTDIR/Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10/ext/ports/x86_64-apple-darwin21.6.0/librdkafka/2.3.0/include/librdkafka
/bin/sh: rdkafka.h: command not found
make[1]: *** [lib-install] Error 127
make: *** [install-subdirs] Error 2
----- end of file -----
rake aborted!
Failed to complete install task

Tasks: TOP => default
(See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10 for inspection.
Results logged to /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/extensions/x86_64-darwin-21/3.2.0/karafka-rdkafka-0.14.10/gem_make.out

  /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:125:in `run'
  /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/site_ruby/3.2.0/rubygems/ext/rake_builder.rb:30:in `build'
  /Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/site_ruby/3.2.0/rubygems/ext/builder.rb:193:in `build_extension'

In particular, zoom in on these lines:

d $DESTDIR/Users/username/.asdf/installs/ruby/3.2.3/lib/ruby/gems/3.2.0/gems/karafka-rdkafka-0.14.10/ext/ports/x86_64-apple-darwin21.6.0/librdkafka/2.3.0/include/librdkafka
/bin/sh: d: command not found

So I go deep and splunk into the source files and into the Makefile and eventually get to this line here: 

$(INSTALL) -d $$DESTDIR$(includedir)/$(PKGNAME)

https://github.com/confluentinc/librdkafka/blob/master/mklove/Makefile.base#L269

Essentially what has happened is that the $(INSTALL) env var is empty, and shell is trying to execute the remainder of the line. Nearer to the top of that Makefile it sets it conditionally: 

INSTALL?=       install

https://github.com/confluentinc/librdkafka/blob/master/mklove/Makefile.base#L51

Why me why is this happening? INSTALL does not appear to be set in my environment. Hmm. Well, so I try just setting it manually as an ENV var before running bundle, and miraculously that works.

So -- for future devs, I suggest there may be strange environmental issues at play here, but if you get just this exact stack trace, try this, it may work:

INSTALL=install gem install karafka-rdkafka
mensfeld commented 8 months ago

@vestedpr-dev would you mind PRing fix for this?

vestedpr-dev commented 8 months ago

Is it something that should be done do you think? Others may want to be able to override it for themselves? I think it must be a strange environmental issue just with my machine.

However if you think it would be good to fix, would you put it somewhere like this? https://github.com/karafka/rdkafka-ruby/blob/main/ext/Rakefile#L51 Considering how far downstream your gem is, it probably wouldn't do too much harm. However I am not an expert on all the places and environments that rdkafka-ruby gets into.

On Tue, Feb 20, 2024 at 2:07 PM Maciej Mensfeld @.***> wrote:

@vestedpr-dev https://github.com/vestedpr-dev would you mind PRing fix for this?

— Reply to this email directly, view it on GitHub https://github.com/karafka/rdkafka-ruby/issues/270#issuecomment-1954971193, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICQC4SAS2VJOULMREBLMUTYUT67RAVCNFSM6AAAAAA2NTUUFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJUHE3TCMJZGM . You are receiving this because you were mentioned.Message ID: @.***>

mensfeld commented 8 months ago

Is it something that should be done do you think? Others may want to be able to override it for themselves?

I think that the spirit of Ruby is to make ppls lives enjoyable :) we clearly did not do a great job in your case, so why not make it better!

I will look into this issue tomorrow and tell you if what you proposed is correct. Thank you for bringing this to my attention.

vestedpr-dev commented 8 months ago

I actually got the same error on my other development machine. I never saw it before even though I have had your gem installed for months. This makes me think it may be more than just a weird environmental issue, although it might have to do with the constellation of gems I have installed. According to my theory, any other thing which sets the bash "INSTALL" environment variable to a blank string, will cause the failure.

Point is: if you want a patch for this, I'll prepare one, because otherwise I'll have to tell all my developers to prefix INSTALL=install to their bundle commands.

On Tue, Feb 20, 2024 at 2:14 PM Maciej Mensfeld @.***> wrote:

Is it something that should be done do you think? Others may want to be able to override it for themselves?

I think that the spirit of Ruby is to make ppls lives enjoyable :) we clearly did not do a great job in your case, so why not make it better!

I will look into this issue tomorrow and tell you if what you proposed is correct. Thank you for bringing this to my attention.

— Reply to this email directly, view it on GitHub https://github.com/karafka/rdkafka-ruby/issues/270#issuecomment-1954983096, or unsubscribe https://github.com/notifications/unsubscribe-auth/AICQC4SA4GWK5B3B36XFOCDYUT733AVCNFSM6AAAAAA2NTUUFKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJUHE4DGMBZGY . You are receiving this because you were mentioned.Message ID: @.***>

mensfeld commented 7 months ago

@vestedpr-dev if you could provide me with clear step-by-step explanation how to reproduce I would love this as well because then I could double-check the potential solution

mensfeld commented 7 months ago

@vestedpr-dev fiendly ping :)