search

karafka / rdkafka-ruby

Modern and performant Kafka client library for Ruby based on librdkafka
https://karafka.io
Other
345 stars 120 forks source link

Gem install is not possible on a local server where the network is blocked. #271

Open smgkoala opened 1 year ago

smgkoala commented 1 year ago

Gem install is not possible on a local server where the network is blocked.

Even though I received the gem file and gave the --local option, it seems that network communication with the outside is essential during installation. can you confirm?

https://rubygems.org/gems/rdkafka/versions/0.13.0

gem install rdkafka-0.13.0.gem --local

Building native extensions. This could take a while... ERROR: Error installing rdkafka-0.13.0.gem: ERROR: Failed to build gem native extension.

current directory: /opt/td-agent/lib/ruby/gems/2.7.0/gems/rdkafka-0.13.0/ext /opt/td-agent/bin/ruby -I/opt/td-agent/lib/ruby/2.7.0/rubygems -rrubygems /opt/td-agent/lib/ruby/gems/2.7.0/gems/rake-13.0.6/exe/rake RUBYARCHDIR\=/opt/td-agent/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/rdkafka-0.13.0 RUBYLIBDIR\=/opt/td-agent/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/rdkafka-0.13.0 2 retrie(s) left for v2.0.2 (Net::OpenTimeout) 1 retrie(s) left for v2.0.2 (Net::OpenTimeout) 0 retrie(s) left for v2.0.2 (Net::OpenTimeout) Net::OpenTimeout rake aborted! Errno::ENOENT: No such file or directory @ rb_sysopen - ports/archives/v2.0.2

Tasks: TOP => default (See full trace by running task with --trace)

rake failed, exit code 1

Gem files will remain installed in /opt/td-agent/lib/ruby/gems/2.7.0/gems/rdkafka-0.13.0 for inspection. Results logged to /opt/td-agent/lib/ruby/gems/2.7.0/extensions/x86_64-linux/2.7.0/rdkafka-0.13.0/gem_make.out

mensfeld commented 1 year ago

ref https://github.com/appsignal/rdkafka-ruby/issues/157

mensfeld commented 1 year ago

FYI I am also in favour of switching the releases to have the sources embedded for the stability. Especially looking what happened when GH changed checksums on artefacts.

@thijsc WDYT?

iplparm commented 1 year ago

Could you please provide a temporary workaround procedure for completely offline installation envs? https://github.com/appsignal/rdkafka-ruby/issues/157 contains pretty complicated workaround with Gemfile modifications etc. Would something simpler work?

mensfeld commented 1 year ago

@iplparm I do not have one. I plan to work on this but this is not my priority at the moment.

silent-e commented 5 months ago

Add my voice to not being able to install due to network access restrictions. Why can't you just put in the readme that librdkafka is a dependency and point people to that repo where there are good instructions on how to install it on their server. They've got brew, apt, and yum instructions. Should be sufficient and would appear to solve this issue.

javier-hcp commented 5 months ago

@silent-e fwiw you can avoid gem install downloading trying to download things over the network by setting RDKAFKA_EXT_PATH to point to the .dylib/.so.

https://github.com/karafka/rdkafka-ruby/blob/main/ext/Rakefile#L13C17-L13C33

mensfeld commented 5 months ago

Why can't you just

Why can't you? I am maintaining an ecosystem of gems for Ruby, and I do not appreciate the tone.

You are free to make a PR to improve documentation at any time. You are not free to be rude or demand things from contributors and maintainers.

There are plans to provide librdkafka as an embedding for various reasons but as I already stated it is not my priority. Since it is OSS if you need this, you are happy to work on it and I will provide you my feedback.

silent-e commented 5 months ago

@mensfeld 100% apologize if I came across as rude there. I asked it with no intention of coming off that way and it was intended as simply a question/suggestion. It was meant to sound more like "Is it possible to…". I appreciate the work you're doing on this gem.

I also am doing a big facepalm for not having noticed the RDKAFKA_EXT_PATH setting. I originally saw the workaround mentioned at https://github.com/karafka/rdkafka-ruby/issues/157#issuecomment-847277632 and didn't dig deeper.

Again, apologies for my tone.

mensfeld commented 5 months ago

@silent-e no harm taken apology accepted :)

I am looking into bundling the distribution of the sources together with the gem for a full offline experience, though, as mentioned, it is not my top priority at the moment, especially given the RDKAFKA_EXT_PATH flag.

mensfeld commented 2 months ago

@thijsc do you have anything against bundling the librdkafka source in the gem to support this case? It would also solve the issue of having external dependency (that already once backfired when GH changed something in their API effectively breaking checksums)