search

y-ken / fluent-plugin-geoip

Fluentd output plugin to geolocate with geoip.
http://rubygems.org/gems/fluent-plugin-geoip
Other
103 stars 23 forks source link

Error while installing #57

Closed nikoskalenzis closed 5 years ago

nikoskalenzis commented 6 years ago

I'm trying to install the plugin on a centos 6 machine and getting this in the output:

current directory: /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/geoip2_c-0.3.1/ext/geoip2 make "DESTDIR=" compiling geoip2.c In file included from geoip2.c:3: /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/geoip2_c-0.3.1/ext/geoip2/libmaxminddb/include/maxminddb.h:118: warning: declaration does not declare anything geoip2.c: In function 'mmdb_entry_data_decode': geoip2.c:115: error: 'MMDB_entry_data_s' has no member named 'utf8_string' geoip2.c:115: error: 'MMDB_entry_data_s' has no member named 'utf8_string' geoip2.c:115: error: 'MMDB_entry_data_s' has no member named 'utf8_string' geoip2.c:119: error: 'MMDB_entry_data_s' has no member named 'double_value' geoip2.c:121: error: 'MMDB_entry_data_s' has no member named 'bytes' geoip2.c:121: error: 'MMDB_entry_data_s' has no member named 'bytes' geoip2.c:121: error: 'MMDB_entry_data_s' has no member named 'bytes' geoip2.c:125: error: 'MMDB_entry_data_s' has no member named 'uint16' geoip2.c:127: error: 'MMDB_entry_data_s' has no member named 'uint32' geoip2.c:132: error: 'MMDB_entry_data_s' has no member named 'int32' geoip2.c:134: error: 'MMDB_entry_data_s' has no member named 'uint64' geoip2.c:137: error: 'MMDB_entry_data_s' has no member named 'uint128' geoip2.c:150: error: 'MMDB_entry_data_s' has no member named 'boolean' geoip2.c:152: error: 'MMDB_entry_data_s' has no member named 'float_value' make: *** [geoip2.o] Error 1

make failed, exit code 2

Gem files will remain installed in /opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/geoip2_c-0.3.1 for inspection. Results logged to /opt/td-agent/embedded/lib/ruby/gems/2.4.0/extensions/x86_64-linux/2.4.0/geoip2_c-0.3.1/gem_make.out

any thoughts?

okkez commented 6 years ago

This is geoip2_c's issue. I've fixed the issue and released new version geoip2_c v0.3.2.

Please try again.

discostur commented 6 years ago

@okkez also getting an error if i try to install fluent-plugin-geoip: 

ERROR:  Error installing fluent-plugin-geoip:
    ERROR: Failed to build gem native extension.

    current directory: /usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2/ext/geoip2
/usr/bin/ruby -r ./siteconf20171227-10-13ynzbe.rb extconf.rb
./bootstrap: line 7: autoreconf: not found
make: *** No rule to make target 'clean'.  Stop.
make: *** No targets specified and no makefile found.  Stop.
checking for rb_sym2str() in ruby.h... yes
creating Makefile

To see why this extension failed to compile, please check the mkmf.log which can be found here:

  /usr/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0/geoip2_c-0.3.2/mkmf.log

current directory: /usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2/ext/geoip2
make "DESTDIR=" clean

current directory: /usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2/ext/geoip2
make "DESTDIR="
compiling geoip2.c
In file included from geoip2.c:3:0:
/usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2/ext/geoip2/libmaxminddb/include/maxminddb.h:12:30: fatal error: maxminddb_config.h: No such file or directory
 #include "maxminddb_config.h"
                              ^
compilation terminated.
make: *** [Makefile:239: geoip2.o] Error 1

make failed, exit code 2

Gem files will remain installed in /usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2 for inspection.
Results logged to /usr/lib/ruby/gems/2.3.0/extensions/x86_64-linux/2.3.0/geoip2_c-0.3.2/gem_make.out

this error happens with your fixed version geoip2_c v0.3.2 ...

nikoskalenzis commented 6 years ago

Just to update this.. The installation went through without issues when installing with full path: /usr/sbin/td-agent-gem install fluent-plugin geoip Even tho it's the only path for td-agent present on the VM.

discostur commented 6 years ago

ok, think this is another error:

gem install fluent-plugin-geoip
...
current directory: /usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2/ext/geoip2
make "DESTDIR="
compiling geoip2.c
In file included from geoip2.c:3:0:
/usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2/ext/geoip2/libmaxminddb/include/maxminddb.h:12:30: fatal error: maxminddb_config.h: No such file or directory
 #include "maxminddb_config.h"

# ls -l /usr/lib/ruby/gems/2.3.0/gems/geoip2_c-0.3.2/ext/geoip2/libmaxminddb/include/
total 12
-rw-r--r--    1 root     root          7355 Dec 27 09:54 maxminddb.h
-rw-r--r--    1 root     root           417 Dec 27 09:54 maxminddb_config.h.in
y-ken commented 6 years ago

How about trying to install older versions? /usr/sbin/td-agent-gem install fluent-plugin geoip -v 0.8.0

nikoskalenzis commented 6 years ago

Just to confirm installing older versions also worked for me.

[root@test-instance /]# /usr/sbin/td-agent-gem install fluent-plugin-geoip -v 0.8.0 Fetching: fluent-mixin-rewrite-tag-name-0.1.0.gem (100%) Successfully installed fluent-mixin-rewrite-tag-name-0.1.0 Fetching: fluent-plugin-geoip-0.8.0.gem (100%) Successfully installed fluent-plugin-geoip-0.8.0 Parsing documentation for fluent-mixin-rewrite-tag-name-0.1.0 Installing ri documentation for fluent-mixin-rewrite-tag-name-0.1.0 Parsing documentation for fluent-plugin-geoip-0.8.0 Installing ri documentation for fluent-plugin-geoip-0.8.0 Done installing documentation for fluent-mixin-rewrite-tag-name, fluent-plugin-geoip after 0 seconds 2 gems installed

i'm guessing you already have it, but just to be sure, do you have geoip-devel installed? This was a requirement for me..

discostur commented 6 years ago

@y-ken installing older versions works:

/ # /usr/bin/gem install fluent-plugin-geoip -v 0.8.0
Fetching: fluent-mixin-rewrite-tag-name-0.1.0.gem (100%)
Successfully installed fluent-mixin-rewrite-tag-name-0.1.0
Fetching: fluent-plugin-geoip-0.8.0.gem (100%)
Successfully installed fluent-plugin-geoip-0.8.0
2 gems installed

@nikoskalenzis yes, these are the steps i do before:

apk add --update --virtual .build-deps sudo build-base ruby-dev
apk add --no-cache geoip geoip-dev libmaxminddb
y-ken commented 6 years ago

@okkez Could you test why this error has happen?

@discostur @nikoskalenzis Thank you for your report!

okkez commented 6 years ago

I'm trying to reproduce @nikoskalenzis's situation in this comment.

I could not reproduce the situation on CentOS6 with both td-agent3 and td-agent2. But in case of td-agent2, td-agent-gem install fluent-plugin-geoip installs fluent-plugin-geoip v1.1.0 and Fluentd v1.0.2. Maybe this is not related to this issue.

I think @nikoskalenzis uses td-agent3. Which version of td-agent3 do you use? Could you show me the result of td-agent --version?

td-agent3 on CentOS6

$ lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.9 (Final)
Release:        6.9
Codename:       Final

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent3.sh | sh
(snip)
$ td-agent --version
td-agent 1.0.2

$ sudo yum groupinstall "Development Tools"
$ sudo yum install epel-release
$ sudo yum makecache
$ sudo yum install geoip-devel
$ sudo td-agent-gem install fluent-plugin-geoip
Building native extensions.  This could take a while...
Successfully installed geoip-c-0.9.1
Fetching: geoip2_c-0.3.2.gem (100%)
Building native extensions.  This could take a while...
Successfully installed geoip2_c-0.3.2
Fetching: fluent-plugin-geoip-1.1.0.gem (100%)
Successfully installed fluent-plugin-geoip-1.1.0
Parsing documentation for geoip-c-0.9.1
Installing ri documentation for geoip-c-0.9.1
Parsing documentation for geoip2_c-0.3.2
Installing ri documentation for geoip2_c-0.3.2
^PParsing documentation for fluent-plugin-geoip-1.1.0
Installing ri documentation for fluent-plugin-geoip-1.1.0
Done installing documentation for geoip-c, geoip2_c, fluent-plugin-geoip after 9 seconds
3 gems installed

td-agent2 on CentOS6

$ sudo lsb_release -a
LSB Version:    :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 6.9 (Final)
Release:        6.9
Codename:       Final

$ curl -L https://toolbelt.treasuredata.com/sh/install-redhat-td-agent2.sh | sh
$ td-agent --version
td-agent 0.12.40

$ sudo yum groupinstall "Development Tools"
$ sudo yum install epel-release
$ sudo yum makecache
$ sudo yum install geoip-devel
$ sudo td-agent-gem install fluent-plugin-geoip
Fetching: serverengine-2.0.5.gem (100%)
Successfully installed serverengine-2.0.5
Fetching: strptime-0.2.2.gem (100%)
Building native extensions.  This could take a while...
Successfully installed strptime-0.2.2
Fetching: dig_rb-1.0.1.gem (100%)
Successfully installed dig_rb-1.0.1
Fetching: fluentd-1.0.2.gem (100%)
Successfully installed fluentd-1.0.2
Fetching: geoip-c-0.9.1.gem (100%)
Building native extensions.  This could take a while...
Successfully installed geoip-c-0.9.1
Fetching: geoip2_c-0.3.2.gem (100%)
Building native extensions.  This could take a while...
Successfully installed geoip2_c-0.3.2
Fetching: fluent-plugin-geoip-1.1.0.gem (100%)
Successfully installed fluent-plugin-geoip-1.1.0
Parsing documentation for serverengine-2.0.5
Installing ri documentation for serverengine-2.0.5
Parsing documentation for strptime-0.2.2
Installing ri documentation for strptime-0.2.2
Parsing documentation for dig_rb-1.0.1
Installing ri documentation for dig_rb-1.0.1
Parsing documentation for fluentd-1.0.2
Installing ri documentation for fluentd-1.0.2
Parsing documentation for geoip-c-0.9.1
Installing ri documentation for geoip-c-0.9.1
Parsing documentation for geoip2_c-0.3.2
Installing ri documentation for geoip2_c-0.3.2
Parsing documentation for fluent-plugin-geoip-1.1.0
Installing ri documentation for fluent-plugin-geoip-1.1.0
Done installing documentation for serverengine, strptime, dig_rb, fluentd, geoip-c, geoip2_c, fluent-plugin-geoip after 4 seconds
7 gems installed
okkez commented 6 years ago

For @discostur's situation.

@discostur uses Alpine Linux on Docker, maybe.

He needs installing build dpendencies to builod geoip2_c gem properly.

nikoskalenzis commented 6 years ago

Hey guys, just to confirm: I tried both td-agent2 and td-agent3, on a google compute engine VM running centos 6. Both have failed when using only td-agent-gem and both installed on the first try when using the full path.

[nikos.kalenzis@test-instance ~]$ sudo su [root@test-instance nikos.kalenzis]# lsb_release -a LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch Distributor ID: CentOS Description: CentOS release 6.9 (Final) Release: 6.9 Codename: Final

[nikos.kalenzis@test-geoip ~]$ sudo su [root@test-geoip nikos.kalenzis]# lsb_release -a LSB Version: :base-4.0-amd64:base-4.0-noarch:core-4.0-amd64:core-4.0-noarch Distributor ID: CentOS Description: CentOS release 6.9 (Final) Release: 6.9 Codename: Final [root@test-geoip nikos.kalenzis]# td-agent --version td-agent 1.0.2

I installed the same dependencies as @okkez did, without the makecache option.

discostur commented 6 years ago

@okkez thanks, your were right, i'm trying to run it on alpine linux inside docker. After installing the missing packets you postet i'm able to install the plugin successfully:

docker run -it --rm alpine:latest

apk update \
 && apk upgrade \
 && apk add --no-cache \
        ca-certificates \
        ruby ruby-irb \
 && apk add --no-cache --virtual .build-deps \
        build-base \
        ruby-dev wget gnupg \
 && update-ca-certificates \
 && echo 'gem: --no-document' >> /etc/gemrc \
 && gem install oj -v 2.18.3 \
 && gem install json -v 2.1.0 \
 && gem install fluentd -v 1.0.2 \
 && apk del .build-deps \
 && rm -rf /var/cache/apk/* \
 && rm -rf /tmp/* /var/tmp/* /usr/lib/ruby/gems/*/cache/*.gem \
 && apk add --update --virtual .build-deps sudo build-base ruby-dev \
 && apk add --no-cache geoip geoip-dev libmaxminddb automake autoconf libtool libc6-compat

gem install fluent-plugin-geoip
okkez commented 6 years ago

Both have failed when using only td-agent-gem and both installed on the first try when using the full path.

I've tried to reproduce @nikoskalenzis's situation on GCP CentOS6. I can install successfully when I use td-agent-gem command (not full path). I wonder that @nikoskalenzis failed when he use td-agent-gem command (not full path).

@discostur Good to know that you succeeded! FYI, fluentd official docker image is available https://hub.docker.com/r/fluent/fluentd/

wonderphil commented 6 years ago

Hi guys,

I am getting the same issue when trying to install on Ubuntu 14.04 using td-agent3(v1.0.2). I tried using both /usr/sbin/td-agent-gem install fluent-plugin-geoip and td-agent-gem install fluent-plugin-geoip

For me though this has only happen after I update td-agent from version 2.

okkez commented 6 years ago

@wonderphil I could not reproduce your situation. Show me reproducible sequence step by step, please.

For example, I've tried like following:

$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 14.04.5 LTS
Release:        14.04
Codename:       trusty
$ sudo apt install -y curl
$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent2.sh | sh
$ sudo apt install -y build-essential libgeoip-dev
$ sudo td-agent-gem install fluent-plugin-geoip -v 0.8.2
$ td-agent --version
0.12.40

No problem with td-agent2 and fluent-plugin-geoip 0.8.2

Upgrade to td-agent3:

$ sudo sed -i -e 's/2/3/' /etc/apt/sources.list.d/treasure-data.list
$ sudo apt update
$ sudo apt upgrade -y
$ td-agent --version
1.0.2
$ td-agent-gem list | grep fluent-plugin
fluent-plugin-elasticsearch (2.4.0)
fluent-plugin-kafka (0.6.5)
fluent-plugin-rewrite-tag-filter (2.0.1)
fluent-plugin-s3 (1.1.0)
fluent-plugin-td (1.0.0)
fluent-plugin-td-monitoring (0.2.3)
fluent-plugin-webhdfs (1.2.2)
$ sudo apt install autoreconf libtool
$ sudo td-agent-gem install fluent-plugin-geoip

I've succeeded to upgrade td-agent 2 to 3.

okkez commented 6 years ago

FYI, you can upgrade td-agent 3 from 3 by running following command:

$ curl -L https://toolbelt.treasuredata.com/sh/install-ubuntu-trusty-td-agent3.sh | sh
learn0208 commented 5 years ago

@discostur yum install libmaxminddb-devel -y && /usr/sbin/td-agent-gem install fluent-plugin-geoip

It's worked for me

trotyoung commented 5 years ago

I have the same issue. Eventually, I get the workaround at this place.My case is Ubuntu Xenial and td-agent 1.4.2. https://github.com/maxmind/MaxMind-DB-Reader-php/issues/47#issuecomment-339151803

Just firstly install libmaxminddb-dev, and then install fluent-plugin-geoip. Like this:

apt install libmaxminddb-dev
td-agent-gem install fluent-plugin-geoip -v 1.3.1

Hope can help.