search

fluent / fluent-plugin-flume

Flume input and output plugin for Fluentd
23 stars 13 forks source link

Cannot assign requested address #4

Closed bryanchapman9999 closed 8 years ago

bryanchapman9999 commented 9 years ago

Hi,

Not having much luck using this plugin. I ran it on the same machine as our flume server initially and could never get it to bind to the port saying it was busy - not a massive surprise. Stopped flume and restarted fluentd and all was well, but of course then flume would not start.

Have now moved over to a different server but having the same problem.

The flume part of my fluentd config:

type flume
bind 172.20.5.220
port 12345

The relevant part of my flume config: log4jsink.sources.avroSource.bind = 0.0.0.0 log4jsink.sources.avroSource.channels = log4j2Avro2File log4j2Avro2Nsca syslog2Avro2File tail2Avro2File log4jsink.sources.avroSource.port = 12345 log4jsink.sources.avroSource.selector.type = replicating log4jsink.sources.avroSource.type = avro

An NMAP from the box I'm running fluent on to the flume server: [root@huginn ~]# nmap -sS 172.20.5.220

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2015-01-27 10:32 GMT Interesting ports on flumeuat15-1.live (172.20.5.220): Not shown: 1671 closed ports PORT STATE SERVICE 22/tcp open ssh 25/tcp open smtp 111/tcp open rpcbind 465/tcp open smtps 587/tcp open submission 875/tcp open unknown 2049/tcp open nfs 5999/tcp open ncd-conf 12345/tcp open NetBus

Nmap finished: 1 IP address (1 host up) scanned in 0.314 seconds

This is my fluentd log: [root@huginn td-agent]# cat td-agent.log 2015-01-27 10:39:09 +0000 [info]: shutting down fluentd 2015-01-27 10:39:09 +0000 [info]: process finished code=0 2015-01-27 10:39:11 +0000 [info]: reading config file path="/etc/td-agent/td-agent.conf" 2015-01-27 10:39:11 +0000 [info]: starting fluentd-0.10.58 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-flume' version '0.1.1' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-influxdb' version '0.1.2' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-mongo' version '0.7.4' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-s3' version '0.5.1' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-scribe' version '0.10.14' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-td' version '0.10.23' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0' 2015-01-27 10:39:11 +0000 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1' 2015-01-27 10:39:11 +0000 [info]: gem 'fluentd' version '0.10.58' 2015-01-27 10:39:11 +0000 [info]: using configuration file: <match td..> type tdlog apikey YOUR_API_KEY auto_create_table buffer_type file buffer_path /var/log/td-agent/buffer/td <match debug.**> type stdout 

type forward

type http
port 8888

type flume
bind 172.20.5.220
port 12345

2015-01-27 10:39:11 +0000 [info]: adding source type="forward" 2015-01-27 10:39:11 +0000 [info]: adding source type="http" 2015-01-27 10:39:11 +0000 [info]: adding source type="flume" 2015-01-27 10:39:11 +0000 [info]: adding match pattern="td.." type="tdlog" 2015-01-27 10:39:11 +0000 [info]: adding match pattern="debug.**" type="stdout" 2015-01-27 10:39:11 +0000 [info]: listening fluent socket on 0.0.0.0:24224 2015-01-27 10:39:11 +0000 [error]: unexpected error error="Cannot assign requested address - bind(2) for \"172.20.5.220\" port 12345" 2015-01-27 10:39:11 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/thrift-0.8.0/lib/thrift/transport/server_socket.rb:40:in initialize' 2015-01-27 10:39:11 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/thrift-0.8.0/lib/thrift/transport/server_socket.rb:40:innew' 2015-01-27 10:39:11 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/thrift-0.8.0/lib/thrift/transport/server_socket.rb:40:in listen' 2015-01-27 10:39:11 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/thrift-0.8.0/lib/thrift/server/simple_server.rb:24:inserve' 2015-01-27 10:39:11 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-flume-0.1.1/lib/fluent/plugin/in_flume.rb:101:in `run' [root@huginn td-agent]#

repeatedly commented 9 years ago

You seems to use <source>. This is for Input plugin, not Output plugin. Could you try <match pattern> instead of <source>?

bryanchapman9999 commented 9 years ago

But I wish to use flume as my input

repeatedly commented 9 years ago

Ah, flume - to -> fluentd? Your flume configuration uses avro as source. This causes port conflicts between fluentd input and flume input.

repeatedly commented 9 years ago

BTW, flume plugin has an important issue: https://github.com/fluent/fluent-plugin-flume/issues/3 Latest flume uses incompatible protocol...

bryanchapman9999 commented 9 years ago

OK. No work around as yet?

repeatedly commented 9 years ago

Currently yes. We are now waiting for new maintainer of this plugin.

repeatedly commented 8 years ago

issue 3 is fixed. I close this issue too.