Closed g2doug closed 6 years ago
Interesting, what were you doing at the time to force such an issue?
Can you tell me what gem list
shows and which versions of fluent and ruby are being used?
Can you provide the configuration you're using and perhaps even an example message being processed by the fluent pipeline?
Thanks for your response. I've been interested in fluentd but this is my first kicking the tires.
fluentd -c fluent.conf -vv
This is the verbose output
2018-04-30 17:41:19 -0700 [info]: #0 fluent/log.rb:322:info: starting fluentd worker pid=38716 ppid=38705 worker=0
2018-04-30 17:41:19 -0700 [error]: #0 fluent/log.rb:362:error: unexpected error error_class=AMQ::Protocol::FrameTypeError error="Must be one of [:method, :headers, :body, :heartbeat]"
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/amq-protocol-2.3.0/lib/amq/protocol/frame.rb:64:in `decode_header'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/bunny-2.9.2/lib/bunny/transport.rb:237:in `read_next_frame'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/bunny-2.9.2/lib/bunny/session.rb:1127:in `init_connection'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/bunny-2.9.2/lib/bunny/session.rb:310:in `start'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluent-plugin-amqp-0.11.0/lib/fluent/plugin/in_amqp.rb:70:in `start'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:165:in `block in start'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:153:in `each'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:153:in `block in lifecycle'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:140:in `each'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:140:in `lifecycle'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:164:in `start'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/engine.rb:274:in `start'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/engine.rb:219:in `run'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:799:in `run_engine'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:549:in `block in run_worker'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:724:in `main_process'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:544:in `run_worker'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/bin/fluentd:8:in `<top (required)>'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/bin/fluentd:23:in `load'
2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/bin/fluentd:23:in `<main>'
actioncable (5.1.5)
actionmailer (5.1.5, 4.2.10, 4.2.4, 4.0.0)
actionpack (5.1.5, 4.2.10, 4.2.4, 4.0.0)
actionview (5.1.5, 4.2.10, 4.2.4)
active_model_serializers (0.10.7)
activejob (5.1.5, 4.2.10, 4.2.4)
activemodel (5.1.5, 4.2.10, 4.2.4, 4.0.0, 3.2.22.2)
activerecord (5.1.5, 4.2.10, 4.2.4, 4.0.0, 3.2.22.2)
activerecord-deprecated_finders (1.0.3)
activeresource (4.1.0, 4.0.0)
activesupport (5.1.5, 5.1.4, 4.2.10, 4.2.4, 4.0.0, 3.2.22.2)
addressable (2.5.2)
akami (1.3.1)
amq-protocol (2.3.0)
angularjs-rails (1.2.9)
arel (8.0.0, 6.0.4, 4.0.1, 3.0.3)
ast (2.4.0, 2.3.0)
atomic (1.1.14)
auto_strip_attributes (2.2.0)
bcrypt (3.1.11)
bcrypt-ruby (3.1.5)
better_errors (2.4.0)
bigdecimal (default: 1.3.4)
bindex (0.5.0)
binding_of_caller (0.8.0)
builder (3.2.3, 3.1.4, 3.0.4)
bullet (5.7.2)
bundler (1.16.1)
bundler-unload (1.0.2)
bunny (2.9.2)
byebug (10.0.0, 9.0.6)
cancan (1.6.10)
cancancan (1.17.0)
capistrano (2.15.7)
capistrano-ext (1.2.1)
capybara (2.18.0, 2.17.0)
carmen (1.1.0)
case_transform (0.2)
childprocess (0.9.0, 0.8.0)
cmath (default: 1.0.0)
coderay (1.1.2, 1.1.1)
coffee-rails (4.2.2, 4.1.1)
coffee-script (2.4.1)
coffee-script-source (1.12.2)
commonjs (0.2.7)
concurrent-ruby (1.0.5)
connection_pool (2.2.1)
cool.io (1.5.3)
crass (1.0.3)
crypt-isaac (0.9.1)
csv (default: 1.0.0)
database_cleaner (1.6.2)
date (default: 1.0.0)
dbm (default: 1.0.0)
debug_inspector (0.0.3)
devise (3.5.10, 1.5.4)
devise-encryptable (0.2.0)
did_you_mean (1.2.0)
diff-lcs (1.3)
dig_rb (1.0.1)
docile (1.1.5)
domain_name (0.5.20170404)
erubi (1.7.1, 1.7.0)
erubis (2.7.0)
et-orbi (1.0.9)
etc (default: 1.0.0)
ethon (0.10.1)
execjs (2.7.0)
executable-hooks (1.3.2)
factory_bot (4.8.2)
factory_bot_rails (4.8.2)
factory_girl (4.9.0, 4.7.0)
factory_girl_rails (4.9.0)
faker (1.7.3)
fakeweb (1.3.0)
faraday (0.14.0, 0.13.1, 0.9.2)
faraday_middleware (0.10.1)
faraday_middleware-multi_json (0.0.6)
fcntl (default: 1.0.0)
ffaker (2.8.1)
ffi (1.9.23, 1.9.18)
fiddle (default: 1.0.0)
fileutils (default: 1.0.2)
flipper (0.12.2)
flipper-active_record (0.12.2)
flipper-ui (0.12.2)
fluent-plugin-amqp (0.11.0)
fluent-plugin-terminal_notifier (0.2.0)
fluentd (1.2.0)
formatador (0.2.5)
gdbm (default: 2.0.0)
gem-wrappers (1.4.0)
globalid (0.4.1)
grease (0.3.1)
guard (2.14.2)
guard-compat (1.2.1)
guard-shell (0.7.1)
gyoku (1.3.1)
hashie (3.5.7)
highline (1.7.8)
http-cookie (1.0.3)
http_parser.rb (0.6.0)
httparty (0.15.6)
httpclient (2.8.3)
httpi (2.4.2)
i18n (0.9.5, 0.9.3, 0.9.1, 0.9.0, 0.7.0, 0.6.9)
interception (0.5)
io-console (default: 0.4.6)
ipaddr (default: 1.2.0)
jbuilder (2.7.0)
jira-ruby (1.5.0)
jquery-rails (4.3.1)
jquery-ui-rails (6.0.1)
json (default: 2.1.0, 1.8.6)
jsonapi-renderer (0.2.0)
kaminari (1.1.1)
kaminari-actionview (1.1.1)
kaminari-activerecord (1.1.1)
kaminari-core (1.1.1)
kgio (2.11.1)
less (2.6.0)
less-rails (3.0.0)
lhm (2.2.0)
listen (3.1.5, 3.0.8)
log4r (1.1.10)
logstash-event (1.1.5)
logstasher (0.5.3)
loofah (2.2.0, 2.1.1)
lumberjack (1.0.13)
macaddr (1.7.1)
mail (2.7.0, 2.5.4)
mechanize (2.7.5)
metaclass (0.0.4)
method_source (0.9.0, 0.8.2)
mime-types (3.1, 1.25.1)
mime-types-data (3.2016.0521)
mini_mime (1.0.0)
mini_portile2 (2.3.0, 2.1.0)
minitest (5.11.3, 5.11.1, 5.10.3, 5.8.4, 4.7.5)
mocha (1.1.0)
msgpack (1.2.4)
multi_json (1.13.1, 1.12.2, 1.12.1, 1.8.4)
multi_xml (0.6.0)
multipart-post (2.0.0)
mysql2 (0.4.10, 0.3.21)
nenv (0.3.0)
neo4j-core (8.0.1)
net-http-digest_auth (1.4.1)
net-http-persistent (2.9.4)
net-scp (1.2.1)
net-sftp (2.1.2)
net-ssh (2.9.4)
net-ssh-gateway (1.2.0)
net-telnet (0.1.1)
newrelic_rpm (4.7.1.340, 4.4.0.336)
nio4r (2.2.0)
nokogiri (1.8.2, 1.8.1)
nori (2.6.0)
notiffany (0.1.1)
ntlm-http (0.1.1)
nunes (0.5.0)
oauth (0.5.4)
openssl (default: 2.1.0)
orm_adapter (0.5.0, 0.0.7)
parallel (1.12.1)
parser (2.5.0.4, 2.4.0.2)
passenger (5.2.0, 5.0.30)
pg (0.18.4)
phony (2.16.3)
phony_rails (0.14.6)
polyglot (0.3.3)
power_assert (1.1.1)
powerpack (0.1.1)
pry (0.11.3, 0.10.4)
pry-byebug (3.4.3)
pry-nav (0.2.4)
pry-rails (0.3.6)
pry-remote (0.1.8)
pry-rescue (1.4.5)
pry-stack_explorer (0.4.9.2)
psych (default: 3.0.2)
public_suffix (3.0.2, 3.0.1)
puma (3.11.3)
r18n-core (1.1.11)
rabl-rails (0.5.2)
rack (2.0.4, 1.6.9, 1.6.8, 1.5.5, 1.5.2)
rack-protection (1.5.3)
rack-proxy (0.6.3)
rack-test (0.8.3, 0.6.3, 0.6.2)
rails (4.2.10)
rails-api (0.4.1)
rails-deprecated_sanitizer (1.0.3)
rails-dom-testing (2.0.3, 1.0.9)
rails-html-sanitizer (1.0.3)
rails-observers (0.1.5, 0.1.2)
railties (5.1.5, 4.2.10, 4.2.4)
rainbow (3.0.0)
raindrops (0.19.0)
rake (12.3.0, 12.1.0, 11.1.2, 10.5.0, 10.4.2, 10.1.1)
rb-fsevent (0.10.3, 0.10.2, 0.9.8)
rb-inotify (0.9.10, 0.9.8)
rconfig (0.5.2)
rdoc (default: 6.0.1, 4.3.0)
redis (3.3.5)
redis-actionpack (5.0.2)
redis-activesupport (5.0.4)
redis-namespace (1.6.0)
redis-rack (2.0.4)
redis-rails (5.0.2)
redis-store (1.4.1)
request_store (1.4.0)
responders (2.4.0)
rspec-core (3.1.7, 3.0.4)
rspec-expectations (3.1.2, 3.0.4)
rspec-mocks (3.1.3, 3.0.4)
rspec-rails (3.1.0, 3.0.2)
rspec-sidekiq (3.0.3)
rspec-support (3.1.2, 3.0.4)
rspec_junit_formatter (0.3.0)
rubocop (0.54.0, 0.52.1)
ruby-growl (4.1)
ruby-progressbar (1.9.0)
ruby_dep (1.5.0)
rubygems-bundler (1.4.4)
rubyntlm (0.3.2)
rubyzip (1.2.1)
rufus-scheduler (3.4.2)
rvm (1.11.3.9)
sass (3.5.5)
sass-listen (4.0.0)
sass-rails (5.0.7)
savon (2.12.0)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
sdoc (0.4.2)
selenium-webdriver (3.11.0, 3.10.0)
sentry-raven (2.7.2, 1.2.3, 0.15.6)
serverengine (2.0.6)
shellany (0.0.1)
sidekiq (5.0.5)
sidekiq-scheduler (2.2.1)
sigdump (0.2.4)
simplecov (0.15.1)
simplecov-html (0.10.2)
simplecov-rcov (0.2.3)
sinatra (1.4.8)
sinatra-r18n (1.1.11)
slop (3.6.0)
socksify (1.7.1)
soda-ruby (0.2.25)
spring (2.0.2)
spring-watcher-listen (2.0.1)
sprockets (3.7.1)
sprockets-rails (3.2.1)
sqlite3 (1.3.13)
statsd-ruby (1.4.0)
string_score (1.1.0)
stringio (default: 0.0.1)
strptime (0.2.3)
strscan (default: 1.0.0)
sys-proctable (0.9.9 universal-darwin)
sys-uname (1.0.3)
systemu (2.6.5)
terminal-notifier (1.8.0)
test-unit (3.2.7)
thor (0.20.0)
thread_safe (0.3.6, 0.1.3)
tilt (2.0.8)
tmuxinator (0.10.1)
treetop (1.4.15)
turbolinks (5.1.0)
turbolinks-source (5.1.0)
twitter-bootstrap-rails (3.2.2)
typhoeus (1.3.0)
tzinfo (1.2.5, 1.2.4, 1.2.3, 0.3.49, 0.3.38)
tzinfo-data (1.2018.4)
uglifier (4.1.7, 4.1.6, 4.1.5)
unf (0.1.4)
unf_ext (0.0.7.4)
unicode-display_width (1.3.0)
unicorn (4.7.0)
uniform_notifier (1.11.0)
uuid (2.3.8)
vagrant (1.5.0)
vcr (4.0.0)
versionist (1.7.0)
warden (1.2.7, 1.2.6)
wasabi (3.5.0)
web-console (3.5.1, 2.3.0)
webpacker (3.2.1)
webrick (default: 1.4.2)
webrobots (0.1.2)
websocket-driver (0.6.5)
websocket-extensions (0.1.3)
xdg (2.2.3)
xmlrpc (0.3.0)
xpath (3.0.0)
yajl-ruby (1.4.0)
yard (0.9.12)
zlib (default: 1.0.0)
# In v1 configuration, type and id are @ prefix parameters.
# @type and @id are recommended. type and id are still available for backward compatibility
## built-in TCP input
## $ echo <json> | fluent-cat <tag>
<source>
@type forward
@id forward_input
</source>
# <source>
# @type dummy
# dummy {"hello": "World"}
# tag dummy
# </source>
<source>
type amqp
host ssmt-ap02.stg.g2
port 61616
format json
tag terminal_notifier
tag_key true
exchange_type headers
queue *
</source>
<match terminal_notifier.**>
@type terminal_notifier
title test
sub_title sub
tag_key true
activate com.apple.Terminal
</match>
<match dummy*>
@type terminal_notifier
title test
sub_title sub
activate com.apple.Terminal
</match>
## built-in UNIX socket input
#<source>
# @type unix
#</source>
# HTTP input
# http://localhost:8888/<tag>?json=<json>
# <source>
# @type http
# @id http_input
#
# port 8888
# </source>
## File input
## read apache logs with tag=apache.access
#<source>
# @type tail
# format apache
# path /var/log/httpd-access.log
# tag apache.access
#</source>
## Mutating event filter
## Add hostname and tag fields to apache.access tag events
#<filter apache.access>
# @type record_transformer
# <record>
# hostname ${hostname}
# tag ${tag}
# </record>
#</filter>
## Selecting event filter
## Remove unnecessary events from apache prefixed tag events
#<filter apache.**>
# @type grep
# include1 method GET # pass only GET in 'method' field
# exclude1 message debug # remove debug event
#</filter>
# Listen HTTP for monitoring
# http://localhost:24220/api/plugins
# http://localhost:24220/api/plugins?type=TYPE
# http://localhost:24220/api/plugins?tag=MYTAG
# <source>
# @type monitor_agent
# @id monitor_agent_input
#
# port 24220
# </source>
# Listen DRb for debug
# <source>
# @type debug_agent
# @id debug_agent_input
#
# bind 127.0.0.1
# port 24230
# </source>
## match tag=apache.access and write to file
#<match apache.access>
# @type file
# path /var/log/fluent/access
#</match>
## match tag=debug.** and dump to console
<match debug.**>
@type stdout
@id stdout_output
</match>
# match tag=system.** and forward to another fluent server
# <match system.**>
# @type forward
# @id forward_output
#
# <server>
# host 192.168.0.11
# </server>
# <secondary>
# <server>
# host 192.168.0.12
# </server>
# </secondary>
# </match>
## match tag=myapp.** and forward and write to file
#<match myapp.**>
# @type copy
# <store>
# @type forward
# buffer_type file
# buffer_path /var/log/fluent/myapp-forward
# retry_limit 50
# flush_interval 10s
# <server>
# host 192.168.0.13
# </server>
# </store>
# <store>
# @type file
# path /var/log/fluent/myapp
# </store>
#</match>
## match fluent's internal events
#<match fluent.**>
# @type null
#</match>
## match not matched logs and write to file
#<match **>
# @type file
# path /var/log/fluent/else
# compress gz
#</match>
## Label: For handling complex event routing
#<label @STAGING>
# <match system.**>
# @type forward
# @id staging_forward_output
# <server>
# host 192.168.0.101
# </server>
# </match>
#</label>
edit: Formatting of long output into details block
Fantastic, I'll try and take a look at whats going on, but I've not updated to latest versions for a few of these components so it might take a little while to debug...
Based on the port, I'm thinking you're using Apache ActiveMQ as your message broker. Some quick research suggests that ActiveMQ supports amqp protocol version 1.0 which the bunny/ruby-amqp libraries do not support.[1]
Furthermore it looks like almost all client integrations to ActiveMQ are using either OpenWire or Stomp as their protocol of choice[2] with only a few client implementations around[3]
⚠️ Looking deeper at port 61616 I find that its infact an OpenWire protocol port, which this fluent-plugin won't work with at all... OpenWire is a binary protocol and I can't find any ruby support for it. Stomp however is much better supported but lower performance (and considerably fewer features compared to Rabbitmq and AMQP 0.9.1)
I'm afraid I'm going to have to mark this as a WontFix as my triage suggests the issue is a result of using Apache ActiveMQ and the OpenWire protocol, neither of which can be supported by the libraries this plugin is using and no appropriate alternatives exist.
Sorry
unexpected error error_class=AMQ::Protocol::FrameTypeError error="Must be one of [:method, :headers, :body, :heartbeat]
I googled the error and got the test suite associated with amq-protocol gem