search

giraffi / fluent-plugin-amqp

Use AMQP broker to send or receive messages via FluentD
MIT License
15 stars 31 forks source link

unexpected error error_class=AMQ::Protocol::FrameTypeError #53

Closed g2doug closed 6 years ago

g2doug commented 6 years ago

unexpected error error_class=AMQ::Protocol::FrameTypeError error="Must be one of [:method, :headers, :body, :heartbeat]

I googled the error and got the test suite associated with amq-protocol gem

warmfusion commented 6 years ago

Interesting, what were you doing at the time to force such an issue?

Can you tell me what gem list shows and which versions of fluent and ruby are being used?

Can you provide the configuration you're using and perhaps even an example message being processed by the fluent pipeline?

g2doug commented 6 years ago

Thanks for your response. I've been interested in fluentd but this is my first kicking the tires. fluentd -c fluent.conf -vv

This is the verbose output

2018-04-30 17:41:19 -0700 [info]: #0 fluent/log.rb:322:info: starting fluentd worker pid=38716 ppid=38705 worker=0
2018-04-30 17:41:19 -0700 [error]: #0 fluent/log.rb:362:error: unexpected error error_class=AMQ::Protocol::FrameTypeError error="Must be one of [:method, :headers, :body, :heartbeat]"
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/amq-protocol-2.3.0/lib/amq/protocol/frame.rb:64:in `decode_header'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/bunny-2.9.2/lib/bunny/transport.rb:237:in `read_next_frame'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/bunny-2.9.2/lib/bunny/session.rb:1127:in `init_connection'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/bunny-2.9.2/lib/bunny/session.rb:310:in `start'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluent-plugin-amqp-0.11.0/lib/fluent/plugin/in_amqp.rb:70:in `start'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:165:in `block in start'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:154:in `block (2 levels) in lifecycle'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:153:in `each'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:153:in `block in lifecycle'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:140:in `each'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:140:in `lifecycle'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/root_agent.rb:164:in `start'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/engine.rb:274:in `start'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/engine.rb:219:in `run'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:799:in `run_engine'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:549:in `block in run_worker'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:724:in `main_process'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/supervisor.rb:544:in `run_worker'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/lib/fluent/command/fluentd.rb:316:in `<top (required)>'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/rubies/ruby-2.5.0/lib/ruby/2.5.0/rubygems/core_ext/kernel_require.rb:59:in `require'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/gems/fluentd-1.2.0/bin/fluentd:8:in `<top (required)>'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/bin/fluentd:23:in `load'
  2018-04-30 17:41:19 -0700 [error]: #0 fluent/supervisor.rb:549:block in run_worker: /Users/dheadley/.rvm/gems/ruby-2.5.0/bin/fluentd:23:in `<main>'

Additional information

gem list 
actioncable (5.1.5)
actionmailer (5.1.5, 4.2.10, 4.2.4, 4.0.0)
actionpack (5.1.5, 4.2.10, 4.2.4, 4.0.0)
actionview (5.1.5, 4.2.10, 4.2.4)
active_model_serializers (0.10.7)
activejob (5.1.5, 4.2.10, 4.2.4)
activemodel (5.1.5, 4.2.10, 4.2.4, 4.0.0, 3.2.22.2)
activerecord (5.1.5, 4.2.10, 4.2.4, 4.0.0, 3.2.22.2)
activerecord-deprecated_finders (1.0.3)
activeresource (4.1.0, 4.0.0)
activesupport (5.1.5, 5.1.4, 4.2.10, 4.2.4, 4.0.0, 3.2.22.2)
addressable (2.5.2)
akami (1.3.1)
amq-protocol (2.3.0)
angularjs-rails (1.2.9)
arel (8.0.0, 6.0.4, 4.0.1, 3.0.3)
ast (2.4.0, 2.3.0)
atomic (1.1.14)
auto_strip_attributes (2.2.0)
bcrypt (3.1.11)
bcrypt-ruby (3.1.5)
better_errors (2.4.0)
bigdecimal (default: 1.3.4)
bindex (0.5.0)
binding_of_caller (0.8.0)
builder (3.2.3, 3.1.4, 3.0.4)
bullet (5.7.2)
bundler (1.16.1)
bundler-unload (1.0.2)
bunny (2.9.2)
byebug (10.0.0, 9.0.6)
cancan (1.6.10)
cancancan (1.17.0)
capistrano (2.15.7)
capistrano-ext (1.2.1)
capybara (2.18.0, 2.17.0)
carmen (1.1.0)
case_transform (0.2)
childprocess (0.9.0, 0.8.0)
cmath (default: 1.0.0)
coderay (1.1.2, 1.1.1)
coffee-rails (4.2.2, 4.1.1)
coffee-script (2.4.1)
coffee-script-source (1.12.2)
commonjs (0.2.7)
concurrent-ruby (1.0.5)
connection_pool (2.2.1)
cool.io (1.5.3)
crass (1.0.3)
crypt-isaac (0.9.1)
csv (default: 1.0.0)
database_cleaner (1.6.2)
date (default: 1.0.0)
dbm (default: 1.0.0)
debug_inspector (0.0.3)
devise (3.5.10, 1.5.4)
devise-encryptable (0.2.0)
did_you_mean (1.2.0)
diff-lcs (1.3)
dig_rb (1.0.1)
docile (1.1.5)
domain_name (0.5.20170404)
erubi (1.7.1, 1.7.0)
erubis (2.7.0)
et-orbi (1.0.9)
etc (default: 1.0.0)
ethon (0.10.1)
execjs (2.7.0)
executable-hooks (1.3.2)
factory_bot (4.8.2)
factory_bot_rails (4.8.2)
factory_girl (4.9.0, 4.7.0)
factory_girl_rails (4.9.0)
faker (1.7.3)
fakeweb (1.3.0)
faraday (0.14.0, 0.13.1, 0.9.2)
faraday_middleware (0.10.1)
faraday_middleware-multi_json (0.0.6)
fcntl (default: 1.0.0)
ffaker (2.8.1)
ffi (1.9.23, 1.9.18)
fiddle (default: 1.0.0)
fileutils (default: 1.0.2)
flipper (0.12.2)
flipper-active_record (0.12.2)
flipper-ui (0.12.2)
fluent-plugin-amqp (0.11.0)
fluent-plugin-terminal_notifier (0.2.0)
fluentd (1.2.0)
formatador (0.2.5)
gdbm (default: 2.0.0)
gem-wrappers (1.4.0)
globalid (0.4.1)
grease (0.3.1)
guard (2.14.2)
guard-compat (1.2.1)
guard-shell (0.7.1)
gyoku (1.3.1)
hashie (3.5.7)
highline (1.7.8)
http-cookie (1.0.3)
http_parser.rb (0.6.0)
httparty (0.15.6)
httpclient (2.8.3)
httpi (2.4.2)
i18n (0.9.5, 0.9.3, 0.9.1, 0.9.0, 0.7.0, 0.6.9)
interception (0.5)
io-console (default: 0.4.6)
ipaddr (default: 1.2.0)
jbuilder (2.7.0)
jira-ruby (1.5.0)
jquery-rails (4.3.1)
jquery-ui-rails (6.0.1)
json (default: 2.1.0, 1.8.6)
jsonapi-renderer (0.2.0)
kaminari (1.1.1)
kaminari-actionview (1.1.1)
kaminari-activerecord (1.1.1)
kaminari-core (1.1.1)
kgio (2.11.1)
less (2.6.0)
less-rails (3.0.0)
lhm (2.2.0)
listen (3.1.5, 3.0.8)
log4r (1.1.10)
logstash-event (1.1.5)
logstasher (0.5.3)
loofah (2.2.0, 2.1.1)
lumberjack (1.0.13)
macaddr (1.7.1)
mail (2.7.0, 2.5.4)
mechanize (2.7.5)
metaclass (0.0.4)
method_source (0.9.0, 0.8.2)
mime-types (3.1, 1.25.1)
mime-types-data (3.2016.0521)
mini_mime (1.0.0)
mini_portile2 (2.3.0, 2.1.0)
minitest (5.11.3, 5.11.1, 5.10.3, 5.8.4, 4.7.5)
mocha (1.1.0)
msgpack (1.2.4)
multi_json (1.13.1, 1.12.2, 1.12.1, 1.8.4)
multi_xml (0.6.0)
multipart-post (2.0.0)
mysql2 (0.4.10, 0.3.21)
nenv (0.3.0)
neo4j-core (8.0.1)
net-http-digest_auth (1.4.1)
net-http-persistent (2.9.4)
net-scp (1.2.1)
net-sftp (2.1.2)
net-ssh (2.9.4)
net-ssh-gateway (1.2.0)
net-telnet (0.1.1)
newrelic_rpm (4.7.1.340, 4.4.0.336)
nio4r (2.2.0)
nokogiri (1.8.2, 1.8.1)
nori (2.6.0)
notiffany (0.1.1)
ntlm-http (0.1.1)
nunes (0.5.0)
oauth (0.5.4)
openssl (default: 2.1.0)
orm_adapter (0.5.0, 0.0.7)
parallel (1.12.1)
parser (2.5.0.4, 2.4.0.2)
passenger (5.2.0, 5.0.30)
pg (0.18.4)
phony (2.16.3)
phony_rails (0.14.6)
polyglot (0.3.3)
power_assert (1.1.1)
powerpack (0.1.1)
pry (0.11.3, 0.10.4)
pry-byebug (3.4.3)
pry-nav (0.2.4)
pry-rails (0.3.6)
pry-remote (0.1.8)
pry-rescue (1.4.5)
pry-stack_explorer (0.4.9.2)
psych (default: 3.0.2)
public_suffix (3.0.2, 3.0.1)
puma (3.11.3)
r18n-core (1.1.11)
rabl-rails (0.5.2)
rack (2.0.4, 1.6.9, 1.6.8, 1.5.5, 1.5.2)
rack-protection (1.5.3)
rack-proxy (0.6.3)
rack-test (0.8.3, 0.6.3, 0.6.2)
rails (4.2.10)
rails-api (0.4.1)
rails-deprecated_sanitizer (1.0.3)
rails-dom-testing (2.0.3, 1.0.9)
rails-html-sanitizer (1.0.3)
rails-observers (0.1.5, 0.1.2)
railties (5.1.5, 4.2.10, 4.2.4)
rainbow (3.0.0)
raindrops (0.19.0)
rake (12.3.0, 12.1.0, 11.1.2, 10.5.0, 10.4.2, 10.1.1)
rb-fsevent (0.10.3, 0.10.2, 0.9.8)
rb-inotify (0.9.10, 0.9.8)
rconfig (0.5.2)
rdoc (default: 6.0.1, 4.3.0)
redis (3.3.5)
redis-actionpack (5.0.2)
redis-activesupport (5.0.4)
redis-namespace (1.6.0)
redis-rack (2.0.4)
redis-rails (5.0.2)
redis-store (1.4.1)
request_store (1.4.0)
responders (2.4.0)
rspec-core (3.1.7, 3.0.4)
rspec-expectations (3.1.2, 3.0.4)
rspec-mocks (3.1.3, 3.0.4)
rspec-rails (3.1.0, 3.0.2)
rspec-sidekiq (3.0.3)
rspec-support (3.1.2, 3.0.4)
rspec_junit_formatter (0.3.0)
rubocop (0.54.0, 0.52.1)
ruby-growl (4.1)
ruby-progressbar (1.9.0)
ruby_dep (1.5.0)
rubygems-bundler (1.4.4)
rubyntlm (0.3.2)
rubyzip (1.2.1)
rufus-scheduler (3.4.2)
rvm (1.11.3.9)
sass (3.5.5)
sass-listen (4.0.0)
sass-rails (5.0.7)
savon (2.12.0)
scanf (default: 1.0.0)
sdbm (default: 1.0.0)
sdoc (0.4.2)
selenium-webdriver (3.11.0, 3.10.0)
sentry-raven (2.7.2, 1.2.3, 0.15.6)
serverengine (2.0.6)
shellany (0.0.1)
sidekiq (5.0.5)
sidekiq-scheduler (2.2.1)
sigdump (0.2.4)
simplecov (0.15.1)
simplecov-html (0.10.2)
simplecov-rcov (0.2.3)
sinatra (1.4.8)
sinatra-r18n (1.1.11)
slop (3.6.0)
socksify (1.7.1)
soda-ruby (0.2.25)
spring (2.0.2)
spring-watcher-listen (2.0.1)
sprockets (3.7.1)
sprockets-rails (3.2.1)
sqlite3 (1.3.13)
statsd-ruby (1.4.0)
string_score (1.1.0)
stringio (default: 0.0.1)
strptime (0.2.3)
strscan (default: 1.0.0)
sys-proctable (0.9.9 universal-darwin)
sys-uname (1.0.3)
systemu (2.6.5)
terminal-notifier (1.8.0)
test-unit (3.2.7)
thor (0.20.0)
thread_safe (0.3.6, 0.1.3)
tilt (2.0.8)
tmuxinator (0.10.1)
treetop (1.4.15)
turbolinks (5.1.0)
turbolinks-source (5.1.0)
twitter-bootstrap-rails (3.2.2)
typhoeus (1.3.0)
tzinfo (1.2.5, 1.2.4, 1.2.3, 0.3.49, 0.3.38)
tzinfo-data (1.2018.4)
uglifier (4.1.7, 4.1.6, 4.1.5)
unf (0.1.4)
unf_ext (0.0.7.4)
unicode-display_width (1.3.0)
unicorn (4.7.0)
uniform_notifier (1.11.0)
uuid (2.3.8)
vagrant (1.5.0)
vcr (4.0.0)
versionist (1.7.0)
warden (1.2.7, 1.2.6)
wasabi (3.5.0)
web-console (3.5.1, 2.3.0)
webpacker (3.2.1)
webrick (default: 1.4.2)
webrobots (0.1.2)
websocket-driver (0.6.5)
websocket-extensions (0.1.3)
xdg (2.2.3)
xmlrpc (0.3.0)
xpath (3.0.0)
yajl-ruby (1.4.0)
yard (0.9.12)
zlib (default: 1.0.0)

Fluent.conf 
# In v1 configuration, type and id are @ prefix parameters.
# @type and @id are recommended. type and id are still available for backward compatibility

## built-in TCP input
## $ echo <json> | fluent-cat <tag>
<source>
  @type forward
  @id forward_input
</source>

# <source>
#   @type dummy
#   dummy {"hello": "World"}
#   tag dummy
# </source>

<source>
  type amqp
  host ssmt-ap02.stg.g2
  port 61616
  format json
  tag terminal_notifier
  tag_key true
  exchange_type headers
  queue *
</source>

<match terminal_notifier.**>
  @type terminal_notifier
  title test
  sub_title sub
  tag_key true
  activate com.apple.Terminal
</match>

<match dummy*>
  @type terminal_notifier
  title test
  sub_title sub
  activate com.apple.Terminal
</match>
## built-in UNIX socket input
#<source>
#  @type unix
#</source>

# HTTP input
# http://localhost:8888/<tag>?json=<json>
# <source>
#   @type http
#   @id http_input
#
#   port 8888
# </source>

## File input
## read apache logs with tag=apache.access
#<source>
#  @type tail
#  format apache
#  path /var/log/httpd-access.log
#  tag apache.access
#</source>

## Mutating event filter
## Add hostname and tag fields to apache.access tag events
#<filter apache.access>
#  @type record_transformer
#  <record>
#    hostname ${hostname}
#    tag ${tag}
#  </record>
#</filter>

## Selecting event filter
## Remove unnecessary events from apache prefixed tag events
#<filter apache.**>
#  @type grep
#  include1 method GET # pass only GET in 'method' field
#  exclude1 message debug # remove debug event
#</filter>

# Listen HTTP for monitoring
# http://localhost:24220/api/plugins
# http://localhost:24220/api/plugins?type=TYPE
# http://localhost:24220/api/plugins?tag=MYTAG
# <source>
#   @type monitor_agent
#   @id monitor_agent_input
#
#   port 24220
# </source>

# Listen DRb for debug
# <source>
#   @type debug_agent
#   @id debug_agent_input
#
#   bind 127.0.0.1
#   port 24230
# </source>

## match tag=apache.access and write to file
#<match apache.access>
#  @type file
#  path /var/log/fluent/access
#</match>

## match tag=debug.** and dump to console
<match debug.**>
  @type stdout
  @id stdout_output
</match>

# match tag=system.** and forward to another fluent server
# <match system.**>
#   @type forward
#   @id forward_output
#
#   <server>
#     host 192.168.0.11
#   </server>
#   <secondary>
#     <server>
#       host 192.168.0.12
#     </server>
#   </secondary>
# </match>

## match tag=myapp.** and forward and write to file
#<match myapp.**>
#  @type copy
#  <store>
#    @type forward
#    buffer_type file
#    buffer_path /var/log/fluent/myapp-forward
#    retry_limit 50
#    flush_interval 10s
#    <server>
#      host 192.168.0.13
#    </server>
#  </store>
#  <store>
#    @type file
#    path /var/log/fluent/myapp
#  </store>
#</match>

## match fluent's internal events
#<match fluent.**>
#  @type null
#</match>

## match not matched logs and write to file
#<match **>
#  @type file
#  path /var/log/fluent/else
#  compress gz
#</match>

## Label: For handling complex event routing
#<label @STAGING>
#  <match system.**>
#    @type forward
#    @id staging_forward_output
#    <server>
#      host 192.168.0.101
#    </server>
#  </match>
#</label>

edit: Formatting of long output into details block

warmfusion commented 6 years ago

Fantastic, I'll try and take a look at whats going on, but I've not updated to latest versions for a few of these components so it might take a little while to debug...

warmfusion commented 6 years ago

Based on the port, I'm thinking you're using Apache ActiveMQ as your message broker. Some quick research suggests that ActiveMQ supports amqp protocol version 1.0 which the bunny/ruby-amqp libraries do not support.[1]

Furthermore it looks like almost all client integrations to ActiveMQ are using either OpenWire or Stomp as their protocol of choice[2] with only a few client implementations around[3]

⚠️ Looking deeper at port 61616 I find that its infact an OpenWire protocol port, which this fluent-plugin won't work with at all... OpenWire is a binary protocol and I can't find any ruby support for it. Stomp however is much better supported but lower performance (and considerably fewer features compared to Rabbitmq and AMQP 0.9.1)

I'm afraid I'm going to have to mark this as a WontFix as my triage suggests the issue is a result of using Apache ActiveMQ and the OpenWire protocol, neither of which can be supported by the libraries this plugin is using and no appropriate alternatives exist.

Sorry

  1. https://github.com/ruby-amqp/bunny/issues/304
  2. http://activemq.apache.org/cross-language-clients.html
  3. https://github.com/rabbitmq/rabbitmq-amqp1.0#clients-we-have-tested