search

splunk / fluent-plugin-splunk-hec

This is the Fluentd output plugin for sending events to Splunk via HEC.
Apache License 2.0
83 stars 89 forks source link

Incompatible with td-agent 3 and 4! #144

Closed pranavmarla closed 2 years ago

pranavmarla commented 4 years ago

What happened: I tried to do 2 things: Step 1: Install td-agent Step 2: Using the bundled Ruby gem (td-agent-gem), install Splunk output plugin

Unfortunately, out of the box, the Splunk plugin is incompatible with both the latest major versions of td-agent (3 and 4)!

What you expected to happen: Installing Splunk plugin for td-agent should just work without issues, similar to installing other output plugins like Kafka.

How to reproduce it (as minimally and precisely as possible):

  1. For td-agent 3 (specifically, 3.8.0):
    • Install td-agent 3 via apt
    • Add Ruby gems source: td-agent-gem sources --add https://rubygems.org/
    • Attempt to install Splunk plugin: td-agent-gem install fluent-plugin-splunk-hec At this point, the Splunk plugin installation will fail because it tries to install activesupport 6.0.3.2, which requires Ruby >= 2.5.0 -- unfortunately, the version of Ruby bundled with td-agent 3 is only 2.4.10: 
      > /opt/td-agent/embedded/bin/ruby --version
ruby 2.4.10p364 (2020-03-31 revision 67879) [x86_64-linux]

Thus, the Splunk plugin is incompatible with td-agent 3.

  1. For td-agent 4 (specifically, 4.0.1):
    • Install td-agent 4 via apt
    • Add Ruby gems source: td-agent-gem sources --add https://rubygems.org/
    • Install Splunk plugin: td-agent-gem install fluent-plugin-splunk-hec This time, the Splunk plugin installation (and activesupport 6.0.3.2 installation) will work, because the version of Ruby bundled with td-agent 4 is 2.7.1: 
      > /opt/td-agent/bin/ruby --version
ruby 2.7.1p83 (2020-03-31 revision a0c7c23c9c) [x86_64-linux]
    • However, when we attempt to actually start up Fluentd (and activate the newly installed Splunk plugin), it fails (assuming the Fluentd config sends logs to Splunk output):
    • If your Fluentd config is complex, you will see a misleading error message complaining about a "duplicated plugin id" (as seen in #138). This is a red herring -- there is no actual duplicate plugin ID.
    • If we simplify the Fluentd config to just 1 source and 1 match (Splunk output), we will see the real error: 
      /opt/td-agent/lib/ruby/2.7.0/rubygems/specification.rb:2243:in `raise_if_conflicts': Unable to activate activesupport-6.0.3.2, because tzinfo-2.0.2 conflicts with tzinfo (~> 1.1) (Gem::ConflictError)

      i.e. When we install td-agent 4, it installs tzinfo 2.0.2 -- however, the Splunk plugin (specifically the activesupport dependency) cannot handle that newer version. As part of the Splunk plugin installation, it installs an older compatible version of tzinfo (1.2.7) -- however, since the newer incompatible version (2.0.2) is still present from the previous td-agent installation, activesupport fails to activate, which means the Splunk plugin fails to activate.

Thus, the Splunk plugin is incompatible with td-agent 4.

Anything else we need to know?:

Workaround: As of now, I was not able to get the Splunk plugin working with td-agent 3. However, I was able to eventually get it working with td-agent 4, by manually uninstalling the newer version of tzinfo (2.0.2) (td-agent-gem uninstall tzinfo -v '>= 2.0') before starting up Fluentd (and activating the Splunk plugin).

Goal: Ideally, it should be possible to just install the Splunk plugin for td-agent and have it work, regardless of whether it's v3 or v4, without us having to do any manual workarounds. At the very least though, given how many people use td-agent, it would helpful to have this incompatibility (and workaround) documented in the main README page.

Environment:

gp510 commented 4 years ago

Thanks @pranavmarla, glad you were able to get this working with td-agent 4. We will add this to our backlog as an enhancement request.

pranavmarla commented 4 years ago

Great, thank you @gp510 !

rockb1017 commented 4 years ago

Thank you for reporting and telling us the workaround! I looked into it and the issue is coming from openid_connect gem is requiring active_model gem, and it requires tz_info ~> 1.1. either these gems fix their dependencies or we have to avoid using them.

hvaghani221 commented 2 years ago

Hey @pranavmarla, I was able to install fluend-hec v1.2.9 plugin in both td-agent 3 and td-agent 4 and received logs in splunk.

Result for td-agent 4

$ rpm -qa | grep td-agent
td-agent-4.3.0-1.el7.x86_64

$ sudo td-agent-gem install fluent-plugin-splunk-hec
Fetching attr_required-1.0.1.gem
Fetching connection_pool-2.2.5.gem
Fetching net-http-persistent-3.1.0.gem
Fetching mail-2.7.1.gem
Fetching activemodel-7.0.0.gem
Fetching validate_url-1.0.13.gem
Fetching mini_mime-1.1.2.gem
Fetching i18n-1.8.11.gem
Fetching activesupport-7.0.0.gem
Fetching validate_email-0.1.6.gem
Fetching aes_key_wrap-1.1.0.gem
Fetching json-jwt-1.13.0.gem
Fetching swd-1.3.0.gem
Fetching webfinger-1.2.0.gem
Fetching fluent-plugin-splunk-hec-1.2.9.gem
Fetching rack-2.2.3.gem
Fetching rack-oauth2-1.19.0.gem
Fetching openid_connect-1.1.8.gem
Successfully installed connection_pool-2.2.5
Successfully installed net-http-persistent-3.1.0
Successfully installed attr_required-1.0.1
Successfully installed i18n-1.8.11
Successfully installed activesupport-7.0.0
Successfully installed activemodel-7.0.0
Successfully installed validate_url-1.0.13
Successfully installed mini_mime-1.1.2
Successfully installed mail-2.7.1
Successfully installed validate_email-0.1.6
Successfully installed aes_key_wrap-1.1.0
Successfully installed json-jwt-1.13.0
Successfully installed swd-1.3.0
Successfully installed webfinger-1.2.0
Successfully installed rack-2.2.3
Successfully installed rack-oauth2-1.19.0
Successfully installed openid_connect-1.1.8
Successfully installed fluent-plugin-splunk-hec-1.2.9
Parsing documentation for connection_pool-2.2.5
Installing ri documentation for connection_pool-2.2.5
Parsing documentation for net-http-persistent-3.1.0
Installing ri documentation for net-http-persistent-3.1.0
Parsing documentation for attr_required-1.0.1
Installing ri documentation for attr_required-1.0.1
Parsing documentation for i18n-1.8.11
Installing ri documentation for i18n-1.8.11
Parsing documentation for activesupport-7.0.0
Installing ri documentation for activesupport-7.0.0
Parsing documentation for activemodel-7.0.0
Installing ri documentation for activemodel-7.0.0
Parsing documentation for validate_url-1.0.13
Installing ri documentation for validate_url-1.0.13
Parsing documentation for mini_mime-1.1.2
Installing ri documentation for mini_mime-1.1.2
Parsing documentation for mail-2.7.1
Installing ri documentation for mail-2.7.1
Parsing documentation for validate_email-0.1.6
Installing ri documentation for validate_email-0.1.6
Parsing documentation for aes_key_wrap-1.1.0
Installing ri documentation for aes_key_wrap-1.1.0
Parsing documentation for json-jwt-1.13.0
Installing ri documentation for json-jwt-1.13.0
Parsing documentation for swd-1.3.0
Installing ri documentation for swd-1.3.0
Parsing documentation for webfinger-1.2.0
Installing ri documentation for webfinger-1.2.0
Parsing documentation for rack-2.2.3
Installing ri documentation for rack-2.2.3
Parsing documentation for rack-oauth2-1.19.0
Installing ri documentation for rack-oauth2-1.19.0
Parsing documentation for openid_connect-1.1.8
Installing ri documentation for openid_connect-1.1.8
Parsing documentation for fluent-plugin-splunk-hec-1.2.9
Installing ri documentation for fluent-plugin-splunk-hec-1.2.9
Done installing documentation for connection_pool, net-http-persistent, attr_required, i18n, activesupport, activemodel, validate_url, mini_mime, mail, validate_email, aes_key_wrap, json-jwt, swd, webfinger, rack, rack-oauth2, openid_connect, fluent-plugin-splunk-hec after 22 seconds
18 gems installed

Result for td-agent 3

$ rpm -qa | grep td-agent
td-agent-3.8.1-0.el7.x86_64

$ sudo td-agent-gem install fluent-plugin-splunk-hec
Fetching: connection_pool-2.2.5.gem (100%)
Successfully installed connection_pool-2.2.5
Fetching: net-http-persistent-3.1.0.gem (100%)
Successfully installed net-http-persistent-3.1.0
Fetching: attr_required-1.0.1.gem (100%)
Successfully installed attr_required-1.0.1
Fetching: i18n-1.8.11.gem (100%)
Successfully installed i18n-1.8.11
Fetching: activesupport-7.0.0.gem (100%)
ERROR:  Error installing fluent-plugin-splunk-hec:
        activesupport requires Ruby version >= 2.7.0.
$ sudo td-agent-gem install -f fluent-plugin-splunk-hec
Successfully installed activesupport-7.0.0
Fetching: activemodel-7.0.0.gem (100%)
Successfully installed activemodel-7.0.0
Fetching: validate_url-1.0.13.gem (100%)
Successfully installed validate_url-1.0.13
Fetching: mini_mime-1.1.2.gem (100%)
Successfully installed mini_mime-1.1.2
Fetching: mail-2.7.1.gem (100%)
Successfully installed mail-2.7.1
Fetching: validate_email-0.1.6.gem (100%)
Successfully installed validate_email-0.1.6
Fetching: bindata-2.4.10.gem (100%)
Successfully installed bindata-2.4.10
Fetching: aes_key_wrap-1.1.0.gem (100%)
Successfully installed aes_key_wrap-1.1.0
Fetching: json-jwt-1.13.0.gem (100%)
Successfully installed json-jwt-1.13.0
Fetching: swd-1.3.0.gem (100%)
Successfully installed swd-1.3.0
Fetching: webfinger-1.2.0.gem (100%)
Successfully installed webfinger-1.2.0
Fetching: rack-2.2.3.gem (100%)
Successfully installed rack-2.2.3
Fetching: rack-oauth2-1.19.0.gem (100%)
Successfully installed rack-oauth2-1.19.0
Fetching: openid_connect-1.1.8.gem (100%)
Successfully installed openid_connect-1.1.8
Fetching: prometheus-client-2.1.0.gem (100%)
Successfully installed prometheus-client-2.1.0
Fetching: fluent-plugin-splunk-hec-1.2.9.gem (100%)
Successfully installed fluent-plugin-splunk-hec-1.2.9
Parsing documentation for activesupport-7.0.0
Installing ri documentation for activesupport-7.0.0
Parsing documentation for activemodel-7.0.0
Installing ri documentation for activemodel-7.0.0
Parsing documentation for validate_url-1.0.13
Installing ri documentation for validate_url-1.0.13
Parsing documentation for mini_mime-1.1.2
Installing ri documentation for mini_mime-1.1.2
Parsing documentation for mail-2.7.1
Installing ri documentation for mail-2.7.1
Parsing documentation for validate_email-0.1.6
Installing ri documentation for validate_email-0.1.6
Parsing documentation for bindata-2.4.10
Installing ri documentation for bindata-2.4.10
Parsing documentation for aes_key_wrap-1.1.0
Installing ri documentation for aes_key_wrap-1.1.0
Parsing documentation for json-jwt-1.13.0
Installing ri documentation for json-jwt-1.13.0
Parsing documentation for swd-1.3.0
Installing ri documentation for swd-1.3.0
Parsing documentation for webfinger-1.2.0
Installing ri documentation for webfinger-1.2.0
Parsing documentation for rack-2.2.3
Installing ri documentation for rack-2.2.3
Parsing documentation for rack-oauth2-1.19.0
Installing ri documentation for rack-oauth2-1.19.0
Parsing documentation for openid_connect-1.1.8
Installing ri documentation for openid_connect-1.1.8
Parsing documentation for prometheus-client-2.1.0
Installing ri documentation for prometheus-client-2.1.0
Parsing documentation for fluent-plugin-splunk-hec-1.2.9
Installing ri documentation for fluent-plugin-splunk-hec-1.2.9
Done installing documentation for activesupport, activemodel, validate_url, mini_mime, mail, validate_email, bindata, aes_key_wrap, json-jwt, swd, webfinger, rack, rack-oauth2, openid_connect, prometheus-client, fluent-plugin-splunk-hec after 28 seconds
16 gems installed

I wasn't able to install the gem directly, but was able to install using -f flag. You need to set require_ssl_min_version to false value if you are using td-agent 3.

pranavmarla commented 2 years ago

Thanks @harshit-splunk . I was able to confirm that this issue is no longer present when installing the Splunk plugin with td-agent 4 (specifically, the Splunk plugin is now able to handle the newer version of tzinfo that is installed by td-agent 4). Presumably, this is because one of the gems @rockb1017 mentioned above fixed their dependencies.