Support Gemfile based plugin management

okkez commented 6 years ago

Continuation of #166.

What is problem

We can see following permissions after td-agent3 installation:

$ ls -laR /etc/td-agent/
/etc/td-agent/:
total 16
drwxr-xr-x  3 td-agent td-agent 4096 Mar  1 08:44 .
drwxr-xr-x 57 root     root     4096 Mar  5 06:14 ..
drwxr-xr-x  2 root     root     4096 Mar  1 08:43 plugin
-rw-r--r--  1 root     root     2381 Mar  1 08:44 td-agent.conf

/etc/td-agent/plugin:
total 8
drwxr-xr-x 2 root     root     4096 Mar  1 08:43 .
drwxr-xr-x 3 td-agent td-agent 4096 Mar  1 08:44 ..

Bundler 1.16.0 checks permissions around --gemfile and --gem-path. For example, we have /etc/td-agent/Gemfile and /etc/td-agent/vendor/bundle does not exist. Bundler checks permission like this.

td-agent will fail to start process with td-agent.service like https://github.com/treasure-data/omnibus-td-agent/issues/166#issue-300500350, because /etc/td-agent/td-agent.conf and /etc/td-agnet/plugins are not writable by td-agent user.

So, we should consider --gemfile and --gem-path options.

About drop-in file

We can use systemd drop-in file instead of overwriting td-agent.service by /etc/systemd/system/td-agent.tservice like https://github.com/treasure-data/omnibus-td-agent/issues/166#issue-300500350.

$ sudo systemctl edit td-agent.service

and creating following /etc/systemd/system/td-agent.service.d/override.conf:

[Service]
Environment=TD_AGENT_OPTIONS='--gemfile=/etc/td-agent/Gemfile'
ExecStart=
ExecStart=/opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS

Proposal

Adding documents about Gemfile based plugin management to fluentd-docs repository.

Use following drop-in file /etc/systemd/system/td-agent.service.d/override.conf for td-agent 3.1.1 :

[Service]
Environment='TD_AGENT_OPTIONS=--gemfile=/etc/td-agent/Gemfile --gem-path=/var/lib/td-agent/vendor/bundle'
ExecStart=
ExecStart=/opt/td-agent/embedded/bin/fluentd --log /var/log/td-agent/td-agent.log --daemon /var/run/td-agent/td-agent.pid $TD_AGENT_OPTIONS

For next version of td-agent:

[Service]
Environment='TD_AGENT_OPTIONS=--gemfile=/etc/td-agent/Gemfile --gem-path=/var/lib/td-agent/vendor/bundle'

And add Gemfile example:

source "https://rubygems.org"

TD_AGENT_GEM_HOME = "/opt/td-agent/embedded/lib/ruby/gems"
def td_agent_gem(gem_name)
  gems = Dir.glob("#{TD_AGENT_GEM_HOME}/*/gems/#{gem_name}-[0-9]*").reverse
  return if gems.empty?
  gem gem_name, gems.first.split('-').last
end

td_agent_gem "fluentd"
# td_agent_gem "fluent-plugin-elasticsearch"
# td_agent_gem "fluent-plugin-kafka"
# td_agent_gem "fluent-plugin-rewrite-tag-filter"
# td_agent_gem "fluent-plugin-s3"
# td_agent_gem "fluent-plugin-td"
# td_agent_gem "fluent-plugin-td-monitoring"
# td_agent_gem "fluent-plugin-webhdfs"

require "pathname"
gemfile_local = Pathname(__dir__) + "Gemfile.local"
if gemfile_local.exist?
  instance_eval(gemfile_local.read)
end

See: https://qiita.com/toyama0919/items/c4d319bf819c01e5d73e (Japanese article)

However we must download gem packages when we use Gemfile based plugin management even if we use td-agent bundled gems only. This is a disadvantage of Gemfile based plugin management.

okkez commented 5 years ago

Maybe we need ExecStartPre to upgrade td-agent version. I'm testing following command:

ExecStartPre=-/bin/sh -c "/opt/td-agent/embedded/bin/bundle check --gemfile=/etc/td-agent/Gemfile || /opt/td-agent/embedded/bin/bundle install --gemfile=/etc/td-agent/Gemfile --path=/var/lib/td-agent/vendor/bundle || /usr/bin/env BUNDLE_GEMFILE=/etc/td-agent/Gemfile /opt/td-agent/embedded/bin/bundle update"

NOTE: Bundler 1.17.1 supports bundle update --gemfile=..., but 1.16.2 does not support it.

kailashyogeshwar85 commented 4 years ago

Doesn't seem to work yet.

treasure-data / omnibus-td-agent