search

treasure-data / omnibus-td-agent

td-agent (Fluentd) Packaging Scripts
https://docs.treasuredata.com/articles/td-agent-changelog
Apache License 2.0
82 stars 131 forks source link

Custom package build fail #93

Open KeNaCo opened 8 years ago

KeNaCo commented 8 years ago

Hi, I try to build custom debian package with some plugins and also custom name to recogize it.

But build end with ackward 

root@9a7373d2c11f (master):~/dev# bin/omnibus build sklik-td-agent2
                              [CLI] I | Using config from 'omnibus.rb'
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
Building sklik-td-agent 2.3.2...
            [Software: preparation] I | Resolving manifest entry for preparation
         [NullFetcher: preparation] I | Fetching `preparation' (nothing to fetch)
               [Software: jemalloc] I | Resolving manifest entry for jemalloc
                   [Software: zlib] I | Resolving manifest entry for zlib
                [Software: cacerts] I | Resolving manifest entry for cacerts
                 [Software: xproto] I | Resolving manifest entry for xproto
            [Software: util-macros] I | Resolving manifest entry for util-macros
           [Software: config_guess] I | Resolving manifest entry for config_guess
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
        [Software: pkg-config-lite] I | Resolving manifest entry for pkg-config-lite
             [Software: makedepend] I | Resolving manifest entry for makedepend
                [Software: openssl] I | Resolving manifest entry for openssl
                [Software: ncurses] I | Resolving manifest entry for ncurses
                [Software: libedit] I | Resolving manifest entry for libedit
                [Software: libtool] I | Resolving manifest entry for libtool
                 [Software: libffi] I | Resolving manifest entry for libffi
                [Software: libyaml] I | Resolving manifest entry for libyaml
               [Software: libiconv] I | Resolving manifest entry for libiconv
                   [Software: ruby] I | Resolving manifest entry for ruby
                [Software: liblzma] I | Resolving manifest entry for liblzma
                [Software: libxml2] I | Resolving manifest entry for libxml2
                [Software: libxslt] I | Resolving manifest entry for libxslt
               [Software: rubygems] I | Resolving manifest entry for rubygems
               [Software: nokogiri] I | Resolving manifest entry for nokogiri
            [NullFetcher: nokogiri] I | Fetching `nokogiri' (nothing to fetch)
             [Software: postgresql] I | Resolving manifest entry for postgresql
                [Software: fluentd] I | Resolving manifest entry for fluentd
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                       [GitFetcher] W | git fetch on a sha1 is not guaranteed to work
                       [GitFetcher] W | Specify a ref name instead of dc4e6d7dbd28f2c00123a395519114a81f280d31 on {:git=>"https://github.com/fluent/fluentd.git"}
              [GitFetcher: fluentd] I | Fetching from `https://github.com/fluent/fluentd.git'
               [Software: td-agent] I | Resolving manifest entry for td-agent
            [NullFetcher: td-agent] I | Fetching `td-agent' (nothing to fetch)
         [Software: td-agent-files] I | Resolving manifest entry for td-agent-files
      [NullFetcher: td-agent-files] I | Fetching `td-agent-files' (nothing to fetch)
             [Software: fluentd-ui] I | Resolving manifest entry for fluentd-ui
             [Software: fluentd-ui] W | Version 8a322e7716370a1e2fdf69fda3266a1be477f221 for software fluentd-ui was not parseable. Comparison methods such as #satisfies? will not be available for this version.
             [Software: fluentd-ui] W | Version 8a322e7716370a1e2fdf69fda3266a1be477f221 for software fluentd-ui was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                       [GitFetcher] W | git fetch on a sha1 is not guaranteed to work
                       [GitFetcher] W | Specify a ref name instead of 8a322e7716370a1e2fdf69fda3266a1be477f221 on {:git=>"https://github.com/fluent/fluentd-ui.git"}
           [GitFetcher: fluentd-ui] I | Fetching from `https://github.com/fluent/fluentd-ui.git'
                     [Software: td] I | Resolving manifest entry for td
                  [NullFetcher: td] I | Fetching `td' (nothing to fetch)
            [Software: td-agent-ui] I | Resolving manifest entry for td-agent-ui
         [NullFetcher: td-agent-ui] I | Fetching `td-agent-ui' (nothing to fetch)
       [Software: td-agent-cleanup] I | Resolving manifest entry for td-agent-cleanup
    [NullFetcher: td-agent-cleanup] I | Fetching `td-agent-cleanup' (nothing to fetch)
       [Software: version-manifest] I | Resolving manifest entry for version-manifest
    [NullFetcher: version-manifest] I | Fetching `version-manifest' (nothing to fetch)
         [GitFetcher: config_guess] I | Fetching from `git://git.sv.gnu.org/config.git'
            [Software: preparation] I | Restored from cache
               [Software: jemalloc] I | Restored from cache
                   [Software: zlib] I | Restored from cache
                [Software: cacerts] I | Restored from cache
                 [Software: xproto] I | Restored from cache
            [Software: util-macros] I | Restored from cache
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
           [Software: config_guess] W | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
           [Software: config_guess] I | Restored from cache
        [Software: pkg-config-lite] I | Restored from cache
             [Software: makedepend] I | Restored from cache
                [Software: openssl] I | Restored from cache
                [Software: ncurses] I | Restored from cache
                [Software: libedit] I | Restored from cache
                [Software: libtool] I | Restored from cache
                 [Software: libffi] I | Restored from cache
                [Software: libyaml] I | Restored from cache
               [Software: libiconv] I | Restored from cache
                   [Software: ruby] I | Restored from cache
                [Software: liblzma] I | Restored from cache
                [Software: libxml2] I | Restored from cache
                [Software: libxslt] I | Restored from cache
               [Software: rubygems] I | Restored from cache
               [Software: nokogiri] W | No version given! This is probably a bad thing. I am going to assume the version `0.0.0', but that is most certainly not your desired behavior. If git caching seems off, this is probably why.
               [Software: nokogiri] I | Restored from cache
             [Software: postgresql] I | Restored from cache
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                [Software: fluentd] W | Version dc4e6d7dbd28f2c00123a395519114a81f280d31 for software fluentd was not parseable. Comparison methods such as #satisfies? will not be available for this version.
                [Software: fluentd] I | Restored from cache
               [Software: td-agent] W | No version given! This is probably a bad thing. I am going to assume the version `0.0.0', but that is most certainly not your desired behavior. If git caching seems off, this is probably why.
               [Software: td-agent] I | Restored from cache
         [Software: td-agent-files] W | No version given! This is probably a bad thing. I am going to assume the version `0.0.0', but that is most certainly not your desired behavior. If git caching seems off, this is probably why.
         [Software: td-agent-files] I | Could not restore from cache
          [Builder: td-agent-files] I | Starting build
          [Builder: td-agent-files] I | <Dynamic Ruby block>: 0.0289s
          [Builder: td-agent-files] I | Build td-agent-files: 0.0296s
/opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1396:in `copy': unknown file type: /root/dev/resources/sklik-td-agent/etc (RuntimeError)
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:469:in `block in copy_entry'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1506:in `call'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1506:in `wrap_traverse'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:466:in `copy_entry'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:441:in `block in cp_r'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1579:in `block in fu_each_src_dest'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1593:in `fu_each_src_dest0'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:1577:in `fu_each_src_dest'
    from /opt/ruby2.1/lib/ruby/2.1.0/fileutils.rb:440:in `cp_r'
    from /root/dev/config/software/td-agent-files.rb:65:in `block (2 levels) in evaluate'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:1012:in `instance_eval'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:1012:in `run'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:843:in `block (3 levels) in execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:865:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:865:in `with_retries'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:842:in `block (2 levels) in execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `measure'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:841:in `block in execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:915:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:915:in `with_clean_env'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:840:in `execute'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:704:in `block (2 levels) in build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:703:in `each'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:703:in `block in build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `call'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/instrumentation.rb:23:in `measure'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/builder.rb:702:in `build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/software.rb:1142:in `execute_build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/software.rb:1010:in `build_me'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/project.rb:1077:in `block in build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/project.rb:1076:in `each'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/project.rb:1076:in `build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/cli.rb:84:in `build'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/command.rb:27:in `run'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/invocation.rb:126:in `invoke_command'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor.rb:359:in `dispatch'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/cli/base.rb:33:in `dispatch'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/thor-0.19.1/lib/thor/base.rb:440:in `start'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/lib/omnibus/cli.rb:42:in `execute!'
    from /opt/ruby2.1/lib/ruby/gems/2.1.0/gems/omnibus-5.4.0/bin/omnibus:16:in `<top (required)>'
    from bin/omnibus:16:in `load'
    from bin/omnibus:16:in `<main>'

And I have just added ./config/projects/sklik-td-agent2.rb and change name, maintainer, homepage and description, and also add custom sklik_plugin_gems.rb

What is suspicious for me is resources folder name:

root@9a7373d2c11f (master):~/dev/resources# ls
sklik-sklik-td-agent  td-agent

Is there problem on my side? Thanks for any help.

repeatedly commented 8 years ago

Maybe you need to rename from sklik-sklik-td-agent to sklik-td-agent, project name.

KeNaCo commented 8 years ago

But this is auto generated folder. Project file name is sklik-td-agent2.rb and package name should be sklik-td-agent.

repeatedly commented 8 years ago

That's weird. Auto generated file path should be sklik-td-agent...

KeNaCo commented 8 years ago

Can you please confirm it's an upstream bug? Or where can be error if it's not? More details about my changes: add ./config/projects/sklik-td-agent2.rb:

require 'erb'
require 'fileutils'
require 'rubygems'

name "sklik-td-agent"
maintainer "Jan Porhincak <jan.porhincak@firma.seznam.cz>"
homepage ""
description ""

install_dir     "/opt/sklik-td-agent"
build_version   "2.3.2"
build_iteration 0

# creates required build directories
dependency "preparation"

override :ruby, :version => '2.1.10'
override :zlib, :version => '1.2.8'
override :rubygems, :version => '2.4.8'
override :postgresql, :version => '9.3.5'

# td-agent dependencies/components
dependency "td-agent"
dependency "td-agent-files"
dependency "td"
dependency "td-agent-ui"
dependency "td-agent-cleanup"

# version manifest file
dependency "version-manifest"

case ohai["os"]
when "linux"
  case ohai["platform_family"]
  when "debian"
    runtime_dependency "lsb-base"
  when "rhel"
    runtime_dependency "initscripts"
    if ohai["platform_version"][0] == "5"
      runtime_dependency "redhat-lsb"
    else
      runtime_dependency "redhat-lsb-core"
    end
  end
end

exclude "\.git*"
exclude "bundler\/git"

compress :dmg do
end

add ./sklik_plugin_gems.rb:

dir 'plugin_gems'
download "httpclient", "2.7.2"
download "td-client", "0.8.81"
download "td", "0.14.0"
download "fluent-plugin-td", "0.10.28"
download "uuidtools", "2.1.5"
download "string-scrub", "0.0.5"
download "fluent-plugin-rewrite-tag-filter", "1.5.5"
download "fluent-plugin-td-monitoring", "0.2.2"
download "fluent-plugin-prometheus", "0.1.3"
repeatedly commented 8 years ago

google-fluentd and several companies can build own package via modified omnibus-td-agent, so master should work.

Can you please confirm it's an upstream bug?

I'm now busy for releasing tasks / fluentd stuff. After finished these tasks, I will check it.

adie commented 5 years ago

I had the same problem, resolved by removing this line as all paths are already contain proper project name: https://github.com/treasure-data/omnibus-td-agent/blob/master/config/software/td-agent-files.rb#L25 So it looks like the problem is reproducable only when custom name contains td-agent part.