Can't build v3.0.1

[dmigous]

Trying to build v3.0.1 and getting following error:

$ bin/gem_downloader core_gems.rb
$ bin/gem_downloader plugin_gems.rb
$ bin/omnibus build td-agent2
...
...
...
Time: 00:00:27 ======================================================================================================================================= 100% (605 KB/sec)
 [NetFetcher: postgresql] I | 2017-09-18T12:49:54+00:00 | Verifying checksum
   [NetFetcher: jemalloc] I | 2017-09-18T12:49:55+00:00 | Retrying failed download due to execution expired (5 retries left)...
   [NetFetcher: jemalloc] I | 2017-09-18T12:50:55+00:00 | Retrying failed download due to execution expired (4 retries left)...                            0% (0 KB/sec)
   [NetFetcher: jemalloc] I | 2017-09-18T12:51:55+00:00 | Retrying failed download due to execution expired (3 retries left)...                            0% (0 KB/sec)
   [NetFetcher: jemalloc] I | 2017-09-18T12:52:55+00:00 | Retrying failed download due to execution expired (2 retries left)...                            0% (0 KB/sec)
   [NetFetcher: jemalloc] I | 2017-09-18T12:53:55+00:00 | Retrying failed download due to execution expired (1 retries left)...                            0% (0 KB/sec)
   [NetFetcher: jemalloc] E | 2017-09-18T12:54:55+00:00 | Download failed - Net::OpenTimeout!                                                              0% (0 KB/sec)
/opt/ruby_build/builds/liquid/lib/ruby/2.3.0/net/http.rb:880:in `initialize': execution expired (Net::OpenTimeout)
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/net/http.rb:880:in `open'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/net/http.rb:880:in `block in connect'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/timeout.rb:101:in `timeout'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/net/http.rb:878:in `connect'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/net/http.rb:863:in `do_start'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/net/http.rb:852:in `start'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:319:in `open_http'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:737:in `buffer_open'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:212:in `block in open_loop'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:210:in `catch'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:210:in `open_loop'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:151:in `open_uri'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/core_extensions/open_uri.rb:51:in `open_uri'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:717:in `open'
    from /opt/ruby_build/builds/liquid/lib/ruby/2.3.0/open-uri.rb:35:in `open'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/download_helpers.rb:80:in `download_file!'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/fetchers/net_fetcher.rb:173:in `download'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/fetchers/net_fetcher.rb:86:in `fetch'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/software.rb:897:in `fetch'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/project.rb:1066:in `block (3 levels) in download'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/thread_pool.rb:64:in `block (4 levels) in initialize'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/thread_pool.rb:62:in `loop'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/thread_pool.rb:62:in `block (3 levels) in initialize'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/thread_pool.rb:61:in `catch'
    from /opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bundler/gems/omnibus-ffbda9ad7d37/lib/omnibus/thread_pool.rb:61:in `block (2 levels) in initialize'

It looks like connection problem, but all previous packages downloads well, and retried to run build multiple times, same issue. Maybe it is related to url it tries to download?

[michaelgillan-zz]

+1

[rennhak]

+1

[repeatedly]

Do you use latest jemalloc.rb? https://github.com/treasure-data/omnibus-td-agent/pull/141

[dmigous]

@repeatedly before tried v3.0.1, where not latest jemalloc.rb. Now tried to build from master, seems better. But getting another issue, probably you will have a quick idea where to start from:

The following shell command exited with status 1:

    $ CFLAGS=-I/opt/td-agent/embedded/include -O2 CPPFLAGS=-I/opt/td-agent/embedded/include -O2 CXXFLAGS=-I/opt/td-agent/embedded/include -O2 LDFLAGS=-Wl,-rpath,/opt/td-agent/embedded/lib -L/opt/td-agent/embedded/lib LD_RUN_PATH=/opt/td-agent/embedded/lib OMNIBUS_INSTALL_DIR=/opt/td-agent PATH=/opt/td-agent/bin:/opt/td-agent/embedded/bin:/opt/ruby_build/builds/liquid/lib/ruby/gems/2.3.0/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/ruby_build/builds/liquid/bin:/liquid/bin PKG_CONFIG_PATH=/opt/td-agent/embedded/lib/pkgconfig /opt/td-agent/embedded/bin/gem install --no-ri --no-rdoc /liquid/base/omnibus-td-agent/plugin_gems/17-fluent-plugin-td-monitoring-0.2.2.gem 

Output:

    Successfully installed systemu-2.5.2
Successfully installed mixlib-cli-1.7.0
Successfully installed mixlib-config-2.2.4
Successfully installed mixlib-log-1.7.1

Error:

    ERROR:  Error installing /liquid/base/omnibus-td-agent/plugin_gems/17-fluent-plugin-td-monitoring-0.2.2.gem:
    mixlib-shellout requires Ruby version >= 2.2.

I saw omnibus installs own ruby version, how to adjust it's version?

[repeatedly]

You seem to build td-agent 2, not td-agent 3. To build td-agent 3, use bin/omnibus build td-agent3, not bin/omnibus build td-agent2. And need BUILD_TD_AGENT_VERSION=3 env ver for gem_downloader like BUILD_TD_AGENT_VERSION=3 ./bin/gem_downloader plugin_gems.rb.

[dmigous]

yeah, was trying to build td-agent 2...

Changed to td-agent 3.

Still having issues... Now it is

      [Software: cacerts] I | 2017-09-26T20:43:49+00:00 | Resolving manifest entry for cacerts
       [NetFetcher: zlib] I | 2017-09-26T20:43:49+00:00 | Downloading from `http://downloads.sourceforge.net/project/libpng/zlib/1.2.8/zlib-1.2.8.tar.gz'

Progress: |
Progress: |Progress: |
Progress: | [Software: config_guess] I | 2017-09-26T20:43:49+00:00 | Resolving manifest entry for config_guess
 [Software: config_guess] W | 2017-09-26T20:43:49+00:00 | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
 [Software: config_guess] W | 2017-09-26T20:43:49+00:00 | Version master for software config_guess was not parseable. Comparison methods such as #satisfies? will not be available for this version.
[GitFetcher: config_guess] I | 2017-09-26T20:43:51+00:00 | Fetching from `https://github.com/chef/config-mirror.git'
=================================================================================|
       [NetFetcher: zlib] I | 2017-09-26T20:43:51+00:00 | Verifying checksum
STDERR: Verification for zlib failed due to a checksum mismatch:

    expected: 44d667c142d7cda120332623eab69f40
    actual:   cb530d737c8f2d1023797cf0587b4e05

This added security check is used to prevent MITM attacks when downloading the
remote file. If you have updated the version or URL for the download, you will
also need to update the checksum value. You can find the checksum value on the
software publisher's website.

[repeatedly]

it is not omnibus-td-agent issue. Currenlty, sourceforge has a problem: https://twitter.com/sfnet_ops/status/912764408643940355

[dmigous]

@repeatedly you're right. Now works. Many thanks for helping out

few questions:

1. at master and v3.0.1 README says $ bin/omnibus build td-agent2 (build td-agent2). Should it be changed to td-agent3?
2. here it is said

   If using Fluentd/td-agent for the first time/a new project, td-agent v2 is strongly recommended.

So it would be good for me to use v2 still... I tried to checkout v2.3.5 and patch there jemalloc.rb to latest (as at master), it fixed jemalloc related error, but still fails with

    ERROR:  Error installing /liquid/base/omnibus-td-agent/plugin_gems/17-fluent-plugin-td-monitoring-0.2.2.gem:
    mixlib-shellout requires Ruby version >= 2.2.

[dmigous]

@repeatedly kind reminder... is there any way now to build td-agent v2?

[repeatedly]

Use this branch: https://github.com/treasure-data/omnibus-td-agent/pull/143

[dmigous]

@repeatedly works for me the issue can be closed

Thank you so much!