Time to goodbye omnibus?

[repeatedly]

We introduced omnibus to reduce maintenance cost for many platforms several years ago. I'm not an expert for deb/rpm/dmg/msi packaging and the maintainer was only me, so omnibus is good project before.

The drawbacks are:

• Update omnibus and omnibus-software sometimes break the build process
• Update chef also breaks the build process
• Can't use system openssl
• Need to setup chef and it takes longer time
• Need to wait update for new platform
• Hard to accept the patch from deb/rpm expert

We have some experience and we now have container/CI/CD tools for packaging. Time to consider more light-weight and better packaging mechanism for td-agent.

[pzakha]

Another good reason to drop Omnibus support is that it is hard to use in enterprise settings that require internal mirroring of all 3rd party build dependencies. Since Omnibus pulls all the dependencies from different external sources and there is no way to guarantee that those sources will always be available (such as seen in this recent incident: https://github.com/treasure-data/omnibus-td-agent/issues/216), the build can be unstable.

[kenhys]

Hi, (I'm a one of a Debian Maintainer so it may have some bias in my opinion)

Surely Omnibus based packaging has some merit for many platforms(we can install td-agent as a package and it doesn't affect system's ruby and so on), but as repeatedly described above, It seems that there are some drawbacks.

If there is not strong reason to provide td-agent in one single binary package (td-agent.deb or .rpm), it may be better to split it into multiple packages in point view of package maintainer for simplicity.

For example, split current td-agent into two packages:

• td-ruby (ruby package for td-agent)
• td-agent (bundle required gems and configurations)

In above packages, it assume that it is able to execute debuild or rpmbuild directly (it means that debian/* or spec file is not separated) without Omnibus, so things become simple (it can be able to get feedback from other deb/rpm maintainer) and td-ruby will be able to use system's openssl, thus, it means that it makes easy to follow security updates.

If build process is simplified as above, it makes easy to migrate into docker based build process. After that, it makes easy to integrate existing CI service such as Travis CI, GitHub Actions and so on. How do you think about split package and make them docker ready?

[kenhys]

https://github.com/treasure-data/omnibus-td-agent/issues/219#issuecomment-549638661 doesn't care about Windows or macOS cases, I'll plan to update description.

[kenhys]

On Windows or macOS replacing omnibus is not so good idea because it seems that it is similar to reinventing the wheel in current status and need to tackle for it

• Still need to bundle OpenSSL for Ruby (Windows/macOS) is not changed
• Add replacement way to support msi or dmg (Wix based?/pkgbuild?)

Instead, improving to use CI service(Travis?) may be better for light-weight mechanism for td-agent.(One concern is execution timeout on such CI service) How do you think about this approach?

[ashie]

On Windows or macOS replacing omnibus is not so good idea because it seems that it is similar to reinventing the wheel in current status and need to tackle for it

I'm now considering other solutions.

• It would be better to replace OpenSSL and other bundled software with system's one or well-maintained 3rd party packages.
• It would be better to introduce modern package management system like apt or dnf (yum) also on these platforms to manage such packages easily.

On macOS, Homebrew, Fink and MacPorts are well known. Probably Homebrew is the best solution recent days. On Windows, Chocolatey seems pretty.

BTW recent Windows has a package management system "PackageManagement" (a.k.a "OneGet") by default which abstracts several PackageManagementProviders like NuGet or Chocolatey.

[cosmo0920]

Homebrew seems good. Like mikutter does. https://github.com/Homebrew/homebrew-core/blob/371a81a6b53aabe8ab09206a25592871a06637ee/Formula/mikutter.rb

[ashie]

I'm sorry for long silence. As @repeatedly mentioned in https://github.com/treasure-data/omnibus-td-agent/issues/209#issuecomment-599378907, we are developing a new lightweight build sytem for td-agent:

• https://github.com/clear-code/td-agent-builder
  • (We'll transfer ownership of this project to treasure-data.)

Now it's ready for testing! Please feel free to test it & report issues.

Current status of this build system:

• Docker base build system
  • Based on Apache Arrow's one: https://github.com/apache/arrow/tree/master/dev/tasks/linux-packages
• Sources of td-agent are taken from omnibus-td-agent
  • configuration files
  • package scripts
  • run time scripts
• Use system's OpenSSL and some other libraries instead of bundling them into our package
• Ruby is still bundle in our package to avoid conflict of gems
  • For GNU/Linux: Build it from source tarball
  • For Windows: Use 7-ZIP binary of RubyInstaller
• Supported platforms:
  • CentOS 7
  • CentOS 8
  • Debian 10 "Buster"
  • Ubuntu 18.04 "Bionic Beaver"
  • Windows 10
  • Windows 2016 or later
• Platforms to be supported in the near future
  • CentOS 6
  • Ubuntu 20.04 "Focal Fossa"
• Drop fluntd-ui

[repeatedly]

Thanks! I will try td-agent 4 build with td-agent-builder :)

[ashie]

Additional comments:

• CentOS 6 support has been landed.
• The path element "embedded" is removed in new packages.
  • It comes from Omnibus so that I think we don't need it in the new packages.
  • Most files are installed just under /opt/td-agent instead of /opt/td-agent/embedded.
• While I was porting the build system, I noticed that Omnibus squashes many warning or errors.
  • Some of them are fixed in td-agent-bulder. e.g.: #230
  • But dozen of errors are still remained. We'll ensure to fix them correctly.
• PostgreSQL isn't bundled.
  • In my understanding it's only for avoiding conflict of OpenSSL.
  • https://github.com/treasure-data/omnibus-td-agent/commit/41f88b427a60e14b678c6f93ec5b4f899a14530c
  • This issue doesn't exist in the new build system since it uses system's OpenSSL.
• Packages are built automatically on every push event. e.g.:
  • GNU/Linux x86_64
  • Windows
  • Not yet for GNU/Linux AArch64 but it will be comming soon.

[repeatedly]

The path element "embedded" is removed in new packages

Awesome :)

[repeatedly]

Basic work of td-agent-builder is done.