Investigate optimizing the size of the td-agent package?

[mr-salty]

Hi, I have noticed the td-agent package is quite large at ~75MB, and when installed it consumes ~200MB in /opt. I did a brief investigation, here are some notes (all of these are based on the installed size):

• none of the binaries/libraries are stripped - doing this saves about 50MB.
• could we remove the directory ruby/gems/2.1.0/cache? That is 28MB.
• is it necessary to preserve the static .a libraries? This is less of a gain after stripping but still 8MB.
• The 'td' gem includes ruby-2.0.0-p0.pkg which is 13MB - is that needed? I see some other large files packaged by the gems that look like they may be unnecessary, although these might be harder to remove:
  • ohai-6.20.0/spec/unit/plugins/darwin/system_profiler_output.rb (~3MB)
  • httpclient-2.4.0/sample/api_rest.pdf (~2MB)
  • td-0.11.8/data/sample_apache.json (~2MB)
  • nokogiri-1.6.3.1/test/files/slow-xpath.xml (~2MB)

If we are able to make all of those changes, the size of the installed /opt/td-agent is reduced from 206MB to 97MB. I didn't build a package but as an approximation I tar/gzipped the directory which was reduced in size from 76MB to 18MB (!).

This isn't an urgent problem but it would be nice to reduce the size.

[repeatedly]

Hi @mr-salty

Awesome investigation :+1: For td, including pkg is not needed and other some files could be removed. I will try to remove these files to reduce package size.

[repeatedly]

td command issue is resolved: https://github.com/treasure-data/td/pull/43 ruby's cache will be removed: https://github.com/treasure-data/omnibus-td-agent/pull/33

I will work on removing other files later.

[repeatedly]

Using latest commit, td-agent 2 size becomes 2x smaller than before.

none of the binaries/libraries are stripped - doing this saves about 50MB.

Do you have a list of stripped binaries?

[mr-salty]

Awesome, thanks!

For my experiment I just did something like find "/opt/td-agent -type f | xargs strip" but hopefully there is a better way to do it during the build process. I can look into it.

[repeatedly]

I see. Maybe, keeping the debug info of Ruby and related libraries is better. I will check which td-agent libraries could be stripped or not.

[mr-salty]

I was looking for information about stripping when building ruby gems (which would solve part of the problem). Although I thought this would be a common thing to do, I couldn't find much info about it.

It looks like it may be possible to pass some cflags (-s would be good) although I'm not entirely sure how to accomplish this for our purposes (do you have to configure every gem independently? That would be a pain): http://stackoverflow.com/questions/6761670/is-there-any-way-to-change-gcc-compilation-options-for-a-gem