Open Fryguy opened 3 years ago
Also ran the following on our specific RPMs:
rpm -qlv manageiq-gemset | sed 's/^.\{40\}//' | sort -nr | head -n 100
rpm -qlv manageiq-core | sed 's/^.\{40\}//' | sort -nr | head -n 100
rpm -qlv manageiq-ui | sed 's/^.\{40\}//' | sort -nr | head -n 100
lets go clockwise starting with 12:00
time | color | size | dir | contents |
---|---|---|---|---|
12:00 | purple | 61mb | bundle/gems | classic-ui: 29MB w/ app/javascript/locale: 12MB |
1:00 | pink | 64mb | vmdb../packs | vendor.js.map: 31mb, vendor.js: 22mb, vendor.js.gz: 4mb |
2:00 | pink | 26mb | ui/service | styles:18mb, assets:24mb, mostly with way too many ttf fonts |
2:45 | salmon | 32mb | locale | 10 locales @3.2mb |
3:00 | orange | 147mb | azure gem | |
6:00 | lime | 74mb | google-api | |
7:00 | green | 70mb | oci | |
7:45 | green | 27mb | azure again | I kid you not |
I blew out the green (gems) section which is 613.5mb:
note: graph shows whole gemset while file listing on the right is particular to just the gems. second biggest purple is content (automate being the whole disk usage)
azure is such a nasty gem - I wonder if we can find a way to affect upstream to avoid so much on-disk space wasting.
Remember that the total image size is around 800mb but different filesystems will show different numbers lets not focus more on >50mb wins. Cleaning up what is downloaded to the client has a bigger win than disk space.
Having multiple ui frameworks is hurting us. Standardizing on common fonts would gain us a lot. Reducing the fonts to just those that we use and using them across both ui interfaces (removing extra formats as mentioned before) is a big win - probably more than 100mb
I do wonder if removing fonts and sass from our pipelines would give us a big boost for rpm/image build time (a lot here) and disk usage. We could tweak our font usage too (this topic is over mentioned. sorry.)
azure_mgmt_*
are used by the azure_stack provider, and has been discontinued by microsoft unfortunately https://github.com/Azure/azure-sdk-for-ruby#important-announcement so I doubt we'll be able to effect any major changes there.
azure_mgmt_*
are used by the azure_stack provider, and has been discontinued by microsoft unfortunately https://github.com/Azure/azure-sdk-for-ruby#important-announcement so I doubt we'll be able to effect any major changes there.
It is no longer supported and they have an alternate interface. Moving to the new one should be a win. no?
Alternatively, if we can determine which azure api we are using and delete the ones that we are not would reduce the gem down from 146mb to 8mb. but this is hackery and would be better if we could get off of it.
That is signing up @agrare for work though. Not sure if they have an easy migration plan. (or if it is within my scope)
It is no longer supported and they have an alternate interface. Moving to the new one should be a win. no?
Are you talking about MsRestAzure
? This is not much more than a thin wrapper around the Rest APIs (see https://github.com/Azure/azure-sdk-for-ruby/blob/master/docs/create_a_virtual_machine.md)
Should we move to this? Yes, eventually Is it going to take a lot of refactoring and a lot of time/effort? Yes
So not a trivial drop-in replacement by any means
Our ruby and javascript calls:
Wonder what part of our app still uses the asset pipeline vs webpacker. If we just used webpacker, we would be able to remove a lot of processing and files from ruby.
part of the reason I would like to simplify patternfly and sass is it would be simpler for us to define the font formats we use (and delete all the extra ones) rather than have fonts automatically included and loose control of pruning these out
@Fryguy mentioned there are duplicate so files in both extensions/
and /gems/*/{lib,ext}
I thought ext
was where native gems were compiles. But many of these seem to have other plans.
Nokogiri looks like it has a number of entries for a number of different versions of something.
Total savings on the table is: 13mb
I think I remember that ruby code used different mechanisms to load a .so
so the directory it needed to be found was different. I seem to remember that some used something closer to require
/the path and found them in extensions, others used something closer to require_relative
/rb files and found it in /gems/
Is this triggering anything for anyone?
$ find . -name \*.so -exec du -k {} \; | sort -nr
3348 ./opt/manageiq/manageiq-gemset/gems/sassc-2.4.0/ext/libsass.so
3348 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/sassc-2.4.0/sassc/libsass.so
2312 ./opt/manageiq/manageiq-gemset/gems/nokogiri-1.12.3-x86_64-linux/lib/nokogiri/3.0/nokogiri.so
2308 ./opt/manageiq/manageiq-gemset/gems/nokogiri-1.12.3-x86_64-linux/lib/nokogiri/2.7/nokogiri.so
2304 ./opt/manageiq/manageiq-gemset/gems/nokogiri-1.12.3-x86_64-linux/lib/nokogiri/2.6/nokogiri.so
2304 ./opt/manageiq/manageiq-gemset/gems/nokogiri-1.12.3-x86_64-linux/lib/nokogiri/2.5/nokogiri.so
1708 ./opt/manageiq/manageiq-gemset/gems/rdkafka-0.9.0/ext/librdkafka.so
1648 ./opt/manageiq/manageiq-gemset/gems/rugged-1.1.1/ext/rugged/rugged.so
1648 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/rugged-1.1.1/rugged/rugged.so
484 ./opt/manageiq/manageiq-gemset/gems/qpid_proton-0.30.0/ext/cproton/cproton.so
484 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/qpid_proton-0.30.0/cproton.so
424 ./opt/manageiq/manageiq-gemset/gems/unf_ext-0.0.7.7/ext/unf_ext/unf_ext.so
424 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/unf_ext-0.0.7.7/unf_ext.so
304 ./opt/manageiq/manageiq-gemset/gems/pg-1.2.3/ext/pg_ext.so
304 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/pg-1.2.3/pg_ext.so
232 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/ffi-1.15.3/ffi_c.so
136 ./opt/manageiq/manageiq-gemset/gems/nio4r-2.5.8/ext/nio4r/nio4r_ext.so
136 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/nio4r-2.5.8/nio4r_ext.so
84 ./opt/manageiq/manageiq-gemset/gems/ovirt-engine-sdk-4.4.1/ext/ovirtsdk4c/ovirtsdk4c.so
84 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/ovirt-engine-sdk-4.4.1/ovirtsdk4c.so
64 ./opt/manageiq/manageiq-gemset/gems/escape_utils-1.2.1/ext/escape_utils/escape_utils.so
64 ./opt/manageiq/manageiq-gemset/gems/byebug-11.1.3/ext/byebug/byebug.so
64 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/escape_utils-1.2.1/escape_utils/escape_utils.so
64 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/byebug-11.1.3/byebug/byebug.so
60 ./opt/manageiq/manageiq-gemset/gems/puma-4.3.8/ext/puma_http11/puma_http11.so
60 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/puma-4.3.8/puma/puma_http11.so
52 ./opt/manageiq/manageiq-gemset/gems/json-2.5.1/ext/json/ext/generator/generator.so
52 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/json-2.5.1/json/ext/generator.so
44 ./opt/manageiq/manageiq-gemset/gems/http-parser-1.2.3/ext/x86_64-linux/libhttp-parser-ext.so
40 ./opt/manageiq/manageiq-gemset/gems/bcrypt_pbkdf-1.1.0/ext/mri/bcrypt_pbkdf_ext.so
40 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/bcrypt_pbkdf-1.1.0/bcrypt_pbkdf_ext.so
36 ./opt/manageiq/manageiq-gemset/gems/json-2.5.1/ext/json/ext/parser/parser.so
36 ./opt/manageiq/manageiq-gemset/gems/hamlit-2.11.1/ext/hamlit/hamlit.so
36 ./opt/manageiq/manageiq-gemset/gems/bcrypt-3.1.16/ext/mri/bcrypt_ext.so
36 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/json-2.5.1/json/ext/parser.so
36 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/hamlit-2.11.1/hamlit/hamlit.so
36 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/bcrypt-3.1.16/bcrypt_ext.so
28 ./opt/manageiq/manageiq-gemset/gems/racc-1.5.2/ext/racc/cparse/cparse.so
28 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/racc-1.5.2/racc/cparse.so
24 ./opt/manageiq/manageiq-gemset/gems/surro-gate-1.0.5/ext/surro-gate/selector_ext.so
24 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/surro-gate-1.0.5/surro-gate/selector_ext.so
12 ./opt/manageiq/manageiq-gemset/gems/websocket-driver-0.6.5/ext/websocket-driver/websocket_mask.so
12 ./opt/manageiq/manageiq-gemset/gems/memory_buffer-0.1.0/ext/memory_buffer/memory_buffer.so
12 ./opt/manageiq/manageiq-gemset/gems/linux_block_device-0.2.1/ext/linux_block_device/linux_block_device.so
12 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/websocket-driver-0.6.5/websocket_mask.so
12 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/memory_buffer-0.1.0/memory_buffer/memory_buffer.so
12 ./opt/manageiq/manageiq-gemset/extensions/x86_64-linux/2.6.0/linux_block_device-0.2.1/linux_block_device/linux_block_device.so
I can't find a rhyme or reason around the .so
files. It looks like some of them have extensions and some of them are in the build directory.
looking at js filies, I noticed a bunch of seemingly duplicate js files
# find js files, figure out their disk usage (in k) # put the filename at the beginning of the line for duplicate detection
find . -name \*.js -exec du -sk {} \; | sed 's|^\([^.]*\)\(\..*/\)\(.*\)|\3 \1\2\3|g' | sort
If you wanted to list the disk space at the beginning, then you out probably use something like \1\3 \2
noVNC
is listed 2 times (ui classic and ui self service), and each of the installs has a vendor
,core
, are duplicated in the lib
directory. So there are potentially 4 copies of each file.
So it seem strange when we have 6 copies of jquery and I think they are served as one of our app bundles (need to look back into asset pipeline how we are bundling jquery)
Our appliance is built to be able to be able to bundle / build the app. So it will, by definition, have most of the files in a build section and the destination section. So many resources will be duplicated. Not sure if that is the reasoning behind the so files.
https://github.com/ManageIQ/manageiq-ui-classic/pull/7838 may help drop sass and by extension, uglifier, execjs, nodejs
nodejs (https://github.com/ManageIQ/manageiq-ui-classic/issues/8300) is gone now as is python 2 (https://github.com/ManageIQ/manageiq-rpm_build/pull/287)
Updated numbers from the najdorf-1 build:
Seems intersight_client is a huge gem cc @agrare
$ du -sh /opt/manageiq/manageiq-gemset/gems/intersight_client-0.1.3
114M /opt/manageiq/manageiq-gemset/gems/intersight_client-0.1.3
~7 MB of that is a quick win - I'll send an update to their repo as 0.1.3 is including the .gem file from 0.1.2~ EDIT: Seems https://github.com/xlab-si/intersight-sdk-ruby/pull/16 actually already took care of this by ensuring only git committed files and so 0.1.4 has that fixed, but we hadn't pulled that into najdorf-1.
changing intersight client to drop docs and spec would save 76MB disk space - opened an issue on upstream to discuss: https://github.com/xlab-si/intersight-sdk-ruby/issues/17
This issue has been automatically marked as stale because it has not been updated for at least 3 months.
If you can still reproduce this issue on the current release or on master
, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions! More information about the ManageIQ triage process can be found in the triage process documentation.
That intersight yaml file is really big. Bet it has support for a ton of versioned endpoints
Also, the provider gems are all much bigger than I would have expected. Are we bringing in specs with links to core or something?
Think fixing providers would be a quick/low risk 10-20mb
Ran the following on our
manageiq-ui-worker
podified image and I see the below for package sizes:Original details - see later posts for updates
``` [root@f417e65db954 /]# rpm -qa --queryformat '%10{size} - %-25{name} \t %{version}\n' | sort -nr | head -n 100 457007231 - manageiq-gemset 13.0.0 111794506 - manageiq-ui 13.0.0 100970189 - ansible 2.9.24 68818612 - openscap 1.3.5 61689207 - gcc 8.5.0 56549434 - geolite2-city 20180605 38968954 - manageiq-core 13.0.0 38168125 - nodejs 12.22.3 36644942 - wmi 1.3.14 34013565 - git-core 2.27.0 33716339 - libicu 60.3 33060312 - nodejs-docs 12.22.3 32310333 - python3-libs 3.6.8 29681414 - cpp 8.5.0 28395840 - nodejs-full-i18n 12.22.3 26036720 - python2-libs 2.7.18 24926720 - binutils 2.30 23728688 - mozjs60 60.9.0 21663376 - samba-client-libs 4.14.5 20921752 - python3-babel 2.5.1 16214561 - python3-vspk 5.3.2 15698925 - npm 6.14.13 15646740 - glibc 2.28 14391428 - perl-interpreter 5.26.3 12482794 - git-core-doc 2.27.0 12275352 - glib2 2.56.4 11565734 - util-linux 2.32.1 11189195 - systemd 239 11082340 - ruby-libs 2.6.7 10826801 - freeipmi 1.6.8 10200695 - perl-Encode 2.97 9922363 - gnupg2 2.2.20 9815016 - cracklib-dicts 2.9.6 9284881 - NetworkManager-libnm 1.32.4 9248996 - glibc-common 2.28 9202588 - libxml2-devel 2.9.7 8801737 - python3-netaddr 0.7.19 8324320 - hwdata 0.314 8264117 - qemu-img 6.0.0 8211094 - python2-pip 9.0.3 8009345 - systemd-udev 239 7480993 - platform-python-pip 9.0.3 7408241 - NetworkManager 1.32.4 6861468 - bash 4.4.20 6660142 - GConf2 3.2.6 6386381 - file-libs 5.33 6281419 - glibc-langpack-en 2.28 6160361 - python3-docutils 0.14 6105197 - perl-libs 5.26.3 5936193 - python3-ovirt-engine-sdk4 4.4.13 5659880 - python3-policycoreutils 2.9 5307300 - kernel-headers 4.18.0 5025844 - python3-lxml 4.2.3 4561344 - postfix 3.5.8 4556468 - subscription-manager 1.28.13 4540286 - systemd-libs 239 4488264 - httpd 2.4.37 4261616 - glusterfs-client-xlators 8.5 4217216 - gsettings-desktop-schemas 3.32.0 4216866 - groff-base 1.22.3 4034012 - python3-libdnf 0.55.0 4018905 - shadow-utils 4.6 3915158 - python3-passlib 1.7.2 3825697 - libsodium-devel 1.0.18 3757208 - openssl-libs 1.1.1g 3669178 - cockpit-system 249 3424334 - geolite2-country 20180605 3346137 - python3-future 0.18.2 3270833 - isl 0.16.1 3270660 - net-snmp-libs 5.8 3168332 - python2-setuptools 39.0.1 2990455 - gnutls 3.6.14 2930503 - platform-python-setuptools 39.2.0 2876729 - tar 1.30 2873607 - python3-cryptography 3.2.1 2866122 - glusterfs 8.5 2760053 - PackageKit 1.1.12 2706225 - python3-setools 4.3.0 2699886 - gawk 4.2.1 2657576 - gdk-pixbuf2 2.36.12 2627103 - pam 1.3.1 2611832 - python3-jinja2 2.10.1 2607520 - setroubleshoot-plugins 3.3.13 2569940 - openssh-clients 8.0p1 2549569 - kbd-misc 2.0.4 2442451 - shared-mime-info 1.9 2395905 - iproute 5.12.0 2385867 - sos 4.1 2343195 - libdnf 0.55.0 2305955 - cockpit-ws 249 2282856 - net-snmp-agent-libs 5.8 2237891 - krb5-libs 1.18.2 2160874 - python3-psutil 5.6.3 2086913 - rpm 4.14.3 2085187 - dnf 4.4.2 2077004 - libuser 0.62 2075095 - cryptsetup-libs 2.3.3 2030257 - glibc-headers 2.28 1922619 - pcre2-devel 10.32 1918153 - openssh 8.0p1 ```@bdunne I'm wondering if there are things in this list maybe we can drop or shrink?