search

g1mv / density

Superfast compression library
Apache License 2.0
1.02k stars 48 forks source link

Extremely large compiled size #55

Closed nemequ closed 6 years ago

nemequ commented 9 years ago

[See http://encode.ru/threads/2313-Brotli?p=45406&viewfull=1#post45406 for more background]

I just checked the sizes of all the Squash plugins, and density is more than thrice the size of the next largest plugin (brotli):

3.3M density/libsquash0.8-plugin-density.so
780K brotli/libsquash0.8-plugin-brotli.so
296K lzham/libsquash0.8-plugin-lzham.so
284K zpaq/libsquash0.8-plugin-zpaq.so
276K lzo/libsquash0.8-plugin-lzo.so
228K bsc/libsquash0.8-plugin-bsc.so
212K lzma/libsquash0.8-plugin-lzma.so
116K zlib/libsquash0.8-plugin-zlib.so
112K lz4/libsquash0.8-plugin-lz4.so
112K zstd/libsquash0.8-plugin-zstd.so
 96K zlib-ng/libsquash0.8-plugin-zlib-ng.so
 92K bzip2/libsquash0.8-plugin-bzip2.so
 76K gipfeli/libsquash0.8-plugin-gipfeli.so
 72K ms-compress/libsquash0.8-plugin-ms-compress.so
 56K zling/libsquash0.8-plugin-zling.so
 48K snappy/libsquash0.8-plugin-snappy.so
 24K yalz77/libsquash0.8-plugin-yalz77.so
 24K wflz/libsquash0.8-plugin-wflz.so
 24K heatshrink/libsquash0.8-plugin-heatshrink.so
 24K fastlz/libsquash0.8-plugin-fastlz.so
 20K lzg/libsquash0.8-plugin-lzg.so
 20K crush/libsquash0.8-plugin-crush.so
 20K pithy/libsquash0.8-plugin-pithy.so
 20K lzf/libsquash0.8-plugin-lzf.so
 20K ncompress/libsquash0.8-plugin-ncompress.so
 16K quicklz/libsquash0.8-plugin-quicklz.so
 16K brieflz/libsquash0.8-plugin-brieflz.so
 16K fari/libsquash0.8-plugin-fari.so
 16K lzjb/libsquash0.8-plugin-lzjb.so
 12K copy/libsquash0.8-plugin-copy.so

I haven't done any research, but I'm pretty confident the massive amount of inlining is to blame. Perhaps it may be time to take a serious look at which plugins really need to be inlined and which don't; this might even improve performance since more of the code could fit in the CPU cache.

One idea could also be a two-level system, where if someone defines something in the preprocessor density becomes a bit more conservative in what it inlines (e.g., have DENSITY_INLINE and DENSITY_ALWAYS_INLINE, and define the former as "" if DENSITY_DONT_ALWAYS_INLINE is defined). In this case, you could also add __attribute__((__hot__)) (for GCC at least, other compilers may have something similar) to DENSITY_ALWAYS_INLINE.

g1mv commented 9 years ago

Thanks, you're right there is a massive amount of inlining. I have started to tidy up the code base in 0.13.0 though, I'll integrate this maybe in some later version like 0.13.1.

g1mv commented 6 years ago

Has to be retested with 0.14.0