all: binaries too big and growing

robpike commented 10 years ago

As an experiment, I build "hello, world" at the release points for go 1.0.
1.1, and 1.2. Here are the binary's sizes:

% ls -l x.1.?
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
% size x.1.?
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096    0   4112    34566848    20f72c0 x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
% 

A near-doubling of the binary size in two releases is a bug of a kind. I will hold on to
the files so they can be analyzed more, but am filing this issue to get the topic
registered. We need to develop a better understanding of the problem and how to address
it.

Marking this 1.3 (not maybe) because I consider it a priority.

A few months ago I exchanged mail with Russ about this topic regarding a different, much
larger binary. To avoid him having to redo the analysis, here is what he said at the
time:

====
i sent CL 13722046 to make the nm -S output a bit more useful.
for the toy binary i now get

  4a2280  1898528 D symtab
  26f3a0  1405936 D type.*
  671aa0  1058432 D pclntab
  3c6790   598056 D go.string.*
  4620c0    49600 D gcbss
  7a7c20    45496 B runtime.mheap
  46e280    21936 D gcdata
  7a29e0    21056 b bufferList
  1ed600    16480 T crypto/tls.(*Conn).clientHandshake
  79eb20    16064 b semtable
  1b3d90    14224 T net/http.init

that seems plausible to me. some notes:

symtab is the plan 9 symbol table. it in the binary but never referenced at run time. it
supports things like nm -S only. it needs to move into an unmapped section of the
binary, but it is only costing at most 8k at run time right now due to fragmentation and
it just wasn't worth the effort to try to move. the new linker will make this easier. of
course, moving it in the file doesn't shrink the file.

the thing named pclntab is a reencoding of the original pclntab and the parts of the
plan 9 symbol table that we did need at run time (mostly just a list of functions and
their names and addresses). as you can see, it is much smaller than the old form (the
symbol table dominates).

type.* is the reflect types and go.string.* is the static go string data. the *
indicates that i coalesced many symbols into one, to avoid useless individual names
bloating the symbol table. if we tried we could probably cut the reflect types by 2-4x.
it would mean packing the data a bit more compactly than an ordinary go data structure
would and then using unsafe to get it back out.

gcbss and gcdata are garbage collection bits for the bss and data segments. that's what
atom symbol did, and it's not clear whether it will last (probably not) and whether what
will replace it will be smaller. time will tell. i have a meeting with dmitriy, carl,
and keith next week to figure out what the plan is.

runtime.mheap, bufferList, and semtable are bss.

you're not seeing the gdb dwarf debug information here, because it's not a runtime
symbol.

g% otool -l $(which toy) | egrep '^  segname|filesize'
  segname __PAGEZERO
 filesize 0
  segname __TEXT
 filesize 7811072
  segname __DATA
 filesize 126560
  segname __LINKEDIT
 filesize 921772
  segname __DWARF
 filesize 2886943
g% 

there's another 3 MB. you can build with -ldflags -w to get rid of that at least.
if you read the full otool -l output you will find

Load command 6
     cmd LC_SYMTAB
 cmdsize 24
  symoff 10825728
   nsyms 22559
  stroff 11186924
 strsize 560576

looks like another 1 MB or so (560576+11186924-10825728 or 22559*16+560576) for the
mach-o symbol table.

when we do the new linker we can make recording this kind of information in a useful
form a priority.

robpike commented 10 years ago

Comment 1:

Note: the binaries were build on amd64 10.7.5 (Lion), with  gcc -version
i686-apple-darwin11-llvm-gcc-4.2
For the record, I couldn't do this experiment on 10.9 with Xcode 5 because the older
releases wouldn't build due to gcc/clang skew.

robpike commented 10 years ago

Comment 2:

Labels changed: added priority-later, removed priority-triage.

gopherbot commented 10 years ago

Comment 3 by jlourenco27:

Just for added reference, this are the size on go 1.1.2 vs 1.2 with OS X 10.9 and Xcode
5 (darwin gcc llvm 5.0 x86_64):
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2

randall77 commented 10 years ago

Comment 4:

This issue was updated by revision f238049a0073538caecfad1c60238a271426f43.

R=golang-dev, rsc
CC=golang-dev
https://golang.org/cl/35940047

rsc commented 10 years ago

Comment 5:

Labels changed: added release-go1.3.

rsc commented 10 years ago

Comment 6:

Labels changed: removed go1.3.

rsc commented 10 years ago

Comment 7:

Labels changed: added repo-main.

zephyr commented 10 years ago

Comment 8:

In this context, please reevaluate if the golang binaries can be changed to work with
UPX (ultimate packer for executables).¹²
For a small amount of computing power, upx can reduce the size of a binary down to a
quarter of its original size. You can authenticate this using a existing example
›fixer‹ programm for golang binaries on linux/amd64.³
While this approach doesn't fix the root of the problem – only the symptoms – it
would be nice to have this possibility always on hand.
For technical background of this problem (PT_LOAD[0].p_offset==0), please look at the
UPX bugtracker⁴.
¹ http://upx.sourceforge.net/
² https://en.wikipedia.org/wiki/UPX
³ https://github.com/pwaller/goupx
⁴ http://sourceforge.net/p/upx/bugs/195/

minux commented 10 years ago

Comment 9:

re #8, I don't think it's Go's problem. upx should be made more flexible to handle this.

leo-liu commented 10 years ago

Comment 10:

minux:
We may figure out why the binaries are growing fast first (lager runtime, optimization,
etc.), before we claim that it isn't Go's problem.

minux commented 10 years ago

Comment 11:

re #10, my #9 reply is to #8, which is about an entirely different problem.
i'm not saying that the ever-growing binaries is not our problem, only that i
don't believe that upx not accepting our binaries is our problem.
it's clear that upx isn't able to handle all possible and correct ELF files (i.e.
if the kernel can execute our binaries just fine, it's upx's problem to not be
able to compress them).

robpike commented 10 years ago

Comment 12:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 j      staff  1525984  2 Dez 21:44 hello_1.1.2
-rwxr-xr-x  1 j      staff  2192672  2 Dez 21:40 hello_1.2
-rwxr-xr-x  1 j      staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
1064960 94720   0   76000   1235680 12dae0  hello_1.1.2
1433600 147896  0   177440  1758936 1ad6d8  hello_1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

robpike commented 10 years ago

Comment 13:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

robpike commented 10 years ago

Comment 14:

More detail. The Plan 9 symbol table is about to be deleted. Here is a reference point,
adding one new entry to the list above:
$ ls -l *1.*
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
$ size *1.*
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096     0  4112    34566848     20f72c0    x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
1699840 160984  0   188944  2049768 1f46e8     x.1.2.x
Text has grown substantially, as has data. At least some of this is due to new
annotations for the garbage collector.

rsc commented 10 years ago

Comment 15:

This issue was updated by revision 964f6d3ec4c6e2bed377878bd2862767bfae463.

Nothing reads the Plan 9 symbol table anymore.
The last holdout was 'go tool nm', but since being rewritten in Go
it uses the standard symbol table for the binary format
(ELF, Mach-O, PE) instead.
Removing the Plan 9 symbol table saves ~15% disk space
on most binaries.
Two supporting changes included in this CL:
debug/gosym: use Go 1.2 pclntab to synthesize func-only
symbol table when there is no Plan 9 symbol table
debug/elf, debug/macho, debug/pe: ignore final EOF from ReadAt
LGTM=r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65740045

bradfitz commented 10 years ago

Comment 16:

After revision 737767dd81fd, I see a 25% reduction:
Before:
-rwxr-xr-x  1 bradfitz  staff  23556028 Feb 18 20:47 bin/camlistored
After:
-rwxr-xr-x  1 bradfitz  staff  17727420 Feb 18 20:48 bin/camlistored

robpike commented 10 years ago

Comment 17:

For my test case before/after deleting the Plan 9 symbol table:
% ls -l ...
-rwxr-xr-x  1 r  staff  2474512 Feb 18 20:27 hello_1.2.x
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
% size ...
__TEXT  __DATA  __OBJC  others  dec hex
1699840 160984  0   188944  2049768 1f46e8     hello_1.2.x
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.x
% 
So deleting the Plan 9 symbol table pretty close to exactly compensates for the GC
information. We're back at Go 1.2 levels, still far too large but it's a start.

rsc commented 10 years ago

Comment 18:

This issue was updated by revision 2541cc81978dc5e41e2e2db6345d8ca7a365ca8.

Every function now has a gcargs and gclocals symbol
holding associated garbage collection information.
Put them all in the same meta-symbol as the go.func data
and then drop individual entries from symbol table.
Removing gcargs and gclocals reduces the size of a
typical binary by 10%.
LGTM=r
R=r
CC=golang-codereviews
https://golang.org/cl/65870044

rsc commented 10 years ago

Comment 19:

This issue was updated by revision ae38b03f6cab6a25f9d8d34a39e33db9857dce2.

For an ephemeral binary - one created, run, and then deleted -
there is no need to write dwarf debug information, since the
binary will not be used with gdb. In this case, instruct the linker
not to spend time and disk space generating the debug information
by passing the -w flag to the linker.
Omitting dwarf information reduces the size of most binaries by 25%.
We may be more aggressive about this in the future.
LGTM=bradfitz, r
R=r, bradfitz
CC=golang-codereviews
https://golang.org/cl/65890043

robpike commented 10 years ago

Comment 20:

After removing gcargs from the symbol table (stepping across CL 65870044)
% ls -l x.1.2.[yz]
-rwxr-xr-x  1 r  staff  2150928 Feb 18 22:28 hello_1.2.y
-rwxr-xr-x  1 r  staff  1932880 Feb 19 08:14 hello_1.2.z
% size x.1.2.[yz] 
__TEXT  __DATA  __OBJC  others  dec hex
1376256 160984  0   188944  1726184 1a56e8    hello_1.2.y
1376256 160984  0   110160  1647400 192328 hello_1.2.z
% 
It's now smaller than at 1.2 but still much bigger than 1.1, let alone 1.0.

gopherbot commented 10 years ago

Comment 21:

I would like to take a look at compressing pclntab. Is it compressible?

ianlancetaylor commented 10 years ago

Comment 22:

The pclntab data should be reasonably compact, and note that fast access to the data is
important, since it is used for runtime.Callers and friends.  If you can find a
significant reduction in size that would be great, but small tweaks are probably not
desirable at this point.

gopherbot commented 10 years ago

Comment 23 by fuzxxl:

Is there any documentation for what the pclntab contains and for what constraints its
data structure must fullfill?

ianlancetaylor commented 10 years ago

Comment 24:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.

gopherbot commented 10 years ago

Comment 25 by allard.guy.m:

From the peanut gallery, AFAICT this breaks pprof interactive command 'list'.  At tip I
get, e.g.:
(pprof) list runner
Total: 6424 samples
objdump: syminit: Success
no filename found in main.runner<400c40>
Which works as expected with 1.2.1.

rsc commented 10 years ago

Comment 26:

Let's not use this issue as a discussion list.  Please ask questions on golang-dev. 
Thanks.
pprof not working is issue #7452.

Labels changed: added restrict-addissuecomment-commit.

rsc commented 10 years ago

Comment 27:

This is as fixed as it is going to be for Go 1.3.
Right now at tip + CL 80370045 on darwin/amd64, compiling this program:
package main
import "fmt"
func main() {
    fmt.Println("hello, world")
}
I get 1830352 bytes for the binary. Assuming this is the same case for which Rob's
numbers are reported, by this metric Go 1.3 will roll back more than half the size
increase caused by Go 1.2 (relative to Go 1.1). Will leave further improvement for Go
1.4.

Labels changed: added release-go1.4, removed release-go1.3.

rsc commented 10 years ago

Comment 28:

This issue was updated by revision a26c01ad446d2853f0c6a7ddaacadb02efa00b7.

LGTM=khr
R=khr
CC=golang-codereviews
https://golang.org/cl/80370045

rsc commented 9 years ago

Comment 29:

Labels changed: added release-go1.5, removed release-go1.4.

josharian commented 9 years ago

From Go 1.4 to Go tip on darwin/amd64, helloworld grows from 1927792 to 2344944 bytes (+22%).

Camlistore shows more modest increases, but they are non-trivial nevertheless:

File	Go 1.4	Go tip	Delta
camdeploy	12750156	13512052	+5.97%
camget	12378300	13261956	+7.13%
camlistored	28374844	29469892	+3.85%
cammount	13586156	14496036	+6.69%
camput	14040524	14954180	+6.50%
camtool	23245596	24260084	+4.36%
devcam	19250156	20153892	+4.69%
hello	12925420	13816564	+6.89%
publisher	19047228	19964308	+4.81%

josharian commented 9 years ago

One more data point. A program containing just a single non-linker-strippable string of length 500000 grows from 1.12mb in Go 1.4 to 1.56mb at tip (+39%). I tried bisecting and found (unfortunately) that the growth was pretty slow and steady: I saw file sizes along the way including at least 1.15, 1.25, 1.30, 1.34, and 1.47, and I wasn't looking hard.

package main
var s = "....very long...."
func main() { _ = s[0] }

robpike commented 9 years ago

Marked this as 1.6. I don't expect it to be fixed but I want to see us develop an understanding of what's going on.