robpike commented 10 years ago

As an experiment, I build "hello, world" at the release points for go 1.0.
1.1, and 1.2. Here are the binary's sizes:

% ls -l x.1.?
-rwxr-xr-x  1 r  staff  1191952 Nov 30 10:25 x.1.0
-rwxr-xr-x  1 r  staff  1525936 Nov 30 10:20 x.1.1
-rwxr-xr-x  1 r  staff  2188576 Nov 30 10:18 x.1.2
% size x.1.?
__TEXT  __DATA  __OBJC  others  dec hex
880640  33682096    0   4112    34566848    20f72c0 x.1.0
1064960 94656   0   75952   1235568 12da70  x.1.1
1429504 147896  0   177440  1754840 1ac6d8  x.1.2
% 

A near-doubling of the binary size in two releases is a bug of a kind. I will hold on to
the files so they can be analyzed more, but am filing this issue to get the topic
registered. We need to develop a better understanding of the problem and how to address
it.

Marking this 1.3 (not maybe) because I consider it a priority.

A few months ago I exchanged mail with Russ about this topic regarding a different, much
larger binary. To avoid him having to redo the analysis, here is what he said at the
time:

====
i sent CL 13722046 to make the nm -S output a bit more useful.
for the toy binary i now get

  4a2280  1898528 D symtab
  26f3a0  1405936 D type.*
  671aa0  1058432 D pclntab
  3c6790   598056 D go.string.*
  4620c0    49600 D gcbss
  7a7c20    45496 B runtime.mheap
  46e280    21936 D gcdata
  7a29e0    21056 b bufferList
  1ed600    16480 T crypto/tls.(*Conn).clientHandshake
  79eb20    16064 b semtable
  1b3d90    14224 T net/http.init

that seems plausible to me. some notes:

symtab is the plan 9 symbol table. it in the binary but never referenced at run time. it
supports things like nm -S only. it needs to move into an unmapped section of the
binary, but it is only costing at most 8k at run time right now due to fragmentation and
it just wasn't worth the effort to try to move. the new linker will make this easier. of
course, moving it in the file doesn't shrink the file.

the thing named pclntab is a reencoding of the original pclntab and the parts of the
plan 9 symbol table that we did need at run time (mostly just a list of functions and
their names and addresses). as you can see, it is much smaller than the old form (the
symbol table dominates).

type.* is the reflect types and go.string.* is the static go string data. the *
indicates that i coalesced many symbols into one, to avoid useless individual names
bloating the symbol table. if we tried we could probably cut the reflect types by 2-4x.
it would mean packing the data a bit more compactly than an ordinary go data structure
would and then using unsafe to get it back out.

gcbss and gcdata are garbage collection bits for the bss and data segments. that's what
atom symbol did, and it's not clear whether it will last (probably not) and whether what
will replace it will be smaller. time will tell. i have a meeting with dmitriy, carl,
and keith next week to figure out what the plan is.

runtime.mheap, bufferList, and semtable are bss.

you're not seeing the gdb dwarf debug information here, because it's not a runtime
symbol.

g% otool -l $(which toy) | egrep '^  segname|filesize'
  segname __PAGEZERO
 filesize 0
  segname __TEXT
 filesize 7811072
  segname __DATA
 filesize 126560
  segname __LINKEDIT
 filesize 921772
  segname __DWARF
 filesize 2886943
g% 

there's another 3 MB. you can build with -ldflags -w to get rid of that at least.
if you read the full otool -l output you will find

Load command 6
     cmd LC_SYMTAB
 cmdsize 24
  symoff 10825728
   nsyms 22559
  stroff 11186924
 strsize 560576

looks like another 1 MB or so (560576+11186924-10825728 or 22559*16+560576) for the
mach-o symbol table.

when we do the new linker we can make recording this kind of information in a useful
form a priority.

gopherbot commented 8 years ago

CL https://golang.org/cl/20334 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/20335 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/20483 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/20701 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/20709 mentions this issue.

bradfitz commented 8 years ago

Progress update.

David Crawshaw's and others work has been paying off...

$ for x in go1.4 go1.5 go1.6 go; do ls -l $x/pkg/tool/linux_amd64/objdump; done
-rwxr-xr-x 1 bradfitz bradfitz 4408008 Feb 18  2015 go1.4/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4885536 Nov 20 19:33 go1.5/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4988208 Feb 17 20:37 go1.6/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3582128 Mar 16 20:01 go/pkg/tool/linux_amd64/objdump

$ for x in go1.4 go1.5 go1.6 go; do ls -l $x/pkg/tool/linux_amd64/cgo; done
-rwxr-xr-x 1 bradfitz bradfitz 4560376 Feb 18  2015 go1.4/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5111632 Nov 20 19:32 go1.5/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5275056 Feb 17 20:36 go1.6/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4161520 Mar 16 20:00 go/pkg/tool/linux_amd64/cgo

$ for x in go1.4 go1.5 go1.6 go; do ls -l $x/bin/go; done
-rwxr-xr-x 1 bradfitz bradfitz 9571864 Feb 18  2015 go1.4/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 11195936 Nov 20 19:33 go1.5/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 12523312 Feb 17 20:37 go1.6/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 9972464 Mar 16 20:08 go/bin/go

$ cat fmt_hello.go 
package main

import "fmt"

func main() {
        fmt.Println("Hello, world.")
}
$ ls -l fmt_hello_*
-rwxr-xr-x 1 bradfitz bradfitz 1941352 Mar 17 20:08 fmt_hello_14
-rwxr-xr-x 1 bradfitz bradfitz 2367120 Mar 17 20:08 fmt_hello_15
-rwxr-xr-x 1 bradfitz bradfitz 2288392 Mar 17 20:08 fmt_hello_16
-rwxr-xr-x 1 bradfitz bradfitz 1620472 Mar 17 20:08 fmt_hello_tip

robpike commented 8 years ago

You really should test a single program. I believe all of those except maybe objdump have changed significantly during the interval. My test with a more guaranteed stable source is seeing about 6% reduction from 1.4, which is good but still far from where it should be. Go back and read the first message to see how much bloat has arrived.

But progress is happening, that's for sure.

gopherbot commented 8 years ago

CL https://golang.org/cl/20825 mentions this issue.

crawshaw commented 8 years ago

@robpike objdump is shrinking more than most programs because the new dead code elimination in the linker can detect statically that it does not call methods via reflect. This lets it remove far more methods. For details see the CL description of https://golang.org/cl/20483

If you have a particular program you'd like me to look at for my next binary size pass, please send it to me.

bradfitz commented 8 years ago

@robpike, the bottom of my comment contains the 1-line hello world program. It doesn't get much simpler than that.

robpike commented 8 years ago

@bradfitz indeed, although it's not clear whether it's a typical Go program. Perhaps it is.

broady commented 8 years ago

Here's cmd/godoc at HEAD (fcde7743)

$ for x in go1.4 go1.5 go1.6 go; do $x/bin/go build -a -o ~/godoc_$x golang.org/x/tools/cmd/godoc; ls -l ~/godoc_$x; done
-rwxr-x--- 1 cbro eng 16267896 Mar 18 16:11 /usr/local/google/home/cbro/godoc_go1.4
-rwxr-x--- 1 cbro eng 17132840 Mar 18 16:11 /usr/local/google/home/cbro/godoc_go1.5
-rwxr-x--- 1 cbro eng 18468712 Mar 18 16:12 /usr/local/google/home/cbro/godoc_go1.6
-rwxr-x--- 1 cbro eng 15425920 Mar 18 16:12 /usr/local/google/home/cbro/godoc_go

gopherbot commented 8 years ago

CL https://golang.org/cl/20968 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21033 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21087 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/20902 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21285 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21284 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21395 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21396 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21583 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21777 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/21776 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/22371 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/22373 mentions this issue.

gopherbot commented 8 years ago

CL https://golang.org/cl/22395 mentions this issue.

rsc commented 8 years ago

Great progress this release cycle. Kicking open issue to next cycle because there's still a bit more to do. But at least the "and growing" has been reversed for one cycle.

quentinmit commented 8 years ago

@rsc It seems like we haven't done any work on this for 1.8. I don't think keeping an umbrella issue like this open is useful. Is there anything in particular you had in mind for this issue, or should we close it now?

crawshaw commented 8 years ago

I have some more typeOff work I was hoping to do in 1.8 (the continuation of the work linked here done in 1.7), though I am busy in other projects right now. Let's at least leave this open until the window closes in a few weeks.

rsc commented 8 years ago

We have very few umbrella issues, but this one is fine to keep. It's good to keep in mind and collect all the work toward this. The binaries are still much bigger than we hope they would be.

gopherbot commented 7 years ago

CL https://golang.org/cl/43090 mentions this issue.

gopherbot commented 7 years ago

CL https://golang.org/cl/43190 mentions this issue.

gopherbot commented 7 years ago

CL https://golang.org/cl/44007 mentions this issue.

bradfitz commented 7 years ago

No notable wins (maybe a slight loss) at tip (to-be Go 1.9) compared to Go 1.8:

bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/go; done
-rwxr-xr-x 1 bradfitz bradfitz 9571864 Feb 18  2015 go1.4/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 11290832 Jan 13  2016 go1.5/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 12534640 Jul 18  2016 go1.6/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 9953979 Aug 15  2016 go1.7/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10068917 Feb 16 19:28 go1.8/bin/go
-rwxr-xr-x 1 bradfitz bradfitz 10346229 Jun  6 00:03 go/bin/go
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/cgo; done
-rwxr-xr-x 1 bradfitz bradfitz 4560376 Feb 18  2015 go1.4/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5111280 Jan 13  2016 go1.5/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 5279368 Jul 18  2016 go1.6/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4114079 Aug 15  2016 go1.7/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 3914818 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/cgo
-rwxr-xr-x 1 bradfitz bradfitz 4015485 Jun  6 00:03 go/pkg/tool/linux_amd64/cgo
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/pkg/tool/linux_amd64/objdump; done
-rwxr-xr-x 1 bradfitz bradfitz 4408008 Feb 18  2015 go1.4/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4889240 Jan 13  2016 go1.5/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 4988648 Jul 18  2016 go1.6/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3622669 Aug 15  2016 go1.7/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3717826 Feb 16 19:28 go1.8/pkg/tool/linux_amd64/objdump
-rwxr-xr-x 1 bradfitz bradfitz 3836105 Jun  6 00:03 go/pkg/tool/linux_amd64/objdump
bradfitz@gdev:~$ for x in go1.4 go1.5 go1.6 go1.7 go1.8 go; do ls -l $x/bin/gofmt; done
-rwxr-xr-x 1 bradfitz bradfitz 3594744 May 11  2016 go1.4/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3944064 Jan 13  2016 go1.5/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3895320 Jul 18  2016 go1.6/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3036195 Aug 15  2016 go1.7/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3481554 Feb 16 19:28 go1.8/bin/gofmt
-rwxr-xr-x 1 bradfitz bradfitz 3257512 Jun  6 00:03 go/bin/gofmt

I'm going to move this ongoing tracking bug to Go 1.10, since I don't see anything more happening for Go 1.9.

gopherbot commented 7 years ago

Change https://golang.org/cl/57130 mentions this issue: cmd/compile/internal/ssa: combine consecutive loads and stores on amd64

mvdan commented 7 years ago

This issue reports slight increases between 1.8 and 1.9: #21653

gopherbot commented 7 years ago

Change https://golang.org/cl/61190 mentions this issue: cmd/compile: specialize map creation for small hint sizes

gopherbot commented 6 years ago

Change https://golang.org/cl/88135 mentions this issue: cmd/compile: don't combine 64-bit loads/stores on amd64

FiloSottile commented 6 years ago

23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with `-ldflags="-s -w"`. (#11799 is relevant to debug info size.)

ALTree commented 6 years ago

I'm seeing a significant regression in binaries sizes between go1.10.1 and the current tip on linux/amd64. A simple hello world is 26% bigger than it was on 1.10:

$ cat test.go 
package main

import "fmt"

func main() {
    fmt.Println("hi!")
}

$ go version
go version go1.10.1 linux/amd64
$ go build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2011612 Apr 27 11:14 test

$ gotip version
go version devel +a3bafcf8cc Thu Apr 26 18:26:06 2018 +0000 linux/amd64
$ gotip build test.go
$ ls -l test
-rwxr-xr-x 1 alberto alberto 2543972 Apr 27 11:15 test

It's not just the hello world. The go binary went from 11MB to 15MB. gofmt from 3.4MB to 4.3MB.

Is this expected?

mvdan commented 6 years ago

@ALTree have you tried the same tests with -ldflags="-w -s"? There have been many changes to debugging info recently, so perhaps it's just that the binaries contain more debug info.

dominikh commented 6 years ago

Generally speaking we do not recommend stripping debug info (it's useful, after all), so that's only useful for figuring out why they grew, not a justification for the growth.

ALTree commented 6 years ago

@mvdan Yes, and stripped binaries are the same size as in 1.10, but that's not the default...

But yeah, if this is an expected effect of the recent work on debug info then... ok, I guess? I just wanted to make sure this was intentional.

mvdan commented 6 years ago

I should have been explicit; I meant it only as a way to quickly figure out where the growth was coming from.

josharian commented 6 years ago

When I looked a few weeks ago, all of the increase (and some) was dwarf. I think the best fix is probably #11799.

gopherbot commented 6 years ago

Change https://golang.org/cl/118276 mentions this issue: cmd/link: compress DWARF sections in ELF binaries

gopherbot commented 6 years ago

Change https://golang.org/cl/127075 mentions this issue: html: lazily populate Unescape tables

katiehockman commented 5 years ago

related issue, specifically for wasm: #29478

bradfitz commented 5 years ago

Go 1.11 got bigger, and Go 1.12 got even bigger: https://github.com/golang/go/issues/27266

gopherbot commented 5 years ago

Change https://golang.org/cl/161337 mentions this issue: cmd/compile: reorganize init functions

golang / go

all: binaries too big and growing #6853

23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with `-ldflags="-s -w"`. (#11799 is relevant to debug info size.)

golang / go

all: binaries too big and growing #6853

23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with -ldflags="-s -w". (#11799 is relevant to debug info size.)

23934 reports an ~8% increase between 1.9.4 and 1.10 for default binaries, and a slight decrease with `-ldflags="-s -w"`. (#11799 is relevant to debug info size.)