Open bradfitz opened 6 years ago
Change https://golang.org/cl/127655 mentions this issue: net: lazily look up the listenerBacklog value on first use
Change https://golang.org/cl/127660 mentions this issue: encoding/gob: lazily initialize registerBasics
Change https://golang.org/cl/127661 mentions this issue: strings: select Replacer algorithm and build machine lazily
Change https://golang.org/cl/127664 mentions this issue: http2: reduce init-time work & allocations
Change https://golang.org/cl/127735 mentions this issue: hash/crc64: lazily initialize slice8Tables
Change https://golang.org/cl/127736 mentions this issue: net/mail: lazily initialize dateLayouts
Wow, that was picked up real fast by the others!
Change https://golang.org/cl/127875 mentions this issue: go/doc: compile regexps lazily
See also my last comment in #2559 about using go generate for some of this.
Change https://golang.org/cl/119715 mentions this issue: encoding/json: change reflect.Type usage to be more idiomatic
Using all.go that Brad posted in the OP, and using go version devel +aa311fecda Tue Aug 21 22:11:05 2018 +0000 darwin/amd64
, here are the current results:
$ gotip build all.go
$ GODEBUG=memprofilerate=1 ./all
278848
$ gotip tool pprof /tmp/all.mem.prof
Type: inuse_space
Time: Aug 21, 2018 at 3:15pm (PDT)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 50
Showing nodes accounting for 228.01kB, 87.93% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
flat flat% sum% cum cum%
37.03kB 14.28% 14.28% 37.03kB 14.28% runtime.procresize
21.39kB 8.25% 22.53% 21.39kB 8.25% vendor/golang_org/x/text/unicode/norm.init
18.95kB 7.31% 29.84% 18.97kB 7.31% encoding/xml.init
12.17kB 4.69% 34.53% 12.17kB 4.69% unicode.init
10.81kB 4.17% 38.70% 10.81kB 4.17% vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
10.58kB 4.08% 42.78% 12.82kB 4.94% sync.(*Map).LoadOrStore
9.17kB 3.54% 46.32% 9.22kB 3.55% html/template.init
9kB 3.47% 49.79% 9kB 3.47% runtime.malg
8.70kB 3.36% 53.14% 56.52kB 21.79% net/http.init
7kB 2.70% 55.84% 14kB 5.40% runtime.allocm
6.03kB 2.33% 58.17% 6.03kB 2.33% syscall.copyenv
5.25kB 2.02% 60.19% 5.25kB 2.02% math/rand.NewSource
4.78kB 1.84% 62.04% 5.19kB 2.00% net/http.init.0
4.03kB 1.55% 63.59% 4.22kB 1.63% net/http.init.2
3.94kB 1.52% 65.11% 3.95kB 1.52% debug/dwarf.init
3.81kB 1.47% 66.58% 3.81kB 1.47% errors.New
3.62kB 1.40% 67.98% 12.34kB 4.76% encoding/gob.validUserType
3.62kB 1.40% 69.38% 3.62kB 1.40% regexp/syntax.init
2.91kB 1.12% 70.50% 2.91kB 1.12% go/types.(*Scope).Insert
2.70kB 1.04% 71.54% 27.16kB 10.47% encoding/gob.init
2.69kB 1.04% 72.58% 2.69kB 1.04% flag.(*FlagSet).Var
2.59kB 1.00% 73.58% 2.59kB 1.00% net/mail.init.0
2.25kB 0.87% 74.44% 2.25kB 0.87% compress/flate.(*huffmanEncoder).generate
2.25kB 0.87% 75.31% 2.25kB 0.87% net/textproto.init.0
2.09kB 0.81% 76.12% 2.09kB 0.81% image/jpeg.(*huffmanLUT).init
1.89kB 0.73% 76.85% 1.89kB 0.73% compress/flate.newHuffmanEncoder
1.80kB 0.69% 77.54% 1.80kB 0.69% go/types.NewTypeName
1.77kB 0.68% 78.22% 1.77kB 0.68% text/template/parse.(*Tree).newText
1.75kB 0.67% 78.90% 4.38kB 1.69% runtime.mcommoninit
1.66kB 0.64% 79.53% 8.85kB 3.41% go/types.init
1.64kB 0.63% 80.17% 10.17kB 3.92% go/doc.init
1.59kB 0.61% 80.78% 3.74kB 1.44% encoding/gob.bootstrapType
1.56kB 0.6% 81.38% 1.56kB 0.6% regexp/syntax.(*compiler).inst (inline)
1.55kB 0.6% 81.98% 4.02kB 1.55% net.init
1.41kB 0.54% 82.52% 1.41kB 0.54% go/types.newBuiltin (inline)
1.39kB 0.54% 83.06% 13.84kB 5.34% go/build.init
1.38kB 0.53% 83.59% 1.38kB 0.53% text/template.addValueFuncs
1.31kB 0.51% 84.10% 1.31kB 0.51% text/template/parse.(*Tree).newPipeline
1.17kB 0.45% 84.55% 3.09kB 1.19% text/template.init
1.16kB 0.45% 85.00% 3.74kB 1.44% encoding/gob.newTypeObject
1.06kB 0.41% 85.41% 14.55kB 5.61% encoding/gob.RegisterName
1.02kB 0.39% 85.80% 4.76kB 1.83% encoding/gob.buildTypeInfo
0.75kB 0.29% 86.09% 1.84kB 0.71% html/template.New
0.75kB 0.29% 86.38% 7.88kB 3.04% runtime.systemstack
0.73kB 0.28% 86.66% 16.71kB 6.44% encoding/gob.init.1
0.69kB 0.27% 86.92% 2.13kB 0.82% reflect.(*rtype).ptrTo
0.67kB 0.26% 87.18% 6.92kB 2.67% crypto/tls.init
0.67kB 0.26% 87.44% 22.42kB 8.65% vendor/golang_org/x/net/http/httpguts.init
0.64kB 0.25% 87.69% 1.42kB 0.55% net/http/cgi.init
0.61kB 0.23% 87.93% 20.99kB 8.10% archive/tar.init
(pprof) top --cum 50
Showing nodes accounting for 160.47kB, 61.88% of 259.32kB total
Dropped 218 nodes (cum <= 1.30kB)
Showing top 50 nodes out of 139
flat flat% sum% cum cum%
0.09kB 0.036% 0.036% 205.41kB 79.21% runtime.main
0 0% 0.036% 203.23kB 78.37% main.init
0 0% 0.036% 57.42kB 22.14% expvar.init
8.70kB 3.36% 3.39% 56.52kB 21.79% net/http.init
0 0% 3.39% 37.41kB 14.42% runtime.rt0_go
37.03kB 14.28% 17.67% 37.03kB 14.28% runtime.procresize
0 0% 17.67% 37.03kB 14.28% runtime.schedinit
2.70kB 1.04% 18.71% 27.16kB 10.47% encoding/gob.init
0.67kB 0.26% 18.97% 22.42kB 8.65% vendor/golang_org/x/net/http/httpguts.init
0 0% 18.97% 21.75kB 8.39% vendor/golang_org/x/net/idna.init
21.39kB 8.25% 27.22% 21.39kB 8.25% vendor/golang_org/x/text/unicode/norm.init
0.61kB 0.23% 27.46% 20.99kB 8.10% archive/tar.init
0 0% 27.46% 20.20kB 7.79% fmt.init
18.95kB 7.31% 34.77% 18.97kB 7.31% encoding/xml.init
0.73kB 0.28% 35.05% 16.71kB 6.44% encoding/gob.init.1
0 0% 35.05% 14.55kB 5.61% encoding/gob.Register
1.06kB 0.41% 35.46% 14.55kB 5.61% encoding/gob.RegisterName
0 0% 35.46% 14.55kB 5.61% encoding/gob.registerBasics
7kB 2.70% 38.16% 14kB 5.40% runtime.allocm
1.39kB 0.54% 38.69% 13.84kB 5.34% go/build.init
10.58kB 4.08% 42.77% 12.82kB 4.94% sync.(*Map).LoadOrStore
0 0% 42.77% 12.34kB 4.76% encoding/gob.userType
3.62kB 1.40% 44.17% 12.34kB 4.76% encoding/gob.validUserType
0 0% 44.17% 12.17kB 4.69% reflect.init
12.17kB 4.69% 48.87% 12.17kB 4.69% unicode.init
0 0% 48.87% 12kB 4.63% runtime.newm
0 0% 48.87% 11.02kB 4.25% vendor/golang_org/x/net/http2/hpack.init
0.05kB 0.018% 48.88% 10.95kB 4.22% vendor/golang_org/x/net/http2/hpack.newStaticTable
10.81kB 4.17% 53.05% 10.81kB 4.17% vendor/golang_org/x/net/http2/hpack.(*headerFieldTable).addEntry (inline)
0 0% 53.05% 10.25kB 3.95% runtime.mstart
1.64kB 0.63% 53.69% 10.17kB 3.92% go/doc.init
9.17kB 3.54% 57.22% 9.22kB 3.55% html/template.init
0 0% 57.22% 9.07kB 3.50% go/importer.init
0 0% 57.22% 9.07kB 3.50% go/internal/gccgoimporter.init
9kB 3.47% 60.69% 9kB 3.47% runtime.malg
1.66kB 0.64% 61.33% 8.85kB 3.41% go/types.init
0 0% 61.33% 8kB 3.08% runtime.startm
0.75kB 0.29% 61.62% 7.88kB 3.04% runtime.systemstack
0 0% 61.62% 7.82kB 3.02% os.init
0 0% 61.62% 7.80kB 3.01% html/template.(*Template).Parse
0 0% 61.62% 7.80kB 3.01% text/template.(*Template).Parse
0 0% 61.62% 7.39kB 2.85% text/template/parse.Parse
0 0% 61.62% 7.04kB 2.71% go/types.init.0
0.67kB 0.26% 61.88% 6.92kB 2.67% crypto/tls.init
0 0% 61.88% 6.89kB 2.66% text/template/parse.(*Tree).Parse
0 0% 61.88% 6.89kB 2.66% text/template/parse.(*Tree).parse
0 0% 61.88% 6.52kB 2.51% text/template/parse.(*Tree).textOrAction
0 0% 61.88% 6.16kB 2.37% crypto/x509.init
0 0% 61.88% 6.05kB 2.33% os.Getwd
0 0% 61.88% 6.03kB 2.33% os.Getenv
Those top two vendor/golang_org/x
ones fixed in x/ but need to be imported into std. That'll happen soon enough.
Change https://golang.org/cl/166459 mentions this issue: cmd/go: further reduce init work
Change https://golang.org/cl/170317 mentions this issue: net/textproto: simplify commonHeader initialization
Change https://golang.org/cl/210284 mentions this issue: text/template: avoid a global map to help the linker's deadcode elimination
Isn't this resolved?
❯ go version
go version go1.17.2 darwin/amd64
❯ go build all.go
❯ GODEBUG=memprofilerate=1 ./all
289408
❯ go tool pprof /tmp/all.mem.prof
Type: inuse_space
Time: Dec 5, 2021 at 12:35pm (CET)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top 50
Showing nodes accounting for 9751.47kB, 100% of 9751.47kB total
flat flat% sum% cum cum%
3585.42kB 36.77% 36.77% 3585.42kB 36.77% runtime.malg
3092.09kB 31.71% 68.48% 3092.09kB 31.71% runtime.procresize
1537.69kB 15.77% 84.25% 4610.75kB 47.28% runtime.allocm
1024.25kB 10.50% 94.75% 2048.66kB 21.01% runtime.mcommoninit
512.02kB 5.25% 100% 512.02kB 5.25% text/template/parse.(*Tree).newList (inline)
0 0% 100% 512.02kB 5.25% html/template.(*Template).Parse
0 0% 100% 512.02kB 5.25% net/rpc.init
0 0% 100% 512.02kB 5.25% runtime.doInit
0 0% 100% 512.02kB 5.25% runtime.main
0 0% 100% 2049.09kB 21.01% runtime.main.func1
0 0% 100% 1024.41kB 10.51% runtime.mpreinit
0 0% 100% 3073.86kB 31.52% runtime.mstart
0 0% 100% 3073.86kB 31.52% runtime.mstart0
0 0% 100% 3073.86kB 31.52% runtime.mstart1
0 0% 100% 2561.30kB 26.27% runtime.mstartm0
0 0% 100% 2561.30kB 26.27% runtime.newextram
0 0% 100% 2561.66kB 26.27% runtime.newm
0 0% 100% 512.20kB 5.25% runtime.newproc
0 0% 100% 1024.41kB 10.51% runtime.newproc.func1
0 0% 100% 1024.41kB 10.51% runtime.newproc1
0 0% 100% 2561.30kB 26.27% runtime.oneNewExtraM
0 0% 100% 512.56kB 5.26% runtime.resetspinning
0 0% 100% 3604.29kB 36.96% runtime.rt0_go
0 0% 100% 3092.09kB 31.71% runtime.schedinit
0 0% 100% 512.56kB 5.26% runtime.schedule
0 0% 100% 512.56kB 5.26% runtime.startm
0 0% 100% 2561.30kB 26.27% runtime.systemstack
0 0% 100% 512.56kB 5.26% runtime.wakep
0 0% 100% 512.02kB 5.25% text/template.(*Template).Parse
0 0% 100% 512.02kB 5.25% text/template/parse.(*Tree).Parse
0 0% 100% 512.02kB 5.25% text/template/parse.(*Tree).action
0 0% 100% 512.02kB 5.25% text/template/parse.(*Tree).itemList
0 0% 100% 512.02kB 5.25% text/template/parse.(*Tree).parse
0 0% 100% 512.02kB 5.25% text/template/parse.(*Tree).parseControl
0 0% 100% 512.02kB 5.25% text/template/parse.(*Tree).rangeControl
0 0% 100% 512.02kB 5.25% text/template/parse.(*Tree).textOrAction
0 0% 100% 512.02kB 5.25% text/template/parse.Parse
(pprof)
Change https://go.dev/cl/460543 mentions this issue: encoding/gob: shave off some init time cost
@cristaloleg it is not resolved; I've found pprof to not show useful numbers for init funcs. GODEBUG=inittrace=1
is a better way to see. See the CL above, which shows those numbers.
Change https://go.dev/cl/460544 mentions this issue: go/types: use internal/lazyregexp for goVersionRx
Change https://go.dev/cl/460545 mentions this issue: internal/profile: use internal/lazyregexp for the legacy parser
Change https://go.dev/cl/455455 mentions this issue: context: reduce init-time allocations
Tracking bug to reduce init-time CPU & memory usage.
Previously:
https://go-review.googlesource.com/c/go/+/127075 - html: lazily populate Unescape tables https://go-review.googlesource.com/c/net/+/127275 - http2/hpack: lazily build huffman table on first use
Open:
26752 - x/text/unicode/norm: reduce init-time memory usage
The program https://play.golang.org/p/9ervXCWzV_z is useful to find offenders: