abcxyz / github-metrics-aggregator

Apache License 2.0
11 stars 3 forks source link

Speed up build times and encapsulate github auth #182

Closed sethvargo closed 7 months ago

sethvargo commented 7 months ago

I wanted to try and understand why the build times for GMA are so much slower than our other projects. TL;DR, it's Beam 😦. I think I've managed to get things a little better in this PR.

Before ```text 176.719s (root) 85.287s github.com 26.653s github.com/apache 10.744s github.com/goccy 7.797s github.com/lestrrat-go 6.892s github.com/abcxyz 5.233s github.com/google 4.768s github.com/docker 4.511s github.com/ProtonMail 3.648s github.com/googleapis 3.578s github.com/cloudflare 3.374s github.com/klauspost 1.465s github.com/andybalholm 1.159s github.com/opencontainers 1.054s github.com/posener 0.885s github.com/golang 0.844s github.com/pierrec 0.795s github.com/gogo 0.760s github.com/shurcooL 0.404s github.com/sethvargo 0.351s github.com/dustin 0.147s github.com/zeebo 0.070s github.com/pkg 0.062s github.com/kr 0.049s github.com/distribution 0.043s github.com/mattn 44.629s std 8.996s 0.171s std/crypto 4.882s std/vendor 4.714s 0.916s std/runtime 3.754s 0.185s std/encoding 3.507s 1.604s std/net 3.120s std/internal 2.302s 0.889s std/mime 1.946s std/debug 1.304s 0.430s std/log 1.016s std/text 0.937s std/compress 0.849s 0.204s std/math 0.807s 0.084s std/hash 0.733s 0.066s std/html 0.637s 0.214s std/unicode 0.633s 0.111s std/os 0.492s 0.142s std/image 0.377s 0.377s std/reflect 0.328s 0.043s std/sync 0.313s 0.313s std/testing 0.309s 0.309s std/syscall 0.309s 0.089s std/regexp 0.262s std/archive 0.210s 0.210s std/time 0.203s 0.057s std/io 0.194s std/container 0.139s std/database 0.138s std/go 0.138s 0.067s std/path 0.135s 0.135s std/flag 0.118s 0.118s std/fmt 0.116s 0.116s std/cmp 0.111s 0.111s std/bytes 0.111s 0.111s std/strings 0.109s 0.109s std/context 0.108s 0.108s std/strconv 0.107s 0.107s std/slices 0.071s 0.071s std/bufio 0.038s 0.038s std/sort 0.036s 0.036s std/embed 0.022s 0.022s std/errors 19.202s google.golang.org 9.144s 0.419s google.golang.org/grpc 6.112s google.golang.org/protobuf 3.158s google.golang.org/api 0.788s google.golang.org/genproto 12.482s golang.org 12.482s golang.org/x 7.388s 0.054s go.opencensus.io 2.619s 0.108s go.opencensus.io/trace 1.528s 1.396s go.opencensus.io/internal 1.155s 0.197s go.opencensus.io/stats 1.092s go.opencensus.io/metric 0.533s go.opencensus.io/plugin 0.273s 0.273s go.opencensus.io/resource 0.135s 0.135s go.opencensus.io/tag 7.301s cloud.google.com 7.301s cloud.google.com/go 0.431s gopkg.in 0.372s 0.372s gopkg.in/yaml.v3 0.059s 0.059s gopkg.in/retry.v1 ```
After ```text 101.417s (root) 49.762s github.com 16.029s github.com/apache 9.471s github.com/goccy 5.694s github.com/abcxyz 4.170s github.com/google 3.573s github.com/lestrrat-go 2.024s github.com/docker 1.680s github.com/klauspost 1.228s github.com/andybalholm 0.977s github.com/googleapis 0.807s github.com/golang 0.693s github.com/gogo 0.581s github.com/opencontainers 0.564s github.com/pierrec 0.538s github.com/shurcooL 0.437s github.com/sethvargo 0.430s github.com/posener 0.228s github.com/mattn 0.195s github.com/pkg 0.142s github.com/dustin 0.136s github.com/distribution 0.107s github.com/zeebo 0.059s github.com/kr 22.032s std 3.514s 1.695s std/net 3.387s 0.912s std/runtime 3.319s 0.066s std/crypto 1.761s std/vendor 1.482s std/internal 1.310s 0.023s std/encoding 0.757s 0.164s std/math 0.630s 0.287s std/log 0.576s std/text 0.566s std/debug 0.441s 0.110s std/os 0.364s 0.364s std/reflect 0.344s std/archive 0.307s 0.307s std/syscall 0.284s 0.022s std/hash 0.266s 0.043s std/html 0.260s std/compress 0.214s 0.106s std/regexp 0.186s 0.144s std/unicode 0.186s 0.123s std/path 0.184s 0.184s std/fmt 0.178s 0.072s std/mime 0.175s 0.175s std/testing 0.165s 0.055s std/io 0.150s 0.150s std/time 0.122s 0.122s std/bytes 0.120s 0.120s std/flag 0.116s 0.045s std/sync 0.104s 0.104s std/strconv 0.099s std/database 0.078s 0.078s std/strings 0.068s 0.068s std/context 0.061s std/container 0.059s 0.059s std/embed 0.050s 0.050s std/bufio 0.048s std/go 0.035s 0.035s std/sort 0.024s 0.024s std/slices 0.024s 0.024s std/errors 0.018s 0.018s std/cmp 14.957s google.golang.org 6.388s 0.550s google.golang.org/grpc 4.065s google.golang.org/protobuf 3.730s google.golang.org/api 0.775s google.golang.org/genproto 7.002s golang.org 7.002s golang.org/x 5.458s cloud.google.com 5.458s cloud.google.com/go 1.641s 0.021s go.opencensus.io 0.760s go.opencensus.io/plugin 0.383s 0.034s go.opencensus.io/stats 0.220s 0.098s go.opencensus.io/trace 0.090s 0.090s go.opencensus.io/tag 0.062s go.opencensus.io/metric 0.058s 0.036s go.opencensus.io/internal 0.047s 0.047s go.opencensus.io/resource 0.565s gopkg.in 0.405s 0.405s gopkg.in/yaml.v3 0.160s 0.160s gopkg.in/retry.v1 ```
pdewilde commented 7 months ago

Just a general question about why the init() causes such a build time difference:

Is it because beam already has a lot of init() and its non linear to add another in the ordering decision?

sethvargo commented 7 months ago

Just a general question about why the init() causes such a build time difference:

Is it because beam already has a lot of init() and its non linear to add another in the ordering decision?

I think there are a few things going against us:

sethvargo commented 7 months ago

/cc @capri-xiyue for Go readability.