search

nytimes / gizmo

A Microservice Toolkit from The New York Times
https://open.nytimes.com/introducing-gizmo-aa7ea463b208
Apache License 2.0
3.76k stars 225 forks source link

gizmo/server is bloated #188

Open marwan-at-work opened 5 years ago

marwan-at-work commented 5 years ago

If you are looking to build something simple, such as SimpleServer, but with just net/http and gorilla/mux you get the following stats: 

~ ls -lh main
7.1M
~ go list -m all
main
github.com/gorilla/mux v1.7.0

But if you build it with gizmo SimpleServer this is what you get: 

~ ls -lh main
14M
~ golist -m all
main
cloud.google.com/go v0.35.1
contrib.go.opencensus.io/exporter/ocagent v0.4.3
contrib.go.opencensus.io/exporter/stackdriver v0.9.1
dmitri.shuralyov.com/app/changes v0.0.0-20180602232624-0a106ad413e3
dmitri.shuralyov.com/html/belt v0.0.0-20180602232347-f7d459c86be0
dmitri.shuralyov.com/service/change v0.0.0-20181023043359-a85b471d5412
dmitri.shuralyov.com/state v0.0.0-20180228185332-28bcc343414c
git.apache.org/thrift.git v0.0.0-20181218151757-9b75e4fe745a
github.com/BurntSushi/toml v0.3.1
github.com/DataDog/zstd v1.3.5
github.com/NYTimes/gizmo v1.1.0
github.com/NYTimes/gziphandler v1.0.1
github.com/NYTimes/logrotate v0.0.0-20170824154650-2b6e866fd507
github.com/NYTimes/sqliface v0.0.0-20180310195202-f8e6c8b78d37
github.com/Shopify/sarama v1.20.1
github.com/Shopify/toxiproxy v2.1.4+incompatible
github.com/anmitsu/go-shlex v0.0.0-20161002113705-648efa622239
github.com/aws/aws-sdk-go v1.16.26
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973
github.com/bradfitz/go-smtpd v0.0.0-20170404230938-deb6d6237625
github.com/bradfitz/gomemcache v0.0.0-20180710155616-bc664df96737
github.com/census-instrumentation/opencensus-proto v0.1.0
github.com/client9/misspell v0.3.4
github.com/coreos/go-systemd v0.0.0-20181012123002-c6f51f82210d
github.com/davecgh/go-spew v1.1.1
github.com/dustin/go-humanize v1.0.0
github.com/eapache/go-resiliency v1.1.0
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21
github.com/eapache/queue v1.1.0
github.com/flynn/go-shlex v0.0.0-20150515145356-3f9db97f8568
github.com/fsnotify/fsnotify v1.4.7
github.com/ghodss/yaml v1.0.0
github.com/gliderlabs/ssh v0.1.1
github.com/go-kit/kit v0.8.0
github.com/go-logfmt/logfmt v0.4.0
github.com/go-sql-driver/mysql v1.4.1
github.com/go-stack/stack v1.8.0
github.com/gogo/protobuf v1.1.1
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b
github.com/golang/mock v1.2.0
github.com/golang/protobuf v1.2.0
github.com/golang/snappy v0.0.0-20180518054509-2e65f85255db
github.com/google/btree v0.0.0-20180813153112-4030bb1f1f0c
github.com/google/go-cmp v0.2.0
github.com/google/go-github v17.0.0+incompatible
github.com/google/go-querystring v1.0.0
github.com/google/martian v2.1.0+incompatible
github.com/google/pprof v0.0.0-20181206194817-3ea8567a2e57
github.com/googleapis/gax-go v2.0.0+incompatible
github.com/googleapis/gax-go/v2 v2.0.3
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1
github.com/gorilla/context v1.1.1
github.com/gorilla/handlers v1.4.0
github.com/gorilla/mux v1.7.0
github.com/gorilla/websocket v1.4.0
github.com/gregjones/httpcache v0.0.0-20180305231024-9cad4c3443a7
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0
github.com/grpc-ecosystem/grpc-gateway v1.6.2
github.com/jellevandenhooff/dkim v0.0.0-20150330215556-f50fe3d243e1
github.com/jmespath/go-jmespath v0.0.0-20180206201540-c2b33e8439af
github.com/julienschmidt/httprouter v1.2.0
github.com/kelseyhightower/envconfig v1.3.0
github.com/kisielk/gotool v1.0.0
github.com/konsorten/go-windows-terminal-sequences v1.0.1
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515
github.com/kr/pretty v0.1.0
github.com/kr/pty v1.1.3
github.com/kr/text v0.1.0
github.com/matttproud/golang_protobuf_extensions v1.0.1
github.com/microcosm-cc/bluemonday v1.0.1
github.com/neelance/astrewrite v0.0.0-20160511093645-99348263ae86
github.com/neelance/sourcemap v0.0.0-20151028013722-8c68805598ab
github.com/nu7hatch/gouuid v0.0.0-20131221200532-179d4d0c4d8d
github.com/openzipkin/zipkin-go v0.1.3
github.com/pierrec/lz4 v2.0.5+incompatible
github.com/pkg/errors v0.8.1
github.com/pmezard/go-difflib v1.0.0
github.com/prometheus/client_golang v0.9.2
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910
github.com/prometheus/common v0.0.0-20181218105931-67670fe90761
github.com/prometheus/procfs v0.0.0-20181204211112-1dc9a6cbc91a
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a
github.com/russross/blackfriday v1.5.2
github.com/sergi/go-diff v1.0.0
github.com/shurcooL/component v0.0.0-20170202220835-f88ec8f54cc4
github.com/shurcooL/events v0.0.0-20181021180414-410e4ca65f48
github.com/shurcooL/github_flavored_markdown v0.0.0-20181002035957-2122de532470
github.com/shurcooL/go v0.0.0-20180423040247-9e1955d9fb6e
github.com/shurcooL/go-goon v0.0.0-20170922171312-37c2f522c041
github.com/shurcooL/gofontwoff v0.0.0-20180329035133-29b52fc0a18d
github.com/shurcooL/gopherjslib v0.0.0-20160914041154-feb6d3990c2c
github.com/shurcooL/highlight_diff v0.0.0-20170515013008-09bb4053de1b
github.com/shurcooL/highlight_go v0.0.0-20181028180052-98c3abbbae20
github.com/shurcooL/home v0.0.0-20181020052607-80b7ffcb30f9
github.com/shurcooL/htmlg v0.0.0-20170918183704-d01228ac9e50
github.com/shurcooL/httperror v0.0.0-20170206035902-86b7830d14cc
github.com/shurcooL/httpfs v0.0.0-20171119174359-809beceb2371
github.com/shurcooL/httpgzip v0.0.0-20180522190206-b1c53ac65af9
github.com/shurcooL/issues v0.0.0-20181008053335-6292fdc1e191
github.com/shurcooL/issuesapp v0.0.0-20180602232740-048589ce2241
github.com/shurcooL/notifications v0.0.0-20181007000457-627ab5aea122
github.com/shurcooL/octicon v0.0.0-20181028054416-fa4f57f9efb2
github.com/shurcooL/reactions v0.0.0-20181006231557-f2e0b4ca5b82
github.com/shurcooL/sanitized_anchor_name v0.0.0-20170918181015-86672fcb3f95
github.com/shurcooL/users v0.0.0-20180125191416-49c67e49c537
github.com/shurcooL/webdavfs v0.0.0-20170829043945-18c3829fa133
github.com/sirupsen/logrus v1.3.0
github.com/sourcegraph/annotate v0.0.0-20160123013949-f4cad6c6324d
github.com/sourcegraph/syntaxhighlight v0.0.0-20170531221838-bd320f5d308e
github.com/stretchr/objx v0.1.1
github.com/stretchr/testify v1.2.2
github.com/tarm/serial v0.0.0-20180830185346-98f6abe2eb07
go.opencensus.io v0.19.0
go4.org v0.0.0-20180809161055-417644f6feb5
golang.org/x/build v0.0.0-20190111050920-041ab4dc3f9d
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9
golang.org/x/exp v0.0.0-20190121172915-509febef88a4
golang.org/x/lint v0.0.0-20181217174547-8f45f776aaf1
golang.org/x/net v0.0.0-20190125091013-d26f9f9a57f3
golang.org/x/oauth2 v0.0.0-20190130055435-99b60b757ec1
golang.org/x/perf v0.0.0-20180704124530-6e6d33e29852
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4
golang.org/x/sys v0.0.0-20181218192612-074acd46bca6
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c
golang.org/x/tools v0.0.0-20181219222714-6e267b5cc78e
google.golang.org/api v0.1.0
google.golang.org/appengine v1.4.0
google.golang.org/genproto v0.0.0-20190128161407-8ac453e89fca
google.golang.org/grpc v1.18.0
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127
gopkg.in/inf.v0 v0.9.1
gopkg.in/yaml.v2 v2.2.2
grpc.go4.org v0.0.0-20170609214715-11d0a25b4919
honnef.co/go/tools v0.0.0-20190106161140-3f1c8253044a
sourcegraph.com/sourcegraph/go-diff v0.5.0
sourcegraph.com/sqs/pbtypes v0.0.0-20180604144634-d3ebe8f20ae4

I know there's work to separate dependencies, but does it make sense to also separate the SimpleServer and not make it this bloated?

patrick-nyt commented 5 years ago

I agree the SimpleServer should be simple, with minimal dependencies.

jprobinson commented 5 years ago

I think a very large portion of whats above is from the examples, but some is also coming from the Google and AWS tooling within the pubsub packages. For those latter dependencies, I think we're stuck with them until we look to explode gizmo out to a suite of repos. (Something I believe go-kit is also considering) Perhaps we should consider this route? I'm down for chatting about approaches.

Beyond that, I do believe there is some low hanging fruit that can be cut out. For instance, I think github.com/NYTimes/logrotate is no longer needed as we don't log to files anymore. At this point, I think we should definitely consider any PR that proposes we remove a dependency from the server.

For now, I'm going to try and get the examples separated at a bare min. That should ease some of the pains.