search

99designs / gqlgen

go generate based graphql server library
https://gqlgen.com
MIT License
9.92k stars 1.16k forks source link

High Memory Usage During Generation #2981

Open agratta-silo opened 6 months ago

agratta-silo commented 6 months ago

What happened?

Running gqlgen in our project with a few hundred types can lead to >12 GB of memory usage on my local MacBook M2, as well as consuming a large portion of the CPU. It runs for ~1-2 minutes locally, but can take 5 minutes or longer (and event time out) in our CI pipelines, likely due to memory constraints.

This seems to be due to binding medium-to-large Go packages, either through autobind or specifically binding particular models.

If I run gqlgen with a tiny dummy GQL schema and no Go bindings, the process uses ~500 MB and finishes within a few seconds. However if I declare a type binding to a single medium-to-large Go package, the process requires 3-5GB depending on the package, and takes >10s. This is true even if I bind to a tiny Go package that happens to import one of the larger packages.

Our real project of course binds to a number of packages, so we see even greater memory usage.

What did you expect?

Much lower memory usage, faster generation times on memory constrained systems.

Minimal graphql.schema and models to reproduce

type Example {
  id: ID!
}

Simply specifying an autobind to one of our larger packages with this tiny schema causes the problem.

versions

agratta-silo commented 6 months ago

I see a similar performance issue, but to a lesser degree, if I instead import the package in the resolver rather than binding to it

agratta-silo commented 6 months ago

it seems like this issue stems from the call to packages.Load, so either that utility has a performance issue or maybe there is a lighter weight alternative that gqlgen could leverage to analyze bindings

agratta-silo commented 6 months ago

A simple solution seems to be reducing the number of options we pass to the packages.LoadMode parameter. packages.NeedImports and packages.NeedDeps seem to be the flags that cause a huge amount of extra data to be loaded, including info for all transitive dependencies and sub-dependencies. This extra info is completely unused by gqlgen AFAICT.

There is only a single usage of package TypeDefs in binder.go which doesn't seem to need the nested dependencies.

Package Imports are only used to add/evict the dependencies from the package cache. There doesn't seem to be any actual usage of these Imports/dependencies, so why load and cache them?

StevenACoffman commented 2 months ago

See #3182 #3181

StevenACoffman commented 2 months ago

So the merged fix #2988 for this causes problems in other builds.

e.g. Bazel builds no longer work, packages are not pulling in their types correctly after https://github.com/99designs/gqlgen/pull/2988 was landed.

So I am minded to revert this fix, but then the original issue of excessive memory consumption comes back. I'm looking for help fixing this issue.