build_runner exponential memory usage and crashes

[Serena867]

I'm using build_runner in a fairly atypical way to generate a substantial amount of boilerplate using templates to punch out all the boilerplate for layered architecture and a few other things. I took basically the opposite approach that the Dart team recommends in that I only need to write small extensions to the generated code when needed. I ran into an issue this evening while trying to run it with 17 builders. I spent a fair amount of time digging through my own code, previous issues, stackoverflow questions, etc. I've checked for white space issues, and anything else that might have come up. I'm aware that I'm using it in a way that it wasn't specifically built for, but up until today it wasn't an issue.

When using 14 or fewer builders the build times are fine all things considered (around 30s - 1min), but with 15 or 16 builders the build times increased exponentially as does the memory usage. However, with 17 builders my memory usage would increase until I got an "Unhandled build failure!" as a result of running out of memory. My build_runner snapshot which would typically be around 4-5GB with 14 builders runs up to about 30GB+ before the process crashes with 17 builders.

What I learned was that I could delete literally any 3 of the builders and my build times would go back to normal. I've set targets, run different build config files, build orders, sources, generate_for(s), etc. The only thing that has worked so far is actually deleting the builders from the build.yaml file within the code generator library itself.

Whether this is a result of a particular limit I'm not aware of, an issue with build_runner itself, or perhaps something else it would be nice to be able to run this amount of code generation.

Unfortunately when it crashes this way it doesn't save any log files either. The packages that it references upon crashing tend to change each time as well.

The computer is a MBP with the M2 Max and 64GB of ram. While I suspect this is irrelevant, I will also mention that the system itself tends to have around 15GB+ of available ram and isn't running swap when these crashes occur.

build_runner version is 2.4.8 dart version is 3.2.3

[INFO] 1m 4s elapsed, 10656/10657 actions completed.Exhausted heap space, trying to allocate 176 bytes.
Exhausted heap space, trying to allocate 48 bytes.
Exhausted heap space, trying to allocate 48 bytes.
[SEVERE] Unhandled build failure!
Out of Memory
package:build_runner_core           _SingleBuild._runPhases.<fn>.<fn>
package:timing/src/timing.dart 179:22  SimpleAsyncTimeTracker.track

[SEVERE] Failed after 7m 32s

[jakemac53]

How are your builders set up, and specifically the build extensions? Are they possibly generating an exponential amount of files?

[Serena867]

How are your builders set up, and specifically the build extensions? Are they possibly generating an exponential amount of files?

The builders themselves are setup to check the library annotations and file extensions prior to code generation so anything that fails those checks won't get run or output.

The build extensions themselves are all set to a single unique output extension and are all setup in a manner similar to this:

build_extensions: { ".dart": [ ".controller.dart" ] }

I picked a number of builders to disable at random in a large number of manual tests and the outputs were what I was expecting each time. A single output per builder per input file.

Edited to add: There are a few other checks in place, but I just wanted to mention the basic ones.

[jakemac53]

Do they all have the .dart input extension?

[jakemac53]

If so, the total # of potential assets will be n^2, and the way build_runner works is the graph contains every single possible asset that might be generated. So just not generating them isn't enough to remove them from the asset graph. We have to do this in order to support "optional" (lazily) built outputs. Any build step can ask to read any potentially generated file, so we have some information stored about all the potential files, to know what to try and build to produce the file that was asked for.

[Serena867]

Do they all have the .dart input extension?

I just confirmed that they do all have the .dart input extension.

Also, the generated files automatically get moved to another directory so generated files themselves are never run in subsequent build_runner builds. The old files get overwritten when the new files are moved if they required a rebuild. A lot of the checks to make sure they're not creating unnecessary outputs that are in places are redundant, but relatively speaking, are reasonably cheap.

[jakemac53]

Ok yeah, so see my comment above then. To prevent this you would have to have more specific input file extensions (if possible), or some other way of limiting the potential outputs, maybe using generate_for etc.

[Serena867]

Ok yeah, so see my comment above then. To prevent this you would have to have more specific input file extensions (if possible), or some other way of limiting the potential outputs, maybe using generate_for etc.

It definitely makes sense that something like this could be the case.

Each builder does use generate_for and only targets the specific directories they need to. All the generated files are moved out of the source directory and placed elsewhere so they're never targeted.

[jakemac53]

All the generated files are moved out of the source directory and placed elsewhere so they're never targeted.

I am not sure what this means?

[Serena867]

All the generated files are moved out of the source directory and placed elsewhere so they're never targeted.

I am not sure what this means?

The generator_for for each builder have a very narrow scope. I list specific directories that only include the original source file. When build_runner outputs new files they all automatically get moved to new directories as a final step.

I definitely use build_runner in a manner for which it was not specifically intended to be used.

Reducing the number of targets does seem to have improved the situation. Do you think I'll have better luck if I use specific build config files to target smaller parts of the project with a more limited number of generate_for for each config file? For what it's worth, this generator isn't being run on what I'd likely consider an excessively large project and I did try to limit the number of potential outputs ahead of time. In that regard I'd say the potential outputs are about as limited as they can get. It seems I ran into issues there relatively quickly for what I'm doing.

We have to do this in order to support "optional" (lazily) built outputs.

Given that's the case, I'd imagine it's unlikely this is something that can be reasonably resolved by the Dart team?

[jakemac53]

Have you looked into capture groups? If you can use those to specify the output directories for your generated files, it might fix things.

When build_runner outputs new files they all automatically get moved to new directories as a final step.

This is something you are doing outside of the build itself right? Like manually moving files afterwords?

If so, this will not help the issue, because according to build_runner they live where it output them, and it will create corresponding asset graph nodes etc as if they are there.

Do you think I'll have better luck if I use specific build config files to target smaller parts of the project with a more limited number of generate_for for each config file?

Yes, in this situation each build only sees the files on disk, so the moving around of them works (the other builds don't know about the "potential" outputs of the other ones).

It seems I ran into issues there relatively quickly for what I'm doing.

n^2 catches up to you quickly haha

Given that's the case, I'd imagine it's unlikely this is something that can be reasonably resolved by the Dart team?

Correct, this is an aspect of the fundamental design you are running into. It is probably theoretically possible to avoid this issue in some way, but it would be quite challenging. Not something we can look into for the foreseeable future (especially with macros potentially coming down the line, which won't have this problem).

[Serena867]

Have you looked into capture groups? If you can use those to specify the output directories for your generated files, it might fix things.

I did take a look at them, and used them early on for most things. I'll take a deeper dive into them again. The issue I had was that I also generate new directories with the builder if they don't already exist so it would have required changing parts of build_runner itself to achieve what I needed there when emitting the output files. I do use them for the protobuf outputs however, and for that they 100% meet my needs.

When build_runner outputs new files they all automatically get moved to new directories as a final step.

This is something you are doing outside of the build itself right? Like manually moving files afterwords?

No, I did it the lazy way, and I use a post process builder to actually move all the files. Based on what you've mentioned I can't imagine that would affect the asset graph nodes for obvious reasons, but I do clear the cache files between runs for that reason. Given what I was doing it made that the simplest, but not best option. I figured I'd circle back to that when the builders were closer to complete and resolve those issues at that point.

The ultimate goal was to actually validate the idea and circle back around to optimize afterwards even if it meant I had to fork build_runner and make some changes for my use case.

If so, this will not help the issue, because according to build_runner they live where it output them, and it will create corresponding asset graph nodes etc as if they are there.

Do you think I'll have better luck if I use specific build config files to target smaller parts of the project with a more limited number of generate_for for each config file?

Yes, in this situation each build only sees the files on disk, so the moving around of them works (the other builds don't know about the "potential" outputs of the other ones).

Awesome, thanks! I suspect between a combination of config files and scripts I can likely hack something together until I start digging through build_runner.

It seems I ran into issues there relatively quickly for what I'm doing.

n^2 catches up to you quickly haha

haha it sure does!

Given that's the case, I'd imagine it's unlikely this is something that can be reasonably resolved by the Dart team?

Correct, this is an aspect of the fundamental design you are running into. It is probably theoretically possible to avoid this issue in some way, but it would be quite challenging. Not something we can look into for the foreseeable future (especially with macros potentially coming down the line, which won't have this problem).

Totally understandable. For what it's worth, I've been following the progress with macros and I'm very excited about them.

Also, I just want to say thank you for being so approachable and responsive. I'm genuinely grateful for your time and help today.

[jakemac53]

Also, I just want to say thank you for being so approachable and responsive. I'm genuinely grateful for your time and help today.

Likewise, thanks for the quick responses on your end, it helps to keep the context in my brain instead of long gaps in between 🤣 .

I am going to close this issue, as there is no open task on our end, but feel free to chime in again here or on a new issue, regarding capture groups or other issues 👍 .

[jakemac53]

@Serena867 another idea that came up, discussing this with the team, is you could use excludes in your generate_for configuration, to exclude specific file extensions that you know are generated and don't want to run on, or possibly all files with multiple extensions as a proxy for generated files:

generate_for:
  include: 
    - lib/stuff/*.dart
  exclude:
    - lib/stuff/*.*.dart # Exclude any dart file with multiple extensions, its probably generated

[Serena867]

@Serena867 another idea that came up, discussing this with the team, is you could use excludes in your generate_for configuration, to exclude specific file extensions that you know are generated and don't want to run on, or possibly all files with multiple extensions as a proxy for generated files:

generate_for:
  include: 
    - lib/stuff/*.dart
  exclude:
    - lib/stuff/*.*.dart # Exclude any dart file with multiple extensions, its probably generated

Thanks Jake!

I'll give that a try shortly here and report back in the next little bit.

[Serena867]

@jakemac53

I'm very happy to report that adding by using very specific includes and adding the excludes it solved the issue, and the code generator is now doing complete builds on the whole project in around 12 seconds.

I need to extend another big thank you to both you and your team. I wasn't expecting to hear anything more about this, and so even just knowing you and your team continued to think on it was a very pleasant surprise that absolutely made my day!