search

higherkindness / rules_scala

Robust and featureful Bazel rules for Scala
Apache License 2.0
66 stars 28 forks source link

Faster incremental compilation options #256

Closed stephenjudkins closed 4 years ago

stephenjudkins commented 4 years ago

Summary

Profiling showed that the majority of time spent–for both small modules with many local Scala dependencies, and incremental compilation–was on unpacking zipfiles and later deleting them. For incremental compilation, I added a flag --extracted_file_cache that can hold these unpacked files between runs. It's content-addressed by jar file content, so there's little chance of conflict.

For non-incremental compilation, I stopped extracting these files and simply treated them as external libraries; this removed the slowdown completely.

Motivation

Our performance problems when incrementally recompiling a file in a "leaf" project, with lots of large local dependencies, were pretty bad. I profiled it and found that the overwhelming majority of time was spent unpacking classfiles from local deps and then deleting them: https://gist.github.com/stephenjudkins/0700dd627d7ec2f807fc3a9dc94a1880

This work was performed regardless if incremental compilation was enabled. On CI, when it is not, we can simply not unpack the files and treat them as library deps; locally, when incremental compilation is used, we can unpack them to a "cache" directory and reuse the unpacked files.

Open questions

Why are the dependencies unpacked in the first place? Why can't Zinc use jars directly? This deserves further investigation, but I don't fully understand at the moment.

Further work

https://github.com/sbt/zinc/pull/712 could allow the plugin to avoid unpacking files entirely

stephenjudkins commented 4 years ago

I'll post some data on the (significant) speedups this gave us later.

SrodriguezO commented 4 years ago

@stephenjudkins thank you for looking into this and for following it up with a PR! I'm not very familiar with the intricacies of this part of the project, so the review won't be trivial (and unfortunately I have no bandwidth till next week).

@pauldraper and @andyscott may be more knowledgeable about this. If they can't get to it early next week, @borkaehw and I can dive in.

andyscott commented 4 years ago

This seems reasonable to me. Once @SrodriguezO's requested changes are addressed, 👍.