Refactors scripts/create_repository.py to make more modular, robust, and efficient, and update Scalatest, protobuf-java, and kind-projector jars. Part of #1482.
There are a lot of changes from individual commits in this one large PR, since @simuons said it wasn't necessary to break up #1639, another large PR to refactor scripts/create_repository.py, and all tests continue to pass. That said, I'm always more than happy to break this one apart if it really is too much.
The highlights are:
Bumps to Scalatest 3.2.19, protobuf-java 4.28.3, and kind-projector 0.13.3
Each of the output dictionaries are now sorted by key.
This makes it much easier to see when and how dependencies change from this point forward, since the ordering of artifacts will remain stable.
There are now several distinct objects, making the relationships between groups of functions and any shared data much clearer.
The object based design also has made it easier to tweak details and add features without fear of violating scoping assumptions.
If the label naming algorithm changes, or any artifacts are added to EXCLUDED_ARTIFACTS, the all script will update existing dictionary data appropriately.
Even if an artifact isn't upgraded by the script, the script will update its dictionary key and all deps labels referring to it, or remove the artifact and its references altogether.
The script does not yet, however, handle updating macros that instantiate artifact repositories from this data. I have ideas for how to do that, but it's out of scope for this change.
Perhaps most significantly, this cuts the number of Coursier commands and artifact downloads significantly, leading to a very large performance improvement. Here are the times for creating a fresh output dir and then updating it before this change (all with a warm Coursier download cache):
$ /usr/bin/time ./scripts/create_repository.py --output_dir before
[ ...snip... ]
66.01 real 13.25 user 9.99 sys
$ /usr/bin/time ./scripts/create_repository.py --output_dir before
[ ...snip... ]
1.76 real 1.41 user 0.49 sys
Here are the times after:
$ /usr/bin/time ./scripts/create_repository.py --output_dir after
[ ...snip... ]
1.16 real 0.84 user 0.39 sys
$ /usr/bin/time ./scripts/create_repository.py --output_dir after
[ ...snip... ]
0.96 real 0.72 user 0.27 sys
Motivation
Overall, this script is making my further work to update dependencies to ease the Bzlmod migration in #1482 so much faster, easier, and safer. But I found ways to improve it further, after #1639, to make the script itself more maintainable, more correct and robust, and much faster.
I'm now using it to bump ScalaPB, protoc-bridge, gRPC, guava, etc. dependencies, and resolving any breakages, in an effort to make rules_scala able to use newer versions of the Protobuf library. Though independent from (and not strictly required by) the Bzlmod migration work, this may be especially important to future Bzlmod users that include later Protobuf versions in their dependencies.
The script is now in a state where I really trust it to do the right thing as I continue this work. This makes this complex upgrade work far more tractable than before.
Description
Refactors
scripts/create_repository.pyto make more modular, robust, and efficient, and update Scalatest, protobuf-java, and kind-projector jars. Part of #1482.There are a lot of changes from individual commits in this one large PR, since @simuons said it wasn't necessary to break up #1639, another large PR to refactor
scripts/create_repository.py, and all tests continue to pass. That said, I'm always more than happy to break this one apart if it really is too much.The highlights are:
EXCLUDED_ARTIFACTS, the all script will update existing dictionary data appropriately.depslabels referring to it, or remove the artifact and its references altogether.Perhaps most significantly, this cuts the number of Coursier commands and artifact downloads significantly, leading to a very large performance improvement. Here are the times for creating a fresh output dir and then updating it before this change (all with a warm Coursier download cache):
Here are the times after:
Motivation
Overall, this script is making my further work to update dependencies to ease the Bzlmod migration in #1482 so much faster, easier, and safer. But I found ways to improve it further, after #1639, to make the script itself more maintainable, more correct and robust, and much faster.
I'm now using it to bump ScalaPB, protoc-bridge, gRPC, guava, etc. dependencies, and resolving any breakages, in an effort to make rules_scala able to use newer versions of the Protobuf library. Though independent from (and not strictly required by) the Bzlmod migration work, this may be especially important to future Bzlmod users that include later Protobuf versions in their dependencies.
The script is now in a state where I really trust it to do the right thing as I continue this work. This makes this complex upgrade work far more tractable than before.