search

google / bazel-common

Common functionality for Google's open-source libraries that are built with bazel.
Apache License 2.0
85 stars 40 forks source link

Script to automatically update all library versions #65

Open cpovirk opened 5 years ago

cpovirk commented 5 years ago

Similar to Maven commands like:

And just to spam more keywords, here are some Maven commands that display updates but don't make the changes:

I poked around the web and found some a project for keeping Bazel deps up to date, but it has its own system that might not play well with bazel-common: https://github.com/johnynek/bazel-deps

I notice that it's mentioned in: https://github.com/bazelbuild/BUILD_file_generator#third-party-maven-dependencies

ronshapiro commented 5 years ago

We could decide to do something with this: https://blog.bazel.build/2019/03/31/rules-jvm-external-maven.html. I'd like to figure out what it does with dependency conflicts first to make sure.

Then we can install that with a private repository name and then wire up processors + exports manually

jin commented 5 years ago

@ronshapiro it has the resolution logic of Coursier. I can imagine instantiating a custom maven_install for bazel-common, like

maven_install(
    name = "bazel_common_maven",
    artifacts = # bazel common artifacts,
    repositories = # repos,
)

And then downstream libraries will depend on @bazel_common_maven//:com_google_guava_guava and @bazel_common_maven//:junit_junit, for example. They can also declare additional dependencies using their own maven_install declarations, which will be namespaced by the repository name.

Let me know if you have other questions or suggestions for rules_jvm_external.

cpovirk commented 4 years ago

Another thing for us to consider is using the Google Cloud Platform Libraries BOM (Maven central, GitHub). It provides a set of libraries that have been statically checked for linkage errors. This could help us avoid errors like this one (fix) (and maybe there was another error that required this?).

If I did my testing right, then the BOM doesn't yet cover everything that we'd like it to. But maybe we can help add some libraries, or at least we can use the BOM as our own starting point.

[edit: The commands below are my attempt to generate a list of libraries that bazel-common includes but the BOM does not specify a version for -- neither explicitly (by listing them) nor implicitly (by listing a library that depends on them).]

$ ( tmp && mvn help:effective-pom -f $HOME/.m2/repository/com/google/cloud/libraries-bom/3.3.0/libraries-bom-3.3.0.pom -Doutput=/tmp/wholeep && ( echo "plugins {"; echo "    id 'java-library'"; echo "}"; echo "repositories {"; echo "    jcenter()"; echo "}"; echo "dependencies {"; xmlstarlet sel -t -m '/_:project/_:dependencyManagement/_:dependencies/_:dependency' -v _:groupId -o : -v _:artifactId -o : -v _:version -n /tmp/wholeep | sort | prepend "    implementation '" | append "'" ; echo '}' ) > build.gradle && $HOME/gradle-6.0.1/bin/gradle :dependencies | grep -v '[-]>' | grep -o '[^: ]*:[^: ]*:[^ ]*' | sort -u | grep -Fv '[' > /tmp/bomversions )

$ ( cd $HOME/clients/bazel-common-gray/bazel-common/ && bazel query --output=xml 'attr(tags, maven_coordinates.*, kind(java_import, deps(//...)))'  | grep -oP 'maven_coordinates=\K[^:]*:[^:]*:[^:"]*' | sort > /tmp/bazelcommonversions )

$ comm -23 <(grep -o '^[^: ]*:[^: ]*' /tmp/bazelcommonversions) <(grep -o '^[^: ]*:[^: ]*' /tmp/bomversions)
com.google.auto:auto-common
com.google.auto.factory:auto-factory
com.google.auto.service:auto-service
com.googlecode.java-diff-utils:diffutils
com.google.common.inject:inject-common
com.google.errorprone:error_prone_annotation
com.google.errorprone:error_prone_check_api
com.google.errorprone:javac-shaded
com.google.googlejavaformat:google-java-format
com.google.guava:guava-beta-checker
com.google.testing.compile:compile-testing
com.google.truth.extensions:truth-java8-extension
com.google.truth:truth
com.squareup:javapoet
io.grpc:grpc-all
javax.annotation:jsr250-api
javax.enterprise:cdi-api
javax.inject:javax.inject-tck
log4j:log4j
net.bytebuddy:byte-buddy
net.bytebuddy:byte-buddy-agent
net.ltgt.gradle.incap:incap
net.ltgt.gradle.incap:incap-processor
org.apache.ant:ant
org.apache.ant:ant-launcher
org.apache.bcel:bcel
org.apache.logging.log4j:log4j-api
org.apache.logging.log4j:log4j-core
org.apache.maven:maven-artifact
org.apache.maven:maven-model
org.apache.maven:maven-plugin-api
org.checkerframework:checker-qual
org.checkerframework:dataflow
org.checkerframework:javacutil
org.codehaus.plexus:plexus-classworlds
org.codehaus.plexus:plexus-component-annotations
org.codehaus.plexus:plexus-utils
org.eclipse.sisu:org.eclipse.sisu.inject
org.eclipse.sisu:org.eclipse.sisu.plexus
org.mockito:mockito-core
org.objenesis:objenesis
org.ow2.asm:asm
org.ow2.asm:asm-commons
org.ow2.asm:asm-tree
org.pantsbuild:jarjar
cpovirk commented 4 years ago

I just saw https://github.com/google/bazel-common/commit/842dd97068470df6ab66a9c69ce9ef2524a78df3, which fixes a linkage error, fly by.

(One thing worth noting from that is that some of the artifacts in our list are build tools, not "traditional dependencies" in the narrower sense. If we're lucky, we can easily find compatible versions of all those artifacts, build tools and "traditional dependencies" both. But if not, it would be sufficient to find a set of compatible versions for each individual tool (in addition, of course, to the set for all the "traditional dependencies").)

cpovirk commented 4 years ago

Another linkage error fixed in https://github.com/google/bazel-common/commit/6db2b87d28f765f0064bc437bd489b432ec101b8.

cpovirk commented 3 years ago

The even easier thing would be automatic updates, like we get from Dependabot for most of our repos.

As of this writing, Dependabot doesn't support Bazel, but RenovateBot does. Note that RenovateBot does not yet support maven_install, but we are not using that yet.

I see a smallish number of renovate.json files in the third-party section of our internal repository. But I see some evidence that it's used from almost all repos under https://github.com/googleapis, and I see that Google's GitHub admins are willing to grant the needed permissions to the RenovateBot app. So this seems doable.

cpovirk commented 3 weeks ago

I suspect that this will essentially happen automatically if we switch to rules_jvm_external or whatever the most standard thing is nowadays.