Bazel support roadmap

[sqs]

We plan to publish a roadmap for Bazel support. There is an internal roadmap at https://docs.google.com/document/d/150neGhxGhV48LcjAqjUy_oiQ_c8O6xPEgbM_cd5MqrU/edit#heading=h.3xkwxm4m6wdd that needs more work and verification.

[nicksnyder]

Closing because this is stale and conversation should be tracked on our actual roadmap https://docs.google.com/document/d/1cBsE9801DcBF9chZyMnxRdolqM_1c2pPyGQz15QAvYI/edit

[felixfbecker]

@nicksnyder FYI the roadmap links to this issue

[nicksnyder]

easy fix: update roadmap to point to doc

[gravypod]

Hey, I was wondering if first class bazel integration was still a goal of sourcegraph? If sourcegraph could integrate with a monorepo build system like bazel to provide intelligent code completion, search, and supporting integration with a CitC IDE (like #2925) it would be an extremely powerful tool. I'm sure the engineers at sourcegraph already know this but I don't see any mention of this - or similar goals - in the roadmap.

[nicksnyder]

~We don’t have any current plans to work on this but will prioritize this if a paying customer requests this.~

We don't have any current plans to work on this. If you are a potential customer and this is blocking you from using Sourcegraph, then please let us know (because this will change how we think about the priority of supporting Bazel)!

[mikewiacek]

For what it's worth, we'd become a paying customer if this feature existed. But for those of us who are using bazel with a monorepo system, there is no reason to pay for sourcegraph as it doesn't work for us. So @nicksnyder, you've kind of put yourself into a catch 22 by saying: "will prioritize this if a paying customer requests this."

[shrouxm]

@mikewiacek I'm on the code intelligence team and can give you an update. We're actively working on support for mono repos, and bazel integration is definitely something we've considered. We already have users getting hovers and running go-to-def/find references queries on their mono repos by periodically performing an LSIF index. It's not obvious yet that directly working with bazel will be a necessary step to supporting such repos so we're focusing on general solutions first, but we'll definitely come back to this if we need to.

[nicksnyder]

@mikewiacek Sorry for the imprecise language, I didn't mean to create a catch 22 situation. I updated my comment to indicate that hearing from potential future customers like yourself helps us prioritize features like this.

[poojaj-tech]

@mikewiacek I'm a Product Manager on the Sourcegraph team. I wanted to share that we aren’t prioritizing Bazel support at this time because our team is investing in improving our LSIF-indexer for C++. You can follow these improvements in this Github issue. Like Nick and Garo said above, we love hearing about features our customers want as it helps us understand what to work on to improve our customer experience, so thank you for sharing all your great feedback @mikewiacek and @gravypod.

[iangudger]

@poojaj-tech, now that the LSIF-indexer for C++ is done, can Bazel support be reconsidered?

[shrouxm]

hey @iangudger, garo from the intel team again! Specifically for C++, there is an existing tool (https://github.com/grailbio/bazel-compilation-database) which can be used to generate compilation databases for Bazel-based projects. These compilation databases can be consumed by our newly released tool lsif-clang tool to produce LSIF indexes.

I haven't yet tested this integration, but based on your comment I scheduled some time in the next month to investigate a little, and I'll get back to you then to let you know what I learned! You're of course welcome to tinker with the two tools in the meantime, I suspect they may work well together out of the box due to Bazel's inclination towards sandboxing.

[iangudger]

@gbrik, cool, thanks for the update! I am actually specifically interested in Go support.

[poojaj-tech]

@iangudger, I’m curious if you are using Bazel in open source or would this be helpful in your work environment for Go?

[ittaiz]

Hi, I saw on twitter your recent announcement on enhancing jvm languages and wondered about bazel support for these. We (at Wix) are using bazel extensively across our virtual* mono repos with jvm languages.

• virtual- git repositories wise we have ~50 but they work with floating source dependencies while still being reproducible

[nicksnyder]

@macraig @scalabilitysolved

[macraig]

Hi @ittaiz, support for Bazel needs some manual configuration and hasn't been widely tested yet (hence it's not currently advertised in our lsif-java documentation. However, we've had a customer successfully set up precise intel for Bazel in the past month, so we can likely help you set it up to accomplish your needs.

I'm ccing @olafurpg who's our JVM expert and can expand on the technical side, we'd be happy to pair with you on the configuration if you're interested!

[olafurpg]

@ittaiz Bazel is unlikely to get the same out-of-the-box support as Gradle and Maven because that would eliminate many of the biggest benefits of Bazel: distributed builds and reliable caching. There are roughly two approaches you can use today to setup lsif-java:

• Add something like config_setting so you can dynamically enable the SemanticDB compiler plugin for Java/Scala/Kotlin in certain CI jobs.
• Auto-generate some kind of semanticdb() targets for every Java/Scala/Kotlin target. The benefit of this approach is that you get optimal parallelism and minimal CPU overhead by reusing the existing build cache from non-SemanticDB compilation. The downside of this approach is that you need to keep the auto-generated targets up-to-date with all targets in your workspace (could be a cron job or a linter).

I haven't validated either approach but I'd be happy to hop on a call to brainstorm or pair on this. Before I joined Sourcegraph, I worked at Twitter on their Bazel migration so I have some experience with Bazel (although not as much as you do 😜 ).

[ittaiz]

Interesting. Thanks! What I'll try to do is find an engineer who might want to use her "20%" time (very different from google but relevant as a metaphor) and take 2-3 weeks to do a POC. If I get someone like that then I'll know when in advance and maybe we can do a session at the beginning to chat about strategies. Thanks for your time!

[olafurpg]

@ittaiz do you have a minimized example of your Bazel setup that I could clone locally and play around with? I can give it a shot to demonstrate what changes are needed.

[ittaiz]

Not really. Maybe a target or two in rules_scala can demonstrate?

[olafurpg]

@ittaiz are you using rules_jvm_external? We need some way to declare dependencies on the following artifacts:

• com.sourcegraph:semanticdb-javac (Java support)
• org.scalameta:semanticdb-scalac (Scala support)

Also, we would need a bazel query to get metadata about external dependencies (groupID, artifactID, version, jar). This is only needed for navigation to external dependencies but that's an important feature IMO.

[olafurpg]

I opened a draft PR to rules_scala with the goal adding an option to dynamically enable the SemanticDB compiler plugin https://github.com/bazelbuild/rules_scala/pull/1329 Any advice on how to debug the error message I posted in the PR would be appreciated. I'll reopen this issue to make it easier for people to track the progress on Bazel support.

Note that the rules_scala PR is only one step towards supporting Scala support (not Java or Kotlin). The Java support should be relatively easy to implement and I'm not up-to-date on how the Kotlin rules work to estimate how hard it would be to implement.

[sluongng]

As a contributor to rules_go, I am very much interested in making code-intel support work for bazel. Especially for the use case of finding definition accross languages i.e. go -> protobuf or go -> cgo -> c/c++.

Also, in the previous model established by Kythe.io, Bazel exposes aspect https://bazel.build/rules/aspects (or previously deprecated, action-listener) which was used by Kythe extractor to hook into Bazel's build graph and created kzip files. Then Kythe indexer pick up these kzip files to transform them into protobuf and store in a graph DB.

I quickly took a peek into the scip-java repo and saw that current approach uses a bazel query which would only work with repositories with relatively smaller build graph. So I would suggest to support indexing code using aspect in the roadmap.🙏

[olafurpg]

I quickly took a peek into the scip-java repo and saw that current approach uses a bazel query which would only work with repositories with relatively smaller build graph. So I would suggest to support indexing code using aspect in the roadmap.

The bazel query part is an optional feature to support cross-repo navigation to rules_jvm_external dependencies. The only query that's invoked is bazel query kind('.*_import', @maven//...) (https://sourcegraph.com/github.com/sourcegraph/scip-java@6d5d37d814b4421c1fd37c451d93175dc49710a6/-/blob/scip-semanticdb/src/main/java/com/sourcegraph/scip_semanticdb/BazelBuildTool.java?L65), which scales with the number of external dependencies, not the size of the build graph itself. I think we can comfortably estimate that most large monorepos have <10k external JVM dependencies, which should return relatively quickly with bazel query. Maven Central has only ~500k unique artifacts (groupID+artifactName, there are more versions of each artifact).