file '@rules_python//python:repositories.bzl' does not contain symbol 'python_register_toolchains'

[sjain-stanford]

Hi, I'm trying to set up clangd on a repo that is built using bazel (https://github.com/cruise-automation/mlir-tcp) and was giving this a shot. However, I get the following error when running bazel run @hedron_compile_commands//:refresh_all. Any pointers are appreciated.

ERROR: Traceback (most recent call last):                                                                                                                                                           
        File "/home/sambhav.jain/.cache/bazel/_bazel_sambhav.jain/10d8afd615b1bb1c9adf398b5b12d4bf/external/hedron_compile_commands/workspace_setup_transitive.bzl", line 3, column 68, in <toplevel
>                                                                                                                                                                                                   
                load("@rules_python//python:repositories.bzl", "py_repositories", "python_register_toolchains")                                                                                     
Error: file '@rules_python//python:repositories.bzl' does not contain symbol 'python_register_toolchains'                                                                                           
ERROR: Error computing the main repository mapping: initialization of module 'workspace_setup_transitive.bzl' failed                                                                                
Loading:                           

[cpsauer]

Hey Sambhav! That's no good--sorry. We just redid a bunch of python stuff to use Bazel's hermetic python and that might have messed things up. Are you using via bzlmod or WORKSPACE? Could I ask you to make sure you're using the latest commit? And in the meantime, to unblock you, could you hop on back to f41ec092c76458374900242e0d6ed5e96f4e3a21 to make sure things work?

[cpsauer]

Also, I see Stanford in your username and am smiling :) (I'm a Stanford grad.)

[cpsauer]

Hmm, was just running some tests to try to narrow this down, but things are working for me with both Bzlmod and WORKSAPCE. Maybe some other dependency is bringing in an old version of rules_python? Maybe try loading this tool first in your WORKSPACE to narrow that down?

[sjain-stanford]

Hey, thanks for the quick response. I'm using the WORKSPACE method. I will try your suggestions and get back shortly. Appreciate it.

[sjain-stanford]

Maybe some other dependency is bringing in an old version of rules_python? Maybe try loading this tool first in your WORKSPACE to narrow that down?

You're right. I was loading hedron_compile_commands towards the end of our WORKSPACE (here). Moving it to before bazel's buildifier definition seems to work around the initial issue with rules_python.

I get the following now, which probably means it worked? Excited to try it out! :D

>>> Automatically added entries to .git/info/exclude to gitignore generated output.                                                                                                                 
>>> Automatically added //external workspace link:                                                                                                                                                  
    This link makes it easy for you--and for build tooling--to see the external dependencies you bring in. It also makes your source tree have the same directory structure as the build sandbox.   
    It's a win/win: It's easier for you to browse the code you use, and it eliminates whole categories of edge cases for build tooling.                                                             
>>> Analyzing commands used in //:tcp-opt                                                                                                                                                           
>>> A source file you compile doesn't (yet) exist: bazel-out/k8-fastbuild/bin/external/llvm-project/mlir/SerializeToCubin_stub.cc                                                                   
    It's probably a generated file, and you haven't yet run a build to generate it.                                                                                                                 
    That's OK; your code doesn't even have to compile for this tool to work.                                                                                                                        
    If you can, though, you might want to run a build of your code with --keep_going.                                                                                                               
        That way everything possible is generated, browsable and indexed for autocomplete.                                                                                                          
    However, if you have *already* built your code, and generated the missing file...                                                                                                               
        Please make sure you're supplying this tool with the same flags you use to build.                                                                                                           
        You can either use a refresh_compile_commands rule or the special -- syntax. Please see the README.                                                                                         
        [Supplying flags normally won't work. That just causes this tool to be built with those flags.]                                                                                             
    Continuing gracefully...                                                                                                                                                                        
>>> Finished extracting commands for //:tcp-opt            

[cpsauer]

Yep! Yay!

[cpsauer]

Might be worth poking them to update. This is one of the annoying things pre-bzlmod

[sjain-stanford]

@cpsauer happy to open a separate issue, but wanted to discuss here before doing that:

1. Is there a way to support builds run within a docker container, but run clangd on VSCode (using non-docker paths)? Right now the compile_commands.json uses "directory": "/opt/src/mlir-tcp" which exists within the container, and so VSCode's clangd is unable to render the suggestions.
2. Even with full compilation (bazel run @hedron_compile_commands//:refresh_all), there are still some symbols that are not recognized by clangd. Any tips on how to go about fixing these?

[cpsauer]

Hello again, Sambhav :)

Docker: I think I've heard of folks using it with (e.g. in issues), but I'm afraid I don't have personal experience yet. Would you be down to pioneer it, since you're using firsthand and can experiment and test (and I'm slammed, sorry :/). See the ImplementationReadme.md--it's easy to hack on this repo and test changes live.

Missing symbols: Is it that headers aren't being found or that they're the wrong headers (or something else)?

A couple possible culprits: (1) If top-level rules are configuring their dependencies, e.g. to cross-compile or put something into the docker container, you'll want to use a refresh_compile_commands rule to list the top level targets you build, like in the readme, rather than :refresh_all. Otherwise things won't be configured correctly, so it might look for the headers in the wrong places. (2) Might they be generated files that haven't been built yet (or are in the docker container)? I'm guessing from the warnings in the log you posted above, but I really don't have enough information yet here.

Thanks for sticking with the tool! I know how tricky bazel setup can be!

[sjain-stanford]

Hey Chris. I didn't get a chance to look into the docker issue yet. On the bright side, the missing symbols issue is resolved after adding all the top-level targets to the refresh_compile_commands rule as you suggested (commit). I see all symbols are picked by clangd now! Thanks for all your help again.

[cpsauer]

Great! Delighted to hear :) I took a quick peek at the docker instructions in your repo, btw. Still not an area I have experience with, unfortunately, but might vscode remote (i.e. dev inside the container) solve things?