search

bazelbuild / intellij

IntelliJ plugin for Bazel projects
https://ij.bazel.build/
Apache License 2.0
763 stars 305 forks source link

Automatically create and update project-view files #6365

Open ahmedyarub opened 7 months ago

ahmedyarub commented 7 months ago

Description of the feature request:

In this ticket, I propose adding a feature that generates and updates the project-view file. There would be an option to whether add new directories/targets as disabled by default. I'm considering implementing the generator as a Python Starlark (.bzl) script so that it can be used both by the plugin and in CI/CD pipelines. When ran inside the plugin, the tool would first show a confirmation dialog box to confirm what is going to be added and what is going to be removed. The script itself can run in dry mode also, returning only the changes which would be shown in the terminal, or in the confirmation modal in the case of the plugin.

Which category does this issue belong to?

No response

What underlying problem are you trying to solve with this feature?

I work on a large mono-repo and I'm trying to add a series of enhancements that would make switching between targets easier. In a previous issue I've added some buttons for disabling/enabling directories and targets, and a button that shows the Add Directory box.

What operating system, Intellij IDE and programming languages are you using? Please provide specific versions.

Windows/IntelliJ UE 2024.1 RC/Java & Go

Have you found anything relevant by searching the web?

No

Any other information, logs, or outputs that you want to share?

No response

ahmedyarub commented 7 months ago
  1. Create a .bzl for updating the directories to the project-view file of the project the Starlark script should add every directory that has a BUILD file. This can be both used from the plugin and from CI/CD.
  2. Add a new option to BlazeImportSettings in workspace.xml that points to the imported file. The reason behind that is that we want the changes in the original project view file to be reflected into our local project-view file (after being updated by CI/CD and rebasing the branch for example).
  3. Check whether files were added or removed in the original file and update it accordingly
  4. Enable hierarchical structure when rendering project view files. Since in some cases we might end with many directories in the project-view file, the best way to guarantee granularity and not having a very cluttered view is to virtually (meaning that there will be no change to the project-view file's structure) grouped. Ex: dir1 has a BUILD file, so it is added automatically to project-view file by the process described above. In addition to that we will have dir1/dir11 & dir1/dir12 etc... Instead of showing all the targets (which can become a huge list in some projects, the latter directories would be grouped under the former, preserving the ability to enable/disable either partially (by toggling a child) or completely (by toggling a parent). This already partially implemented by the PR I sent yesterday.
  5. After that I would start on the main goal of this whole process: every operation mentioned above should make good use of both Bazel and Shared-Indexes. I won't go in details into this process but the goal is to make most of the operations instantaneous even for huge repos.
tpasternak commented 7 months ago

Oh, that's a lot 😅

@blorente @mai93 @LeFrosch may I ask you for your opinions, too?

ahmedyarub commented 7 months ago

Oh, that's a lot 😅

@blorente @mai93 @LeFrosch may I ask you for your opinions, too?

I'll do that in small pieces. Each one will have its issue, (a small) PR, and be completely backward compatible.

blorente commented 7 months ago

I love the idea! It looks like a huge chunk of work to implement correctly and efficiently, but if you're up for it I'm more than excited!

I see a few big questions we should discuss before diving into specific implementations:

Add a new option to BlazeImportSettings in workspace.xml that points to the imported file.

Some implementation-specific questions:

Create a .bzl for updating the directories to the project-view file of the project the Starlark script should add every directory that has a BUILD file. This can be both used from the plugin and from CI/CD.

  • How about using a gazelle extension for this? Gazelle is already the de-facto file generator standard for Bazel and the ecosystem seems to be collating around it.
ahmedyarub commented 7 months ago

1- I did not talk about file-watchers here. These are in another ticket. 2- You are right, I have just noticed this feature yesterday. 3- Why do you think that it competes with Query Sync? As you might have already know: I have access to huge Bazel based Go, Python, and Java repos which I'm planning to use to test Query Sync and fix any problem I might face (it currently simply doesn't work for us but that is an issue for another thread). 4- I will certainly consider that!

tpasternak commented 7 months ago

I'm not entirely sure about the source data that would be passed into the generator. Would it include the entire directory structure along with all the BUILD files?

ahmedyarub commented 7 months ago

My idea for the first version was simply adding every folder that has a BUILD file. I know that this would generate a very bulky view file BUT in my next PR I will add collapsing: parent directories are displayed and their sub-directories are collapsed by default and can be expanded when required. Of course all the parents and children would have the fancy checkboxes to granularly enable/disable any folder. I'm open to any other suggestions btw.

ahmedyarub commented 7 months ago

I love the idea! It looks like a huge chunk of work to implement correctly and efficiently, but if you're up for it I'm more than excited!

I see a few big questions we should discuss before diving into specific implementations:

  • I think the biggest risk for performance is having a file watcher over the whole repo to update changes. In particular, what if I run gazelle while the system watches files? How could we architect the solution so that a huge, sudden burst of changes doesn't freeze the editor? Could we go into a bit more detail on how you expect the file watching lifetime to work, which file watcher you're intending to use, etc?

Add a new option to BlazeImportSettings in workspace.xml that points to the imported file.

  • This sounds like it could be accomplished by having the local project view file (the one we would generate with this feature) import the repository's project view file. We can already accomplish that from the wizard (by choosing "Import project view file" when we import a project). Could you elaborate on why we'd need a new option?
  • This feels like it competes with Query Sync. @mai93, do you have any insight on the charter for that feature, and how it would fit with this one?

Some implementation-specific questions:

Create a .bzl for updating the directories to the project-view file of the project the Starlark script should add every directory that has a BUILD file. This can be both used from the plugin and from CI/CD.

  • How about using a gazelle extension for this? Gazelle is already the de-facto file generator standard for Bazel and the ecosystem seems to be collating around it.

I talked to my fellow engineers and it seems that gazelle is only working correctly for Go projects. What do you recommend in this case?

tpasternak commented 7 months ago

My idea for the first version was simply adding every folder that has a BUILD file. I know that this would generate a very bulky view file BUT in my next PR I will add collapsing: parent directories are displayed and their sub-directories are collapsed by default and can be expanded when required. Of course all the parents and children would have the fancy checkboxes to granularly enable/disable any folder. I'm open to any other suggestions btw.

I can only speak for myself since I don't know how other users feel. Personally, I prefer typing with good autocompletions to clicking through an expandable tree and checking boxes. We used to suggest that organizations maintain a separate projectview file for each team, which made maintenance easier for the teams themselves.

Regarding the directory collapsing feature, it's important to remember that the order of items in these files matters. Have you considered asking for feedback on the Bazel Slack channel?

ahmedyarub commented 7 months ago

I totally understand you it all depends on the type of the project. I propose having a toggle to enable/disable collapsing according to your needs. As for maintaining separate projectview files: this doesn't really work in the case where you need to work on an arbitrary set of sub-projects in a mono-repo. Another option that we might consider is having a centralized projectview that imports the other projectview files, I think that this would add too many files to big projects, one for each BUILD file. That's why I'm suggesting using the already existing BUILD files as a base for the automatic creation of projectview files. As for the ordering, yes I do understand the problem but I have two solutions for that that we can choose from: 1- Auto-ordering (which can be enabled from options). 2- Collapse the children regardless of their orders.

TLDR: For a mono-repo automatic generation and updating of these files and the ability to quickly show/hide targets at various granularity levels is crucial.

blorente commented 7 months ago

I talked to my fellow engineers and it seems that gazelle is only working correctly for Go projects. What do you recommend in this case?

That was the case around 6 months ago, but it's no longer true. There are well-supported and reasonably complete extensions for Java and Rust (we use both daily in big projects), and I think people are working on the Python, JS and Swift extensions.

However, I've also thought more about it, and it might not be worth the hassle to introduce Go to the repository. Gazelle gives you tons of things out of the box (particularly in parsing BUILD files), but you can only write extensions in Go and I'm not sure we want the extra maintenance burden of introducing the language to the repository. So probably not a good choice, sorry for the distraction.

blorente commented 7 months ago

1- I did not talk about file-watchers here. These are in another ticket.

Fair point, that was my mistake. I read this bit:

Check whether files were added or removed in the original file and update it accordingly

And assumed this would have been driven by a file watcher, which is not necessarily true. How did you imagine the update process being triggered? Just re-run the script?

3- Why do you think that it competes with Query Sync?

As far as I understand (from the outside, I haven't really touched the development of Query Sync), the QS experience is:

That experience sounds similar to what you're proposing, where essentially users opt into and out of targets via a UI.

blorente commented 7 months ago

I have no strong opinions on the collapsing or clicking of targets. As long as everything ends up being (sorted) plaintext and I can go back to the current UX, I'm okay with it :D.

There is something to be said about pruning directories. For instance, if we have the repo:

.
├── c
└── src
    ├── a
    └── b

And I want to add both src/a and src/b, I'd probably expect the plaintext project view file to contain:

directories:
    src

And then, in the UI, this could get expanded to toggles for each subdirectory.

Not a crucial feature, just an idea :D

ahmedyarub commented 7 months ago

Check whether files were added or removed in the original file and update it accordingly

And assumed this would have been driven by a file watcher, which is not necessarily true. How did you imagine the update process being triggered? Just re-run the script? Either from inside the plugin, or from CI/CD (which is how we are going to use it). Once you merge the changes into your local branch and the original projectview file is updated, a comparison can be made IF AND ONLY IF there is an import in the projectview file that you created. The goal here is two folds: 1- The (master) projectview file can be changed and committed. 2- We should not commit enabled/disabled directories and targets, that's why we have another projectview file that imports from the automatically updated one.

3- Why do you think that it competes with Query Sync?

As far as I understand (from the outside, I haven't really touched the development of Query Sync), the QS experience is:

  • We'll start by querying all your targets to know the sources but not syncing (building) any of them.
  • If you want to sync a target, you have to click a button in the IDE, and then we'll automagically add it to your project.

That experience sounds similar to what you're proposing, where essentially users opt into and out of targets via a UI. Oh nice this really complements what I want to achieve in the sense that I can use and or adapt your scripts (and I'm even planning to make the required fixes to make it work with huge repos because this great feature unfortunately does not work on ours); adapt them and reuse them as a part of my script. Great idea!

ahmedyarub commented 7 months ago

I have no strong opinions on the collapsing or clicking of targets. As long as everything ends up being (sorted) plaintext and I can go back to the current UX, I'm okay with it :D.

There is something to be said about pruning directories. For instance, if we have the repo:

.
├── c
└── src
    ├── a
    └── b

And I want to add both src/a and src/b, I'd probably expect the plaintext project view file to contain:

directories:
    src

And then, in the UI, this could get expanded to toggles for each subdirectory.

Not a crucial feature, just an idea :D

In our case IT IS a crucial feature because we have hundreds of folders and sub-folders, so there is an unquestionable need for my org to be able to both see a "brief" view of the parent folders, and have the ability to granularly enable/disable targets.

ahmedyarub commented 6 months ago

I have just started a discussion on expanding genquery(). I would really like to hear your opinion :)