Reorganize per-workflow settings file structure

Summary

Move all per-workflow settings to a special folder called .bonsai placed next to the workflow files. Proposed example structure:

📦ProjectFolder
 ┣ 📂.bonsai
 ┃ ┣ 📂Workflow1
 ┃ ┃ ┣ 📜.editor
 ┃ ┃ ┗ 📜.layout
 ┃ ┗ 📂Workflow2
 ┃ ┃ ┣ 📜.editor
 ┃ ┃ ┗ 📜.layout
 ┣ 📜Workflow1.bonsai
 ┗ 📜Workflow2.bonsai

Motivation

The work in #1870 has motivated the need to persist additional editor settings on a per-workflow basis (e.g. the explorer tree view state). Saving these passive editor settings in the .layout file would be convoluted since these two pieces of state are completely unrelated. Furthermore, it would introduce additional dependencies on the layout file which we are trying to move away from.

Adopting this proposal would help declutter the project folders, and explicitly de-emphasize versioning of .layout files. Suggestions for minimizing the impact of this change in current development workflows are presented below.

Detailed Design

This proposal introduces a new .bonsai folder (not to be confused with the local environment folder) in the project base directory which will be used to store any per-workflow configuration or passive editor settings. The .bonsai folder is organized into sub-folders, one for each workflow in the parent directory, where the name of each folder is the name of the corresponding workflow file without extension (e.g. the Workflow1 folder targets the Workflow1.bonsai file in the project base directory).

Inside each workflow folder, configuration files are saved as dotfiles with specific extensions depending on the type of recorded configuration state (e.g. .layout instead of the current Workflow.bonsai.layout; .editor for the new explorer tree view state).

Using only dotfiles will allow more convenient retargeting of these folders in case the linked workflow file needs to be renamed.

Backwards compatibility

To facilitate transitioning away from the .bonsai.layout files, the editor would keep recognizing existing .bonsai.layout files and convert them to the new structure when saving the workflow. We could delete any existing old files when saving the new structure to encourage migration, or otherwise require legacy files to be removed manually.

Drawbacks

Application UI startup

The main drawback of this proposal is that .bonsai.layout files are quite often used in application deployment, as they are currently the only option to specify the initial visualizer state for the application at startup time. This is crucial in experiment automation where UI needs to launch with the workflow. Further isolating the .bonsai.layout files from the main workflow runs the risk of introducing confusion, especially if we double down on the intent to ignore versioning of these files.

One option to address this drawback is to introduce a new operator for workflows (e.g. LaunchVisualizer) which could be used to explicitly mark which visualizers should be launched and/or made available on startup. This would help more clearly delineate which visualizers are part of the application, and avoid confusing environment-dependent visualizer window state (e.g. position and location) with application logic.

Explicitly launched visualizers would continue to save their environment-specific state in the .layout file.

Bootstrapper environment folders

A minor drawback is confusion with the .bonsai environment folder which is used as the default name of the portable environment template. We could either rename one of these folders or ignore the current name clash and allow them to mix together if they happen to coexist in the same base directory (this should mostly be fine except for potentially problematic name clashes if workflows have interesting names such as Packages.bonsai, Extensions.bonsai or Gallery.bonsai).

Alternatives

Save editor state system-wide or in the bootstrapper folder

Per-workflow editor state could be saved in the bootstrapper folder, next to the Bonsai.exe.settings file, either in a folder or files nearby (e.g. similar to the cache files in Bonsai.exe.WebView2 storing Chromium state for the documentation and annotation control). The internal structure would be similar to the one described in this proposal, and these additional settings files could be saved either per-environment or system-wide in the roaming AppData folder.

Similar to other applications we could compute deterministic pseudo-identifiers from the hash of the absolute file path and use them to identify the corresponding per-workflow config folder.

The main drawback of a centralized location is lack of transparency and making it harder to associate these state files with the workflow itself. Moving or renaming the location of the workflow file would automatically invalidate the corresponding config files and it would not be immediately obvious how to recover the link, especially if hashing is used to identify folders.

Introduce additional files next to the workflow

Alternatively, we could create additional .bonsai.* files next to the workflow, e.g. Workflow.bonsai.editor. The drawbacks would be further cluttering of the project folder, harder to rename all config files in one operation, and overall having a more confusing structure for version control systems.

Unresolved Questions

Flesh out details of the new launch visualizer operator
- Name: ShowVisualizer, LaunchVisualizer, Visualize, VisualizerLauncher
- Namespace: Bonsai.Design
- Proposed Properties:
- Visible
- DefaultBounds
- DefaultWindowState

I propose using a single ".bonsai" folder to house both environment and config files, with a dedicated subfolder inside called "cache" or "config" for storing workflow-specific settings. Each workflow-specific folder can be named using a hash of the absolute path, and we can have an option in the menu/welcome page to clear cache. I like this design because it keeps the bonsai files organized and it allows for developing hierarchical user preferences down the road, similar to how .vscode and .git/config work.

Additionally, by defaulting to local caching, this could reduce friction for new users who otherwise have to explicitly manage multiple files per workflow. I don't see a strong reason to have the .editor files be user-facing, though maintaining user agency over the .layout files is important. We could add a feature to export the .layout files similar to the export workflow mechanism, or include a setting to save .layout files when saving the workflow.

I propose using a single ".bonsai" folder to house both environment and config files, with a dedicated subfolder inside called "cache" or "config" for storing workflow-specific settings. Each workflow-specific folder can be named using a hash of the absolute path, and we can have an option in the menu/welcome page to clear cache.

This seems equivalent to alternative 1 discussed in the above proposal. The reason I am not a big fan of this option is that using a hash of the absolute path makes projects much less portable: simply moving a folder around in your hard-drive, renaming your top-level user "projects" folder, or checking out a project through one-drive will immediately break and invalidate the file association and you will lose all your workspace settings. While this may be fine for explorer tree view state, I am less keen of doing this on .layout files right now.

There are two main advantages I see currently with this option and I will try to clarify for each of them why I don't see it as valuable enough to overcome the current proposal.

Allow multiple users or environments with different settings to operate on the same workflow files

This to me is the biggest advantage of this proposal. By not tying the local editor settings to the project files, we would allow different editors with different stored settings to provide different views of the exact same project files.

As nice as this may sound conceptually, in a world where distributed version control and multi-user systems are the norm, I find this advantage to be more marginal than might first appear. Most likely different users will have different physical folders to begin with, and workflow files tend to be very minimal in size, so most often I have seen people simply copy project folders around when they want to work isolated without changing someone else's workspace. I've seen this pattern of usage even if they do not use version control, or even when working on single-login shared workstations. Even if you use version control, this doesn't help much when you switch between branches anyway, since even though the links may have been preserved in that case, the contents could be entirely scrambled.

Furthermore, even in cases where we do open the same project with a different version of the Bonsai editor I still see some advantages of the current proposal to keep project-level settings close to the source files . For example, we will often prototype pre-release versions of the editor by using source builds to open existing large projects from source, and it is nice to be able to open the project structure as it was prepared beforehand, especially for the more complex projects.

Centralized location for workflow settings and layout files

The second advantage of this proposal is that it really declutters the project folder structure, since everything but the workflow .bonsai files themselves are removed. It also eliminates possible clashes between co-existing .bonsai folders, for example in projects where .bonsai is used as the name for the local bootstrapper environment.

However, there are two main disadvantages that come with it that to me even out the possible gains.

Lack of transparency: the .layout and other editor files will be hidden inside a folder in the bootstrapper, which will make it really hard for them to be found by people after the change. This is especially true if we hash the folder names, and it might make it seem as if the new editor version is simply deleting all the .layout files unless we advertise what we are doing really well. Because of this, we will have the same main drawback of the current proposal in that we will still need an explicit solution to launch visualizers from the workflow, but without making it possible to easily understand and discover where the files are. It also makes it harder to manually reset the cache (we would definitely need an explicit menu for this if we go with this option, whereas in the current proposal we could defer it to the user for a while).
Encourage environment sharing: because this solution centralizes work patterns and settings, I feel it would encourage people to share more their environment across multiple projects for convenience of keeping their settings intact, whereas we have been trying to push everyone to make independent environments for every project to emphasize reproducibility of experimental workflows.

I like this design because it keeps the bonsai files organized and it allows for developing hierarchical user preferences down the road, similar to how .vscode and .git/config work.

Just wanted to clarify here that the current proposal is about managing per-workflow settings in specific projects and not about managing all editor settings in general. I am definitely with you and agree that editor-wide settings which are not tied to the specific workflows should live close to the local bootstrapper environment, possibly using a hierarchical structure similar to vscode and git config which you mentioned. My issues with this approach begin only once we start to have an interaction between the environment and the local project contents.

I don't see a strong reason to have the .editor files be user-facing, though maintaining user agency over the .layout files is important. We could add a feature to export the .layout files similar to the export workflow mechanism, or include a setting to save .layout files when saving the workflow.

In terms of managing these folders this will be handled transparently by the editor, and I am currently more inclined to not having .layout files be user-facing at all. I think we should provide explicitly in the workflow mechanisms to address the cases where workflow developers have traditionally abused .layout files to declare some kind of UI or monitoring layer for their applications.

It should be possible to have a fully functional workflow even if all .layout files were to be entirely removed. This to me is the biggest challenge with any of the proposals and definitely something I would like to have solved for the 2.9 release.

A minor drawback is confusion with the .bonsai environment folder which is used as the default name of the portable environment template.

@ncguilbeault I am realizing now that perhaps your proposal was trying to address this limitation. Indeed one way to workaround this issue might be to introduce an extra level of indirection, e.g.:

📦ProjectFolder
 ┣ 📂.bonsai
 ┃ ┗ 📂.config
 ┃ ┃ ┣ 📂Workflow1
 ┃ ┃ ┃ ┣ 📜.editor
 ┃ ┃ ┃ ┗ 📜.layout
 ┃ ┃ ┗ 📂Workflow2
 ┃ ┃ ┃ ┣ 📜.editor
 ┃ ┃ ┃ ┗ 📜.layout
 ┣ 📜Workflow1.bonsai
 ┗ 📜Workflow2.bonsai

By introducing a known interim folder name which we control, it would be easier to get .bonsai environments and .bonsai config folders to coexist under the same base folder. Still not as clean as the original proposal in my opinion, but maybe with the right names it could be worth it.

I will leave the points above as reference for discussion anyway.

The directory structure proposed in OP makes sense to me. I will just leave a few notes.

A minor drawback is confusion with the .bonsai environment folder which is used as the default name of the portable environment template.

I am not sure if this is true. Is there a "de facto" standard? We started off simply using bonsai as the name of the bootsrapper folder (Which i am still using) and at some point some repos started using .bonsai. I understand how it may be appealing to add the "." since the folder is supposed to be an ephemeral environment but, as opposed to python's venv, the bootstrapper folder actually contains dependencies for the bootstrapper (E.g. bonsai.config) that for venv are kept in a separate directory. This is all to say that the problem may not be the name of this settings folder but instead the name of the folder of the default environment.

Application UI startup

I am quite optimistic about this one. The only way to explicitly pass a .layout is to use the CLI. In mind, this creates the following scenarios:

Default case (there is a workflow.bonsai.layout next to workflow.bonsai) where the person opens the workflow file. The editor will default to the existing layout if it exists. If it does, update to the new proposed structure and everything will work the same.
Explicit case. Assuming the CLI still supports the visualizer flag, open the workflow with the target layout, and ignore (or attempt to recover the editor state only from the .bonsai folder?) the .bonsai folder settings. This mode should not override the visualizer inside .bonsai, as it is currently the case regarding the default layout file.
Explicit state + modifications. This is the trickier one. In my mind, it boils down to what is the role of this new folder. If you see it as a purely temporary folder for non-user-facing cache (my preference since it would allow excluding it from git without excluding a wildcard bonsai.layout pattern), then I would suggest allowing people to explicitly export layout files from the editor using some sort of dialog. This would be enough to afford modifications/creation of custom layouts. It would then be up to the user where to save them but as these are explicitly created I think that's a fair ask as they may require versioning.

I understand how this proposal might seem incompatible with :

In terms of managing these folders this will be handled transparently by the editor, and I am currently more inclined to not having .layout files be user-facing at all. I think we should provide explicitly in the workflow mechanisms to address the cases where workflow developers have traditionally abused .layout files to declare some kind of UI or monitoring layer for their applications. It should be possible to have a fully functional workflow even if all .layout files were to be entirely removed. This to me is the biggest challenge with any of the proposals and definitely something I would like to have solved for the 2.9 release.

However, I think it honors the idea that workflow stands on its own (e.g. default state) but is also able to run with an explicitly defined layout should the developer's application define it (wether via CLI or an eventual load layout file dialog).

One option to address this drawback is to introduce a new operator for workflows (e.g. LaunchVisualizer) which could be used to explicitly mark which visualizers should be launched and/or made available on startup

I like this (it would also solve #1862 as a bonus :P), but it seems like it would greatly increase the verbosity of the language and, once again, move visualizers away from being debugging tools. I also don't think it would necessarily solve the problem. People would still need a way to create the layouts (in this case add the operators, set their state, move the windows around and save a layout, since the layout is still responsible for keeping the position of the windows for instance). I may be missing something but this creates a situation where the state of the visualizers will still need a layout file by default to regenerate the state (which btw feels a bit weird as now the workflow operator properties depend on the layout file).

Once again, I think that allowing users to explicitly export the bonsai.layout files would be sufficient here.

the problem may not be the name of this settings folder but instead the name of the folder of the default environment.

This is a good point, and indeed my preference would be to keep a different name for the default environment. I'm not sure I like the idea of having the default names be the same with the only difference being a single . character, but another option could be to go back on the idea of a default name and just require that people always pass in a name.

I think that allowing users to explicitly export the bonsai.layout files would be sufficient here.

This is a good idea, and actually fits in really well with the new "compressed" form for .layout files which will be introduced in #1870. It is still index-based so not perfect yet, but at least as long as you know your indices you can have a .layout file that has just the windows you want to launch and nothing else, and I also like the idea that those files you version separately from the .bonsai folder which would be more for debugging and editing.

bonsai-rx / bonsai