Plan out the stucture of the repository, cad, and documentation source and how this creates the final documentation

[julianstirling]

As there are a good number of comments here I am leaving this issue where it is. But it is in the wrong repository. It should be in Wakoma/nimble!

Sorry

[julianstirling]

The repository structure is still somewhat evolving. The biggest issue here is python package management and imports.

Relative python imports cause a number of weird issues, it is much more reliable to create packages that can be pip installed. But due to the other assets needed this is still somewhat problematic unless we include in place, and because the scripts we call are not packaged.

This structure is likely to evolve as the server becomes more contained.

[julianstirling]

In the rush to get things building and serving for Re:Publica I pushed forward with a somewhat-odd python module structure where we pip install in place (i.e. with -e) and a lot of things rely on local paths. We also have really have 3 packages in one (the server, the orchestration, and the shelf code). I think it is time to detangle this before the knots I created become the structure that we build upon evermore.

Long term we want this workflow to be available to any modular project, not just to Nimble. For this reason I suggest we split the repo in two repositories.

Repo 1: Nimble (the main nimble repo) Keep under CERN OHL-S

This will still contain:

• The CadQuery scripts that make the shelves/assemblies
• The base GitBuilding docs
• The shelf builder python module
• Nimble specific configurations for orchestration
• Nimble specific instructions for the server (i.e. the CSS, the wording, the configuration form)

Repo 2: Modular orchestrator Relicense to GPL, or AGPL or something

This will become a stand alone python module which has a server, and handles orchestration. For now, as Nimble is the only project using it, it will be somewhat Nimble focussed; however, we should endeavour to have any nimble-specific information in the Nimble. This will contain

• Utilities such as the Yaml Cleaner, or ExSourceDefinition writer
• The underlying server code. - This server code should look for a configuration file that tells it which questions to ask, the website text, logo, CSS, etc.
• The a generic orchestration module, containing the generic steps of "generate component cad", "generate assembly renders", "generate assembly CAD", "generate documentation".
• Entrypoints for running the server.

---

Suggested workflow for using orchestrator in Nimble or any future project.

• Create your CAD scripts
• Create your static GitBuilding files
• pip install orchestrator
• Create config file that explains component types and documentation steps/templates for modular templates - Temporarily this may need to be done via child classes of some of the orchestrator classes
• Create config file that explains what the web-configurator should ask, and how it should look
• run orchestrator serve

I think that even if the orchestrator starts of being Nimble focussed, it gives a path we can show NLNet for how it will be separable and reusable in the future.

---

Other questions:

@jmwright @drayde @ericnitschke Would be great to know your thoughts on this. Specifically to the following questions:

1. Are you happy for me to implement this?
2. Do you have a better name for the orchestrator repository/python module than orchestrator
3. Where does the orchestration repo live? I think it may make sense to live within the "GitBuilding" organisation on GitLab, as the orchestrator will be tied to both ExSource and GitBuilding which already sit under this organisation. I would of course make people team members of this project so that you can continue dev. I am not trying to grab ownership the orchestration project. I am just thinking about where it feels like it belongs. I'm happy to put it elsewhere.

[jmwright]

I had expected the orchestration layer and the server to be in separate repos, especially since you do pip install orchestrator. Is it just the Nimble UI that will be in a separate repository, and this "server" is just an API interface to the orchestration script?

[julianstirling]

I suppose my thought was that a generic orchestrator will always need a server to provide the interface to use it? We can then provide ways that projects can style and add content/options to their server?

[jmwright]

I had been thinking that the orchestrator was just a CLI utility, and that the server was added as a layer on top of that. However, if the main/only mode of interacting with the orchestrator will be through the server layer, then the server could be more tightly coupled to it.

If Repo 1 is structured in such a way that it can be cloned inside of Repo 2 (submodule or whatever) and could be swapped out for another project, I expect that would go a long way towards the NLnet goal of making this work applicable to other open hardware projects.

[julianstirling]

My thoughts exactly. In fact I'm hoping even easier than cloning/submoduling. I am hoping that Repo 1, just pip installs Repo2, and now has an server interface for orchestration (in the same way it can pip install gitbuilding to get a local "server" interface for modifying the docs). If @smamudhan can then use is containerisation skills to make a generic container for Repo two it should be easy for people to deploy.

[drayde]

I too was assuming orchestrator would not have any server code. There are use cases, where having a server with the orchestrator does not make sense, like using it in a build pipeline. If we want to do a general purpose (not Nimble specific) configuration server interface to orchestrator, maybe have it in yet another repo?

[julianstirling]

It is fine to use the orchestration without ever starting up the server. It is just that the server can never run without the orchestration, as it needs to directly interact with the orchestration. Therefore developing them in two repositories is a unnecessary overhead. If the server dependencies are deemed to large we can always just have it be

pip install orchestrator-package

for no server. But if you want the server: pip install orchestrator-package[server]

[julianstirling]

Closed by https://github.com/Wakoma/nimble/pull/84