Make all Coherent System projects have a minimal `pyproject.toml`

[bswck]

This issue aims to point out problems resulting from not adhering to PEP 517/518, that is storing a minimal pyproject.toml file in every Coherent project:

[build-system]
requires = ['coherent.build']
build-backend = 'coherent.build'

At the moment, this file is only supplied temporarily in certain contexts and coherent.build has full control over its lifespan. I am thinking this isn't the most optimal approach, because:

1. Coherent projects aren't universally identifiable—there is no official way of identifying what projects use the essential layout. Since that layout is inseparably tied to coherent.build, it would be easy to figure that out by looking into build-system.build-backend. I think that assuming the presence of __init__.py or __main__.py is slightly short-sighted: this definitely isn't the essential layout.
2. Coherent projects don't play well with the existing installers—coh install makes assumptions about the preferred way of installing a project. I personally like to develop using virtual environments and that is one of the most popular practices, enabling inline hints and more. It should be possible to simply install Coherent projects using whatever method preferred: for instance, I want to develop 3 coherent projects in my super directory using editable installs.
3. Coherent projects aren't detected by third-party software as Python packages—while it could often be easily done just by seeing .py files in the root directory, this is often short-sighted too and might not be enough to prove a directory constituing a standalone package. For that reason Starship looks for specific files, with pyproject.toml being the common part of two enablers (https://starship.rs/config/#package-version, https://starship.rs/config/#python).

While the idea of temporarily supplying pyproject.toml allows to save some boilerplate, I still do think it comes with much more caveats than benefits. I'd like to suggest every Coherent project has a pyproject.toml so that it is just easier to work with.

Now that poses some questions, what about (meta)/. If we statically keep pyproject.toml for every Coherent project, shouldn't we rather let others customize ancillary details inside pyproject.toml and then, optionally, in (meta)/? Personally, I think that it would mean even easier migration to the Coherent System from flat/src layouts.

---

• [ ] Update docs to reflect initialization steps.
• [ ] (optional) Create a routine to initialize a repo with the boilerplate.
• [x] Apply changes to existing coherent projects (coherent.*, tempora@experiment/coherent).

[jaraco]

This issue has a lot of overlap with https://github.com/coherent-oss/system/issues/8.

I do agree, there are some serious downsides to the current experience.

I think you're right that as a practical matter, we should, at least for now, require that every coherent project include the boilerplate file.

How should we go about implementing the change? I can think of at least a couple places that need to be updated - in the getting started guide, in the builder readme.

Probably now we also need a command to write the boilerplate, no? Something like coh init (🤮) (which would also set up things like the GitHub actions). Or should coherent system projects have a skeleton to manage these common concerns? My instinct is no, but if these boilerplate files start changing at the system level, it's going to quickly become untenable to manage them through scripts and manual operations.

Now that poses some questions, what about (meta)/. If we statically keep pyproject.toml for every Coherent project, shouldn't we rather let others customize ancillary details inside pyproject.toml and then, optionally, in (meta)/?

Yes, possibly. We can consider those concerns later.

Are you willing to work on the effort? I'll add some subtasks in the original description.

[jaraco]

What I'm considering out-of-scope for now:

• Removing the logic that provides pyproject.toml on-demand. That logic will give precedence to an extant pyproject.toml, so let's rely on that for now.

[bswck]

Yes. It's been tempting for a while now. Always happy to remove code.

[bswck]

Are you willing to work on the effort? I'll add some subtasks in the original description.

Thanks, sure!

[jaraco]

I made a change to the coherent.build github workflow in order to allow it to be manually triggered. I'm trying to decide how to keep that in sync with the other projects... and I'm quickly starting to feel like the skeleton approach is needed again. After all, how do I keep track of which projects have gotten which updates (to the boilerplate).

Even if we have a coh init or a coh update-boilerplate, it's going to create new commits in each repo, each with unique hashes and intermediate results. That seems undesirable. I know we don't plan to update these files, but when we're forced to do so, we'll want a way to reliably apply the changes to each coherent project. Is there a better way?

[bswck]

I made a change to the coherent.build github workflow in order to allow it to be manually triggered.

Why was it necessary?

[bswck]

When it comes to managing boilerplate--maybe we could somehow not store the boilerplate in repos and force coh to manage boilerplate via file links locally? That would require some coh routine to be run after every checkout.

[jaraco]

I made a change to the coherent.build github workflow in order to allow it to be manually triggered.

Why was it necessary?

Well, it wasn't strictly necessary, but I did want to have the ability to trigger workflows when something changes in the system (such as today, where I added a workaround for the pipx issue, then wanted to run the workflows on the coherent projects with the change). So I added https://github.com/coherent-oss/coherent.build/commit/a95e65df11c86658a689a7b7f5f6626321802f7e to coherent.build, but I'd like that change to apply to every workflow. I'm 90% sure, though I'd like to be proven wrong, that change needs to be present in every repo.

When it comes to managing boilerplate--maybe we could somehow not store the boilerplate in repos and force coh to manage boilerplate via file links locally? That would require some coh routine to be run after every checkout.

In the case of github workflows, I don't think we have that option.

There are a handful of other people with the same desire not to manage common config across every project.

Another option could be to find a CI solution other than GitHub actions that honors coherent principles, but I'm not hopeful.

[bswck]

Well, it wasn't strictly necessary, but I did want to have the ability to trigger workflows when something changes in the system

I see. I used to re-run workflows ("Re-run all jobs") of HEAD in that case. But I can see workflow_dispatch being helpful.

In the case of github workflows, I don't think we have that option.

Oh, true, an obvious unsolved problem in my suggestion.

Well, that's an open problem then.