search

cylc / cylc-admin

Project planning for the Cylc Workflow Engine.
https://cylc.github.io/cylc-admin/
GNU General Public License v3.0
5 stars 13 forks source link

meta packages #130

Open oliver-sanders opened 3 years ago

oliver-sanders commented 3 years ago

closes: https://github.com/cylc/cylc-admin/issues/76, https://github.com/cylc/cylc-admin/issues/27, https://github.com/cylc/cylc-admin/issues/75, https://github.com/cylc/cylc-flow/pull/4158

Open proposal to discuss the packaging-end-game for Cylc 8.

Doesn't really need a full-blown proposal-document, there isn't any fine print to haggle over.

cylc => cylc-flow

Early in the Cylc 8 project we renamed cylc => cylc-flow, some of the motivation:

In retrospect this might not have been the best idea:

The meta-package

The intention of the meta package was to provide a one-click install for a fully-functional Cylc stack.

# i.e. this:
$ conda install cylc

# does the same as this:
$ conda install cylc-flow cylc-uiserver cylc-rose metomi-rose

Advantages:

  1. The versions of each component get "pinned" in the meta-package ensuring environment consistency.

    • However, we don't actually need a meta-package to enforce this consistency (1).
    • For example cylc-rose has both metomi-rose and cylc-flow as dependencies.
    • This means that if you install cylc-rose your package manager (pip, conda) will ensure the two are compatible.

  2. Ease of installation.

    • Seemed like a good idea at the time, however, environments may have lots of variables.
    • What bits of the system do they want installed in which places?
    • What if they want to set the version of Python? Wouldn't we want to fix that in the meta-package?
    • What if they want (or don't want) optional Cylc dependencies (e.g. Pandas used for cylc-report-timings)?
    • What if they want to install the UI Server with the optional "hub" dependency.
    • What if they want to install the Hub without the "configurable-http-proxy".
    • Handling the matrix of "Cylc packages", "Cylc package installation options", "Cylc package dependency installation options" gets really, really messy.

Disadvantages:

  1. We can't "deselect" packages to install (with conda, pypi is different).

    • With Conda you can have multiple outputs, however, these aren't like optional-dependencies in PyPi.
    • They are more like top-level projects. E.G. both cPython and PyPy produce the python output.
    • So at the moment if you conda install cylc you get the whole lot whether you wanted it or not.
    • To get around this we would need special meta outputs e.g. cylc-meta-flow, cylc-meta-uiserver, etc.

  2. Differing versions between cylc and cylc-flow.

    • We would expect the meta-package version to start out as the cylc-flow version?
    • If we want to make a maintenance release for cylc-uiserver we would need to bump the meta-package maintenance version?
    • So the meta-package and cylc-flow versions can mismatch.
    • Unless we mandate that there is a cylc-flow maintenance version for each meta maintenance release?
    • Otherwise cylc version shows the version of cylc-flow not the meta package which is confusing if you installed via the meta package.
    • Where does the meta-package version get shown?

And so the meta-package has begun to look like a [complex] solution in need of a problem.

We would probably be better off adding a few example Conda recipes in cylc-doc and letting users decide how they want to install the stack.

The Future

Consistency

Currently all Cylc projects have a dependency on Cylc Flow.

We can give this dependency a very narrow version range (version "pinning") in order to ensure consistency between the different components.

Terminology: <major>.<minor>.<maintenance>

Packages

There should be one-- and preferably only one --obvious way to do it.

If the meta-package serves little function we should probably cut our losses and get rid of it.

Doing so would leave us with three problems:

Options:

  1. We could delete the cylc packages (makes us vulnerable to typo-squatting)
  2. We could make them dead-end packages (i.e. if you try to install them you get an error telling you not to install them).
    • I think you can do this on PyPi.
    • I have asked the conda-forge people, so far they have said that typosquatting protection comes from the review process...
    • ... This is fair, however, would this happen in practice?
  3. We could retire the cylc-flow packages and use these instead.

Related: Should we claim the namespace cycl (and possibly clyc?) since it is such a common typo on pypi/conda-forge.

I think:

Proposal

Ok, enough background, the actual proposal is quite simple:

1) Consistency

Make Cylc projects depend on minor releases of cylc-flow.

Examples:

2) Namespace

Go for option (2), we don't need a special meta-package, try to make the existing cylc packages "dead-ends".

3) WIP

Close:
* https://github.com/cylc/cylc-flow/pull/4158
* https://github.com/cylc/cylc-doc/pull/234

Thoughts @cylc/core?

dpmatthews commented 3 years ago

Makes sense to me

hjoliver commented 3 years ago

Agreed :+1:

oliver-sanders commented 3 years ago

No one screamed, actioning.

kinow commented 3 years ago

Regarding the old cylc metapackage, I think we will have to:

hjoliver commented 3 years ago

Thanks for investigating that @kinow

oliver-sanders commented 3 years ago

Progress:

Version Pinning (pin to cylc-flow minor version):

Conda Forge Outputs:

Misc: