Resolving "cyclic" dependency with ruamel.yaml.clib

[mbargull]

I gave the awkward "circular dependency" mentioned in https://github.com/conda-forge/ruamel.yaml.clib-feedstock/pull/3#issuecomment-549126205 a look this morning.

Here is what we've got:

• The Python package ruamel.yaml itself has an optional dependency on ruamel.yaml.clib: Guarded by :
  • try: ... from _ruamel_yaml import ... except: in ruamel.yaml.main
  • try: ... from .cyaml import ... except: in ruamel.yaml.__init__.py (ruamel.yaml.cyaml has the unconditional from _ruamel_yaml import ...)
• The distribution ruamel.yaml has a hard* dependency on ruamel.yaml.clib: It is in extras_require but with an empty "extra" plus platform specifier which makes it a platform-dependent hard dependency, AFAIK. (* except for python >=3.8 for which ruamel.yaml.clib is not built for PyPI)
• The Python module (or rather C/Cython extension ) _ruamel_yaml has a hard dependency on ruamel.yaml: Has a bunch of from ruamel.yaml... imports.
• The distribution ruamel.yaml.clib has no dependencies. Which means its actual content, i.e., _ruamel_yaml, is simply broken if one pip/conda installs ruamel.yaml.clib (:tada: ... /s).

So, to cleanly resolve this messy situation (and assuming upstream doesn't change things in the foreseeable future), we can create yet another ruamel.yaml.* package that hosts only the Python code for ruamel.yaml (since the Python package has only an optional dependency on the C extension, as noted above). My proposal for the ruamel packages is thus:

• ruamel:
  • no changes (i.e., empty ruamel/__init__.py and nothing else).
• ruamel.yaml:
  • convert to noarch: generic package which is itself empty and only has

    requirements:
    run:
    - ruamel.yaml.pylib {{ version }}
    - ruamel.yaml.clib >=0.1.2

• ruamel.yaml.clib:
  • add run dependency on ruamel.yaml.pylib >=0.16 (version compatibility is not specified upstream => TODO: raise issue upstream at https://bitbucket.org/ruamel/yaml.clib/issues)
• ruamel.yaml.pylib:
  • new package
  • we could add an awkward _, e.g., ruamel.yaml._pylib if we were to accommodate the possibility of upstream creating another ruamel.yaml.pylib package (personally, I'd just go for ruamel.yaml.pylib, TBH)
  • content is exactly that of the current ruamel.yaml
  • no requirement on ruamel.yaml.clib (breaking the cycle)
  • cannot be noarch: python just like the current package due to Python 2.7 support (https://bitbucket.org/ruamel/yaml/src/ff02b83b8f9127ca8e7fea3aaa15d0fa758dc0be/__init__.py#lines-17,18)

Now, a slight issue/decision remains: We could make the current recipe have multiple outputs and create the proposed ruamel.yaml and ruamel.yaml.pylib packages from this repo. However, this would introduce a dependency cycle not in the actual package builds but between ruamel.yaml-feedstock and ruamel.yaml.clib-feedstock:

• ruamel.yaml.clib-feedstock would depend on ruamel.yaml-feedstock due to ruamel.yaml.clib needing ruamel.yaml.pylib
• ruamel.yaml-feedstock would depend on ruamel.yaml.clib-feedstock due to ruamel.yaml needing ruamel.yaml.clib

So, to avoid this "conda(-forge)-build-system-only" dependency, we'd need to not use outputs here, but

1. Submit a new package ruamel.yaml.pylib to staged-recipes to create ruamel.yaml.pylib-feedstock.
2. Change this repo's recipe to be noarch: generic and depend on ruamel.yaml.pylib and ruamel.yaml.clib (Even though this package will be "empty", we should retain the source: to continue getting updates via the bot and stay in-sync with the new ruamel.yaml.pylib.)
3. Fix/update ruamel.yaml.clib's recipe to depend on ruamel.yaml.pylib.

Thoughts, @conda-forge/ruamel, @conda-forge/ruamel-yaml, @conda-forge/ruamel-yaml-clib, @jjhelmus, @mingwandroid, @msarahan?

[mbargull]

If we update/split the packages as outlined above, one case we have to take into account is this:

• ruamel.yaml=NEW: new build that depends on ruamel.yaml.pylib=NEW
• ruamel.yaml=OLD: old build whose payload is identical to ruamel.yaml.pylib=NEW

  
  # install ruamel.yaml=NEW and ruamel.yaml.pylib=NEW
  conda install ruamel.yaml=NEW

remove ruamel.yaml=NEW, leave ruamel.yaml.pylib=NEW in place, install ruamel.yaml=OLD

conda install ruamel.yaml=OLD

if conda only warns in case of clobbering, ruamel.yaml.pylib=NEW is now broken

remove ruamel.yaml.pylib=NEW, leave ruamel.yaml=OLD in place

conda remove ruamel.yaml.pylib

now ruamel.yaml=OLD is broken, too, i.e., import ruamel.yaml fails

Solutions:
1. Add a `constrains` requirements to `ruamel.yaml.pylib` for `ruamel.yaml>=NEW`:
   - This would introduce a dependency cycle in the `constrains`, which is not optimal.
     I know `conda` had issues with this at some point; not sure what the current state of this is.
   - `>=NEW` hits a limitation of `conda` if it does not indicate a version change; meaning:
     If we do not increase the version but only the build number, we are not able to express "either a new version or newer builds of the current version" with `conda`'s current capabilities.
- Add a `constrains` requirement on old `ruamel.yaml` builds:
  - metadata patching
  - can just do `constrain: ["ruamel.yaml.pylib<0a0" ]` to prevent `ruamel.yaml=OLD` and `ruame.yaml.pylib` from being simultaneously installed.

I'm generally no big fan of patching metadata. However, in this case it is the less troublesome solution and only affects `constrains` dependencies and "packages coming from the future" (from the PoV of `ruamel.yaml=OLD`) and thus should not change the consistency of current installations.

[mbargull]

If we do not increase the version but only the build number

We could lump the build into a "custom" conda-forge-specific version number, of course, meaning: have contrains: [ "ruamel.yaml>=0.16.5.0.post0" ] or something. This still leaves the "cycle in constrains dependencies" issue, though..