Closed zaneselvans closed 1 year ago
zaneselvans
commented
1 year ago @bollwyvl I'm not familiar with the mechanics of building multiple subpackages in a single recipe, but it seems like maybe something has gone a bit funny here with the mature vs. immature library version numbers that are associated with the conda packages.
bollwyvl
commented
1 year ago Yep, somewhere along the line (ca 0.15.8), the multi-outputs scheme got too clever for itself, and started shipping incorrect, but internally-consistent versions, even though it was able to continuously pull the correct upstream version tarballs, and users that pinned dagster 1.y.z were still getting dagster-* ?.y.z were still getting functional solves.
Unless this is some upstream conda-forge bug, probably the only way to fully confidently fix this, and still get some level of automation (critical at this scale, with interrelated pins, etc), is to split this feedstock into two, still multi-outputs feedstocks:
dagster-feedstock
dagsterdagster-integrations-feedstock (or extensions, or plugins, or whatever is appropriate)
dagster-* packagesBut: as the previous million downloaders haven't complained until now, we're not going to go back and mark all of those broken, and will have to deal with actual conflicts later down the line, e.g. when 1.0.2 rolls around. Hopefully, that will be never and all the versions will at least be monotonically increasing, such that there is never another new upstream dagster-*-1.0.2.tar.gz.
bollwyvl
commented
1 year ago Kicking tires here: https://github.com/conda-forge/staged-recipes/pull/24424
bollwyvl
commented
1 year ago All the linked PRs have been merged, closing.
If something new is broken/inconsistent, let's start on a new issue.
it's... unpredictable what will happen while both (1.5|0.21).6 can both solve, may have to see how x.y.7 works out.
zaneselvans
commented
1 year ago Thank you so much for fixing all this so quickly! I'll keep an eye on our lockfiles and see how it goes.
Solution to issue cannot be found in the documentation.
Issue
As I mentioned in this discussion, it appears that the immature library packages (currently v0.21.6) are getting assigned the mature package version (currently v1.5.6) somehow in the
conda-forgepackaging, which leads to a big version discrepancy between the PyPI andconda-forgeversions for the same package, e.g.dagster-postgres. Full text of the above discussion:We are using
conda-lockto create a reproducible environment that we run Dagster in. It reads dependencies and version constraints frompyproject.toml, solves the environment usingmamba, and produces a complete environment specification in a lockfile. We're in the process of switching to using Postgres instead of SQLite for the EventLogs, since SQLite has been locking up on us due to concurrency. However, when trying to add the new dependency I discovered that the PyPI andconda-forgeversioning for thedagster-postgrespackage seem to be completely unrelated to each other.conda-forgethe current version is 1.5.5Is it supposed to be this way? Typically conda packages track the same versions as their PyPI counterparts, and
dagster-postgresseems to be an official package. It looks like the0.21.6version is probably the "libs" version, while 1.5.5 is the overall Dagster version.Having these two versioning schemes be out of sync makes it difficult to specify the environment in a straightforward way, because the tools parsing
pyproject.tomlcould ultimately seek the package either from PyPI orconda-forge.I notice that this seems to be a common, but not universal, situation, across many of the
dagster-*packages. Searchingconda-forgepackages it looks like everything has a most recent version of 1.5.5:While many but not all of the same
dagster-*packages have the libs version 0.21.6 on PyPI.Looking at the conda-forge/dagster-feedstock which produces these packages, it seems like the intention is for the immature library packages to have the immature library version
lib_version(currently 0.21.6), but for some reason that's not working:It also seems like the intention in the documentation about library versioning
Looking at the conda build docs on multiple outputs it does seem like the
versionskey is supposed to allow each of the subpackages to have its own version, but for some reason that's clearly not happening.Installed packages
Environment info