Lock files don't mesh with dynamic package version info

hynek commented 2 months ago

Since tox-uv just shipped lock files, I took a stab at moving attrs to a fully-locked dev environment.

Here's the experiment PR: https://github.com/python-attrs/attrs/pull/1349

A problem I've run into is that packages often use dynamic packaging data that is based on git metadata (setuptools-scm, hatch-vcs, etc). As such, attrs's package version changes after every commit, which allows us to upload to test PyPI and continually verify our package. See, for example, https://test.pypi.org/project/attrs/24.2.1.dev19/

Unfortunately the current project gets locked like this:

[[package]]
name = "attrs"
version = "24.2.1.dev20"
source = { editable = "." }

Which means it's outdated after each commit, because every commit increments the number behind dev.

Is there a way around this that I've missed? If not, could there be built one? 😇 AFAICT, this is currently the biggest blocker for FOSS use from uv's side. I'm also not sure what the point of that version lock is in the first place for the current, editable project? But that's a topic for another day. :)

❯ uv --version
uv 0.4.12 (2545bca69 2024-09-18)

sbidoul commented 2 months ago

@hynek have you seen tool.uv.cache-keys ?

hynek commented 2 months ago

I have while searching this bug tracker, but I couldn't find a way for it to affect uv.lock? Changing the value to cache-keys = [{ git = true }, { file = "pyproject.toml" }] at least doesn't do anything when running uv.lock.

From the documentation (and its name 🤓) I've gathered it's mostly about how caching, not about locking? And TBH it sounds to me as if adding git as a git key, I'm actually adding cache busting.

charliermarsh commented 1 month ago

I think you could pass --upgrade-package attrs to ensure that attrs is rebuilt and that the lockfile is updated. Or even set it in your configuration:

[tool.uv]
upgrade-package = ["attrs"]

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. I don't know how to solve this holistically. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

hynek commented 1 month ago

But there's sort of an infinite loop here, right? Since every time you commit the lockfile, it will be outdated. So you then re-lock, and commit again, etc. We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

Yeah that's what I meant in my last sentence, but I'm always assuming Chesterton's Fence. :) Setting it to dynamic seems the best way since it maps nicely to project.dynamic. Just dunno if it would be a problem that there's a case where there's an invalid version "number" in the field which would speak for omitting.

jkeifer commented 1 month ago

I too have this problem. I generally subscribe to the philosophy that I use a VCS for versioning, thus what is in the VSC repo shouldn't track it's own version because that's what I'm using the VCS for. Said another way, at best a version tracked within repo is imperfect for exactly the reasons described above about how you can't update a version from a commit without making another commit, i.e., a new version.

I'd love to understand the rationale behind storing this version in the lock file. Can anyone elaborate on that?

burgholzer commented 1 month ago

We are also facing a similar problem over at https://github.com/cda-tum/mqt-core/pull/706 where we use renovate to keep the uv lock file up to date, which would be pretty convenient in principle. However, currently it triggers a never ending chain of renovate update PRs because every update to the lock file adds a commit, which changes the setuptools-scm version, which causes another update PR to be triggered. (see https://github.com/cda-tum/mqt-core/pull/703, https://github.com/cda-tum/mqt-core/pull/704, https://github.com/cda-tum/mqt-core/pull/705, https://github.com/cda-tum/mqt-core/pull/706)

LordFckHelmchen commented 1 month ago

This problem is not only for dynamic versioning. We have a release-please workflow, which will create a PR for the next release. This will update the pyproject.toml version correctly but of course does not know about the uv.lock version. If there would be an option to omit the project version from the uv.lock it would be great, as I currently also don't understand why it needs to be kept in the lock-file itself.

david-waterworth commented 1 month ago

As I mentioned in ttps://github.com/astral-sh/uv/issues/7994 I usepython-semantic-release which behaves the same. The way I work around the infinite build loop is to configure my Jenkins script to skip commits that contains the python-semantic-release message - these commits include the changed pyproject.toml and __version__.py file, along with a generated CHANGELOG.md

As a workaround, I also added uv lock at the start of the build script and git add uv.lock at the end - this ensures that python-semantic-release commit includes the correctly versioned uv.lock file (I'm not sure yet if this causes other problems, i.e. updating other packages - ideally there would be a way of only updating the package version in uv.lock and nothing else.

alejandrodnm commented 2 weeks ago

I've just encountered this. We are also using release-please and uv in an internal project, after a release then the surprise is that the uv.lock was outdated.

Ours plans were to add uv to https://github.com/timescale/pgai but since we are already using release-please, we are hesitant due to this issue.

RazerM commented 1 week ago

While the setuptools-scm case seems difficult to solve, I think another common reason[^1] a dynamic version is used is when the version is statically defined but not in pyproject.toml. Typically as a __version__ attribute like in this example:

Create example project
```
uv init --lib mpypkg && cd mpypkg
```

Configure pyproject.toml to use a dynamic version

cat <<EOF > pyproject.toml
[build-system]
requires = ["setuptools"]
build-backend = "setuptools.build_meta"

[project]
name = "mpypkg"
requires-python = ">=3.12"
dynamic = ["version"]

[tool.uv]
cache-keys = [{ file = "pyproject.toml" }, { file = "src/mpypkg/__init__.py" }]

[tool.setuptools.dynamic]
version = { attr = "mpypkg.__version__" }

[tool.setuptools]
package-dir = { "" = "src" }

[tool.setuptools.packages.find]
where = ["src"]
EOF

Add __version__ attribute to __init__.py

cat <<EOF >> src/mpypkg/__init__.py
__version__ = "1.0"
EOF

Run uv lock

Change version number

sed -i.bak 's/__version__ = "1\.0"/__version__ = "1.1"/' src/mpypkg/__init__.py

Run uv lock

What I expected:

 version = 1
 requires-python = ">=3.12"

 [[package]]
 name = "mpypkg"
-version = "1.0"
+version = "1.1"
 source = { editable = "." }

What happened: uv.lock still has version = "1.0"

I've also tried uv lock --no-cache --refresh to no avail.

The version doesn't change until I do an unrelated uv add or something. This seems like a cache invalidation problem more than anything else, which I expected cache-keys to solve.

[^1]: sometimes for backward compatibility reasons or in my case I can't move version into pyproject.toml until some other tooling I use catches up.

❯ uv --version
uv 0.4.30 (61ed2a236 2024-11-04)

charliermarsh commented 1 week ago

We can fix this, but I still strongly recommend against setting the version dynamically like that.

charliermarsh commented 1 week ago

I've fixed the issue @RazerM pointed out in https://github.com/astral-sh/uv/pull/8867. I guess I'll leave this issue open though since it's more about the inherent incompatibility between lockfiles and VCS-based versioning.

cpascual commented 1 week ago

Hi @charliermarsh, you mention "inherent incompatibility of lock files and VCS-based versioning" , but you also mentioned:

We could omit versions for editables, maybe? Or even, write "dynamic" or something for the version, for packages that have dynamic versions?

To me the above workaround seems good enough, specially if it is controlled by an opt-in configuration (e.g. omit_lock_version=["mypackage",]).

Is there any fundamental problem with this approach?

cpascual commented 1 week ago

my use case is that I use setuptools-scm and the presence of commit-dependent version numbers in uv.lock is a frustrating cause of git merge conflicts preventing what otherwise would be an automated merge.

ceejatec commented 1 week ago

We can fix this, but I still strongly recommend against setting the version dynamically like that.

@charliermarsh Out of curiosity, why?

Coruscant11 commented 1 week ago

I just discovered this issue with the recent release 0.5.0. Using --locked when syncing the project makes everything fails. I have a dynamic version based on git tags, and when we release we tag our repository. So the lock file can be outdated without any code change.

I think excluding editable dependencies in the lock file could be a good workaround. Or we can also write "dynamic" as a version. By the way I will check what tools like poetry for example are doing in this use case. @charliermarsh Is there any news on this? In the meantime, we will pin our version to the 0.4.30.

By the way I would suggest to list it in the breaking changes of the release

Thanks !!!

Edit: Forgot to mention, I was wondering if you already have some implementation ideas, I totally volunteer to work on this if needed. I will try a draft pull-request on my side for fun

Stealthii commented 5 days ago

We can fix this, but I still strongly recommend against setting the version dynamically like that.

If references or opinion articles exist around this topic, they are few in number. I think a few on this issue would be interested in wider discussion around this. PEP 621 introduced toml metadata, and as of writing the official specification still allows version to be set dynamically, with no recommendation to the contrary.

It's a pretty standard paradigm (see hatch-vcs, setuptools-scm, poetry-dynamic-versioning) for single project repos where versioning, point release branches, development builds, etc. are all managed and controlled via VCS change control.

Whilst alternatives exist for separating static versions (such as 0.5.1 from a generated identifier ref like f399a5271 in uv 0.5.1 (f399a5271 2024-11-08)), there are some compelling reasons to use dynamic (such as dunamai) formed versions:

Unique reference of all release, post, dev, dirty distributions of Python packages
All tarball/wheel builds being uniquely available in an internal PyPI repository for further testing or triage
Rules around versioning and releases can be set by project owners irrespective of development language
No room for user error (such as forgetting to update a static version of 0.5.1 for the git tag v0.5.1 and publishing builds)
Exposing the version in test logs, headers, etc. automatically shows the exact build used, aiding debugging

My expectation for the project package that has a dynamic version (source = { editable = "." } specifically) would be simply not defining version in uv.lock for the local package (or providing a dynamic reference). This by nature isn't a lockable reference (especially when set by VCS signatures) whereas other uses of uv lock such as multi-project mono repos or static versions uphold the current behaviour.

It's worth noting that although version is defined dynamically in source control, builds of the package provide the version statically:

❯ ls -1 dist/*
dist/mypackage-0.5.0.dev5+g5d056c1-py3-none-any.whl
dist/mypackage-0.5.0.dev5+g5d056c1.tar.gz
❯ unzip -p dist/*.whl '*.dist-info/METADATA' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1
❯ tar -Oxf dist/*.tar.gz '*.egg-info/PKG-INFO' | grep -E '^Version: '
Version: 0.5.0.dev5+g5d056c1

charliermarsh commented 5 days ago

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

charliermarsh commented 5 days ago

...simply not defining version in uv.lock for the local package (or providing a dynamic reference)

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

Coruscant11 commented 5 days ago

Unfortunately it's not super simple -- it breaks the fundamental assumption that packages have versions, so we have to take that into account everywhere.

Yes this is what I saw this week-end when I was trying to contribute on this 🥲 But anyway, I had the opportunity to deep dive in the code base so it's fine. We will have to trust you on this 😄 (No need to worry at all then)

Thanks!

ceejatec commented 5 days ago

it breaks the fundamental assumption that packages have versions

Not exactly, I don't think. The package always DOES have a version. It's just that the version may change outside of uv's immediate notice. Perhaps it would be fine if uv lock and uv sync and similar commands re-assessed the version if it's marked dynamic in the pyproject.toml. That wouldn't handle cases where the dynamic version contains the git commit SHA in some form, but I suspect that's pretty much an intractable problem.

That said, I feel like a cleaner solution would be to omit the main package from uv.lock entirely. However I'm definitely not familiar with the implementation here, so I don't know what unforeseen consequences that might have.

hynek commented 5 days ago

If references or opinion articles exist around this topic, they are few in number.

This is just my personal opinion! Dynamic metadata makes a number of things more complicated than they need to be. I understand the use-case for scm versioning, but in my opinion using dynamic metadata to read a version from __init__.py is a really heavy hammer with the only benefit being that you get to avoid writing out a constant twice -- I was responding to that specific case, where I don't think the tradeoffs are very good.

By the way, this can be inverted and made more standard-compliant while doing so by putting something like this into your __init__.py:

def __getattr__(name: str) -> str:
    if name != "__version__":
        msg = f"module {__name__} has no attribute {name}"
        raise AttributeError(msg)

    from importlib.metadata import version

    return version("YOUR-PACKAGE")

You get static metadata with no duplication.

ninoseki commented 5 days ago

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

hatch-vcs: 2.2k files
poetry-dynamic-versioning: 1.6k files

roteiro commented 5 days ago

I have the same pain point and would like to share how many projects in the wild do Git based versioning for your information.

hatch-vcs: 2.2k files

poetry-dynamic-versioning: 1.6k files

add to that:

versioningit 420 files

jamesbraza commented 4 days ago

I think https://github.com/astral-sh/uv/pull/8867 released in uv==0.5.0 leads uv to correctly respect dynamic metadata, which now breaks setuptools-scm + pre-commit users using the uv-lock hook: https://github.com/astral-sh/uv-pre-commit/issues/25.

This is textbook Hyrum's Law 😆

To share on the SCM versioning debate, I use setuptools-scm as it's a nice way for an automated (no human error) workflow of GitHub tagged release to PyPI publish with matching version.

charliermarsh commented 4 days ago

Yeah, it seems like fundamentally the only way to make this work with setuptools-scm is to omit the version or otherwise mark it as dynamic. I don't think it's impossible, but it's not, like, an hour of work. I'm also fairly worried about how we would integrate this if PEP 751 gets approved -- the PEP doesn't make any carveouts for dynamic versions, and there hasn't been any discussion of it in the PEP thread. If we add a feature like this here, and it doesn't make it into the PEP, we'll have to remove it later.

apollo13 commented 3 days ago

I think #8867 released in uv==0.5.0 leads uv to correctly respect dynamic metadata, which now breaks setuptools-scm + pre-commit users using the uv-lock hook: astral-sh/uv-pre-commit#25.

Can confirm that our workflow (using hatchling as build backend with a dynamic version) breaks when trying to do uv sync --locked since uv 0.5. 0.4.30 is fine. Whether it is actually #8867 that broke it I cannot tell, but sounds plausible.

charliermarsh commented 3 days ago

It likely was https://github.com/astral-sh/uv/pull/8867 that led to the change, though sadly that really was a bugfix -- the existing behavior was wrong. Right now there just isn't a clear way to use scm-based versioning with lockfiles, but I'm thinking about it. My biggest hesitation to investing in it deeply is the aforementioned PEP 751. I don't want to do something that will become invalid once the standard exists.

dsully commented 3 days ago

Is @brettcannon aware of this problem? Should PEP-751 take this usage into account?

(I'm also not 100% sold we need a universal lockfile format. I believe the 715 discussion thread also mentions this. 🤷 )

charliermarsh commented 3 days ago

I've been deeply involved in the thread and don't think it's come up, but I'll post about it today.

(Edit: @hynek already did.)

apollo13 commented 3 days ago

(I'm also not 100% sold we need a universal lockfile format. I believe the 715 discussion thread also mentions this. 🤷 )

Having a universal lockfile seems better than trying to add support for 100 different lock file formats to every other SBOM tool out there :D

brettcannon commented 3 days ago

Having a universal lockfile seems better than trying to add support for 100 different lock file formats to every other SBOM tool out there :D

I'm going to ask that if @dsully or anyone else wants to express concern over standardizing lock files they bring it up on the PEP 751 discussion over on discuss.python.org and not here as I don't think the uv issue tracker is the best place to talk about PEPs that affect the wider Python packaging ecosystem 😉

hynek commented 3 days ago

For posterity: https://discuss.python.org/t/pep-751-now-with-graphs/69721/86

dsully commented 2 days ago

Will do @brettcannon, thanks! Though looks like Hynek got there first 😎

tomrutter commented 11 hours ago

Sorry to pipe in on a largely completed debate but I have one idea to put.

My issue here seems the same as many other users:

My python package uses automatic versioning based on git tag for fewer user errors and easier tracking of built artefacts.
The package version is included in lock file, which when committed changes commit tag and thus the version.
As part of CI, I check lock file is up to date (for reproducibility). This check fails because of the version change of the local workspace package.

Changing any one of these links could fix the issue. I think most of the discussion has been about 1 and 2. That is, either moving away from dynamic versioning or changing how lock files work for local packages.

However, a simpler resolution might be via step 3? If uv were to allow users to check that lock files are up to date apart from the versions of any current workspace packages that have dynamic versions then the current workflow would be able to continue. Effectively this returns to the workflow of the pre 0.5.0 uv, while still correctly updating the lock file (and re-installing the package). I would expect this would catch most of the current use cases for uv lock --frozen

This could work by adding an option to uv lock --frozen etc to allow versions of local packages with dynamic versioning to change but to error if any other dependencies change.

astral-sh / uv

Lock files don't mesh with dynamic package version info #7533