Closed ksamuel closed 4 years ago
At the moment this is not possible no.
poetry version
will only bump the version
in pyproject.toml
. This is not intended and that's definitely something I plan on adding.
I really wish that the version
field in the .toml
ended up superseding the __version__
string in __init__.py
. This would reduce the burden of having to keep multiple places in the code in sync.
Setuptoos has solved this problem for 2 years:
https://setuptools.readthedocs.io/en/latest/setuptools.html#configuring-setup-using-setup-cfg-files
You can put all metadata in setup.cfg instead of setup.py. For the version, you can specify:
version = attr: src.__version__
Which will load the version from the package directly, this way you have it in one place only. It works as long as you has setuptools >= 30.3 (current version is 38.5.1) so basically any package with a setup.py can already benefit from that without using any additional tool.
Le 27/05/2018 à 07:28, Victor-Nicolae Savu a écrit :
I really wish that the |version| field in the |.toml| ended up superseding the |version| string in |init.py|. This would reduce the burden of having to keep multiple places in the code in sync.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/sdispater/poetry/issues/144#issuecomment-392306362, or mute the thread https://github.com/notifications/unsubscribe-auth/AANivj7pFmuXBG79h1VOEG9_SM0V4Tj1ks5t2jl7gaJpZM4UO8xi.
@ksamuel Thanks for pointing that out! I use that setuptools
feature as well :) It feels a bit backwards that it needs to be referenced in two places, but I suppose that is just my ignorance with respect to the benefits of having programmatic access to the version of a package. I can kind of imagine a scenario, but I have never seen a real-world example of where that was used (again, it's just my inexperience, I am not criticizing such a practice :) )
you may use that
import toml
from pathlib import Path
def get_version():
path = Path(__file__).resolve().parents[1] / 'pyproject.toml'
pyproject = toml.loads(open(str(path)).read())
return pyproject['tool']['poetry']['version']
__version__ = get_version()
Now that doesn't work when the file system is not available, or when the project file is not shiped.
Le 27/05/2018 à 20:21, jgirardet a écrit :
you may use that
|import toml from pathlib import Path def get_version(): path = Path(file).resolve().parents[1] / 'pyproject.toml' pyproject = toml.loads(open(str(path)).read()) return pyproject['tool']['poetry']['version'] version = get_version() |
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sdispater/poetry/issues/144#issuecomment-392352521, or mute the thread https://github.com/notifications/unsubscribe-auth/AANivtdMQKQ53fmYDl0RaC2Wh-ZviBhlks5t2u6TgaJpZM4UO8xi.
Personally I think storing a __version__
in __init__.py
seems duplicative. You can use pkg_resources
to retrieve the version for the installed package using:
pkg_resources.get_distribution("packagename").version
If used with:
poetry develop
This will work even when developing locally (it requires the egg-info).
It doesn't work with vendoring, or anything using sys.path.append. And it's way harder to type in an interpretter. Who remember that ?
Le 30/05/2018 à 19:54, Bert JW Regeer a écrit :
Personally I think storing a |version| in |init.py| seems duplicative. You can use |pkg_resources| to retrieve the version for the installed package using:
|pkg_resources.get_distribution("packagename").version |
If used with:
|poetry develop |
This will work even when developing locally (it requires the egg-info).
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/sdispater/poetry/issues/144#issuecomment-393257700, or mute the thread https://github.com/notifications/unsubscribe-auth/AANivn9NPZz_2OgTFG4OPozzcHa3zB_xks5t3tzvgaJpZM4UO8xi.
Using pkg_resources
also introduces an install-time dependency on setuptools, no?
Currently workaround can be to use bump2version
package (maintained fork of bumpversion
).
For my package poetry_demo
with default poetry_demo/__init__.py
and tests/test_poetry_demo.py
content I have following .bumpversion.cfg
file (use of setup.cfg
file name is also possible):
[bumpversion]
current_version = 0.1.0
commit = True
tag = True
[bumpversion:file:pyproject.toml]
search = version = "{current_version}"
replace = version = "{new_version}"
[bumpversion:file:poetry_demo/__init__.py]
search = __version__ = '{current_version}'
replace = __version__ = '{new_version}'
[bumpversion:file:tests/test_poetry_demo.py]
search = __version__ == '{current_version}'
replace = __version__ == '{new_version}'
To bump the version:
$ bump2version patch
and all version instances will become 0.1.1.
Provided configuration will also commit the modification and create tag. This (and many more features) are configurable.
A little problem is, that currently it requires extra configuration file .bumpversion.cfg
, but there is an issue https://github.com/c4urself/bump2version/issues/42 planning to use pyproject.toml
The tool seems rather mature, has many features and is used for quite some time.
To implement bumping versions in poetry, it may serve at least as nice inspiration.
Apart from poetry itself, __version__
bumping should be implemented in the scaffold project that is being created by poetry new
too. In that particular case, I think it would make sense to stick to the standard library instead of adding a toml depenedency, so something like this could be used:
import configparser
parser = configparser.ConfigParser()
parser.read("pyproject.toml")
__version__ = parser["tool.poetry"]["version"]
__all__ = ["__version__"]
One thing to keep in mind with the automatisms like that is that they will access the disk on importing the library. Code execution on import is not that uncommon in Python, but not desireable to keep down import times and minimize side effects.
Does the pyproject.toml file even ship with wheels?
@pmav99 you can't do that unless you also ship pyproject.toml
with your WHEEL for instance.
I also think this is something very important.
Another example is when you integrate it with a tool like Sentry, you are willing to know for which version it applies. Currently, I couldn't find a proper way to include it except writing a __version__
string in the __init__.py
Depending on your build process, one solution is to use a shell script for bumping the version instead of running poetry version
explicitly. I prefer using makefiles, so I added these two rules to mine to solve this problem.
poetryversion:
poetry version $(version)
version: poetryversion
$(eval NEW_VERS := $(shell cat pyproject.toml | grep "^version = \"*\"" | cut -d'"' -f2))
sed -i "" "s/__version__ = .*/__version__ = \"$(NEW_VERS)\"/g" x12_utils/__init__.py
This command can be run as follows: make version=1.2.3 version
If you dislike specifying the version(type) that way, this could just as easily be implemented as a shell script such as:
poetry version $1
new_vers=$(cat pyproject.toml | grep "^version = \"*\"" | cut -d'"' -f2)
sed -i "" "s/__version__ = .*/__version__ = \"${new_vers}\"/g" x12_utils/__init__.py
In my case reading from pyproject.toml
does not make sense as I ship a CLI and myapp version
has to be able to answer to this question without pyproject.toml
.
I would also like to see poetry version
bump the version in:
pyproject.toml
<package_name>/__init__.py
tests/*.py
# any tests which have a version literal in themSince pyproject.toml
is already handled, it seems something like using astor might be a good choice for safely modifying python source.
I've prototyped this feature and would like some feedback. I consider this an 80% solution that will make most people happy.
In addition to updating pyproject.toml
, the new command will update version strings in specific python contexts; specifically assignments and comparisons. It will also do a global replacement on text files with specific suffixes:
*.[mM]*[dD]*
for Markdown*.[rR][sS][tT]
for ReStructured Text*.[tT]*[xX][tT]
for plain text filesThe updated command syntax is:
poetry version <rule> [--dry-run]
@JnyJny If I understand correctly, your current approach for text files is just replacing the version string wherever it occurs in any file with a matching extension. I feel like that has a relatively high potential for false positives, e.g. if you list required versions for dependencies anywhere in your documentation.
I haven't dug into your implementation for python files at all but in principle I think it would be less likely to cause problems there (given you can analyze code more deeply than documentation, and code is less likely to include unrelated version strings).
Also, I think this would be a nice quality-of-life improvement even if it was only applied to __init__.py
s (and given the potential for false positives, that may be preferable anyway).
@dmontagu Thank you for the feedback!
Regarding text files, that was creeping-featurism on my part. I agree that it could result in false positives (change logs) or other instances where the modifications might not be welcome. I toyed with requesting a confirmation for each file from the user before a file is added to the change set, but this seemed clunky to me. Another way to solve the problem is command line options to selectively control which file types are modified:
poetry version prerelease [--update=text,source|--all]
Yet another solution would be to become more opinionated about the structure of the package, like version strings are only in __init__.py
or __version__.py
files and where version strings should appear in tests. Honestly, the python source code is easiest part of this problem.
Lastly we could grow pyproject.toml to include a list of specific files that will be updated with new version numbers:
[tool.poetry]
version_files = [<package>/__init__.py, tests/test_version.py, README.md ]
All of these things are doable with varying amounts of effort, but none of them herculean. My overall goal was to impose the least amount of opinion about how packages should be structured.
I utilized the latter strategy (looking for a list of defined version files in pyproject.toml
) when I created AutoPub, a tool that enables project maintainers to release new package versions to PyPI by merging pull requests. AutoPub needs to increment all relevant version numbers before publishing new packages to PyPI, and while I was hoping Poetry could handle this, I eventually solved it within AutoPub itself.
First, AutoPub looks for a tool.autopub.version-strings
key and creates a VERSION_STRINGS
list of the version file paths contained within, if any.
AutoPub defines an update_version_strings
function that increments version numbers. This function is used to update the relevant version string(s) if any version files are defined.
Perhaps these code samples could be useful when a solution is devised for Poetry. I would love to remove that code and rely on Poetry to comprehensively handle version string updates in all the relevant files.
The reasons I think this latter approach is preferred relative to others include the following:
Is this still relevant? If so, what is blocking it? Is there anything you can do to help move it forward?
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Still relevant, as per @sdispater comment:
poetry version
will only bump theversion
inpyproject.toml
. This is not intended and that's definitely something I plan on adding.
In fact, proper version management was one of two reasons we postponed using poetry and moved back to using pbr
. The other reason was usability with private repositories which we need in our continuous integration process.
I really like the workflow, where the version is assigned by adding tag to git repo, I even plan to adopt usage of reno for generating release notes.
I wonder, if similar workflow could be possible with poetry. Taking into account, that pyproject.toml includes explicit package version, this does not seem trivial. One idea is to allow version having alternative value git-tag
and in such a case to generate the value from git tag (the style pbr
does that seems working well).
I'm with @vlcinsky on this one. The best I've found so far to support git tag driven versions is https://pypi.org/project/setuptools-scm/.
poetry
is becoming the tool simplifying python package development.
However, to succeed in wider range, it must fit into automated build processes and this includes managing package versions.
Risk: When poetry fails to meet expectations (about version management) of a developer, we are likely to loose such user.
--version
flagMeeting all these requirements is in fact impossible. There will be always space for well defined and respected procedures in package version management. The solution shall try to make these procedures as simple as possible anyway.
Most packages use this approach.
Tools like bump2version provide commands to bump version using simple call updating content of arbitrary number of files. Sometime the tool may even attempt to commit and tag the changes.
Note: currently poetry supports bumping version in pyproject.toml
, so it partially falls into this category. Anyway, other features (edit content of other files) is not provided (yet).
Tools like pbr or setuptools_scm read version related information from commits and tags. When commit is tagged as v1.2.3, such package version is built. Following commits get incremented versions (using suffixes such as dev{distance}
etc).
Note: While there are multiple SCM systems (git, hg, …), some tools work only with git and this seems to be generally more and more acceptable decision.
Here are some options, poetry could take in regards to package version management.
Skip version command, leave user with their own choice of external tools such as bump2version
.
Bumping would update version in pyproject.toml
and in another file (like module/__init__.py
).
Adopt pbr
or setuptools_scm
approach and make it simpler.
To be flexible enough, use plugins (similarly as pytest
is using them) for versioning.
It would require poetry to:
pyproject.toml
and add information about (single) version bumping plugin to use, probably allowing also extra section for plugin specific configuration parameters.There are packages which can make plugin integration rather easy, such as pluggy.
First two options ("No version bumping support" and "Bump version on predefined locations") are not ideal, as most builds result in packages declaring the same version regardless of having different codebase. This could be partially solved by rule to bump to dev version after each release, but this is quite inflexible and not easy to follow.
I would be very happy with "Adopt commit/tag based versioning" option. Anyway, I am not sure, if such solutions seems enough acceptable to wider audience.
Last option ("Support poetry versioning plugins") seem to be the most flexible one. It adds some complexity to poetry source code, but on the other hand it could outsource version bumping effort to other parties and it also has the potential to "please everybody".
Proposed next actions:
anybody: provide feedback
I'll give you my (very opinionated) two cents.
My background is Java, but I recently find myself in a department of Python developers. For me, this is a massive culture shock - I assumed that a popular language like Python would have a mature ecosystem of tools, standard patterns for CI/CD, but that's not what I have found.
Nevermind; when life gives you lemons, make lemonade 🍋😄
The first problem I must solve is packaging, so I need:
Obviously those two systems must agree on how to canonically identify a package, i.e. a coordinate scheme for artifacts [ namespace, name, version ].
Repository options in Python seem limited to Artifactory and pypiserver
.
I infer that pypiserver
's coordinate system must be some de facto standard.
So, I must now find a package manager which can deploy to pypiserver
.
If you do some Googling, or read Hacker News, everyone is saying, "use Poetry, that's the project to watch".
So, I tried Poetry, and those people are correct - it's already far superior to the alternatives, so good job 👍
However, I immediately ran into this issue #144 , and this prevents me enforcing Poetry as the standard Python packaging tool across my company.
When poetry fails to meet expectations (about version management)... we are likely to lose such a user.
I am that user!
You have presented some options @vlcinsky very clearly, which I appreciate.
I understand that you are constrained by things outside of Poetry, you have compatibility concerns, and so on. However, your suggestions don't address the real problem in my opinion, which is this:
Only the package manager can define a package's version, and it must be canonical.
Whatever this __init__.py
nonsense is, get rid of it.
Yes, I know, this file has some semantics for the Python interpreter, and is necessary in certain circumstances.... I really don't care, that file has no business trying to define the version of my package, versioning is the responsibility of the package manager and no-one else.
Most of the suggestions in this thread are about incrementing a version, which is interesting, but futile if there isn't a single canonical version we can think about incrementing.
Repository options in Python seem limited to Artifactory and
pypiserver
. I infer thatpypiserver
's coordinate system must be some de facto standard. So, I must now find a package manager which can deploy topypiserver
.
@woolinsilver-kainos, if you're looking for a private python package index, I've had good experiences with devpi. While I'm waiting for a better tool, a regular setup.py
with setuptools_scm
in conjunction with pip-tools
has been the best combination for supporting a CI/CD workflow with version set through annotated git tags.
@woolinsilver-kainos interesting point about package metadata to be controlled out of code (in our case by git tags).
Btw: we use devpi (as @ironhouzi mentioned) and we are happy with it.
For packaging, we used pbr, then spent few weeks trying to adopt poetry into our CI/CD pipeline, even keeping our customized version, finally went back to pbr.
Anyway, in long term poetry is the tool we want to use.
I have the impression, the discussion here goes beyond the aim of poetry:
Poetry is a tool for dependency management and packaging in Python.
"Packaging" is the important word. Not CI/CD nor Deployment.
If you package your project, the version number is integrated into it. Then use importlib_metadata
(since python 3.8 as build-in importlib.metadata
) to access the version number during run time if needed. There is no need to have the version number in any additional file.
With poetry version
one have a cli to manipulate the version number in pyproject.toml
(which is used when building the package through poetry). So integrating this into a pipeline shouldn't be that hard, should it?
Just my two cents :)
I use poetry + bump2version. Bump2version updates the code, project files and tags appropriately. Works with travis CI. So I think the above taxonomy puts it in the wrong bucket. If poetry version just called bump2version I'd be fine with it, but wouldn't change the complexity of my release process, which stems from testing, building docs etc. I have longish checklist to go through. I wish poetry release captured that complexity, but I digress. Python packaging has to work without SCM, until core decides that SCM is an assumption that can be made. But one also need to tag versions in git, I think there is agreement on this, when using SCM (always). It's not DRY but resolving this is bigger than poetry. So we have version info in pyproject, init.py and git. At best, we can make one copy authoritative and discourage editing the others and have tools keep them in sync. bump2version does that. I would love a clarification on that statement by @vlcinsky that "following commits build into packages declaring the same version regardless having different codebase". I was wondering what level of granularity one wants in release numbers, since the next level is the commit level. We don't want releases to map one-to-one to commits, or do we? As far pointing to a single immutable codebase, that's what the tag is for. The number in pyproject though does not , that's correct. So if you are running off git instead of a released version, you may not know exactly at runtime what code you are running. But, if you are running off git, you are supposed to know that as opposed to a regular user who is running a package off pypi. Or am I missing something? Finally a comment on the scope of this issue. It started as a bug report about poetry version. @vlcinsky expanded to include the possibility of dropping poetry version completely and @finswimmer even questions the scope of poetry. I find both questions worth a discussion, but I also think we can't cram everything interesting in one issue and come to a conclusion. I was wondering if we need to update the title of this issue to recognize an expanded scope or opening a different issue for the big picture items.
following commits build into packages declaring the same version regardless having different codebase
I think this means that after a X.Y.Z version is bumped (and tagged, built, released), some tools do not follow up with another bump to X.Y.Z+1.dev, so without that the following commits still have X.Y.Z versions but contain different code.
Concerning the original issue, I wanted to point to the fact, that there is no need to write or bump the version in other files than pyproject.toml
.
@finswimmer, here are some examples when you need __version__
:
pyproject.toml
)__version__
to determine package version (__version__
is even mentioned in pep8, so everyone is expecting your package to have this variable.)Yeah, you don't need __version__
to create the package, but you need it to use the package (or at least it is good to have it).
__version__
is important, and packaging tools such as poetry must somehow take it into account.
@vanyakosmos But based on @finswimmer's comment, couldn't you just use this logic to achieve that?
import importlib # probably want a conditional for pre-3.8
__version__ = importlib.metadata("mypackage").version
As you said importlib.metadata
will work only in python3.8. And importlib_metadata
is an extra dependency for extracting single string that already should be in the package. So right now I don't see importlib.metadata
as the solution (unless some of your package dependencies already use importlib_metadata
).
I am right now checking out importlib_metadata in 3.7 (backport of importlib.metadata suggested by @dmontagu ) It seems to pick the contents of pyproject, not __init__.py
after a couple of tests.
The peps that describe metadata are 214, 314, 345 and 566. Pep 8 is about formatting, as we all know, and its mention of __version__
is cursory, in an example. 214 mentions two files, PKG_INFO and setup.py. I wonder where this use of putting some metadata in init.py comes from (I follow it too, without much thought; not questioning it's widespread). The optional pep396 (status: deferred) is the closest I could find. It states
The version attribute in a classic distutils setup.py file, or the PEP 345 [7] Version metadata field SHOULD be derived from the
__version__
field, or vice versa.
I have 2 cents to spend here as well:
I stopped a long time ago to write any metadata in __init__.py
. Others explained it or explain it better in other comments or blog articles: metadata has no place in the source code. I don't want any __author__
, __version__
or else in __init__.py
. This information:
pkg_resources
or importlib.metadata
.Maybe writing __version__
in __init__.py
was/is very common, but it's not a standard thing. In the end, I don't think poetry
should bother too much to support bumping __version__
in __init__.py
.
Solution to get version at runtime:
# utils.py
import pkg_resources
def get_version():
try:
distribution = pkg_resources.get_distribution("my_package")
except pkg_resources.DistributionNotFound:
return "dev" # or "", or None
# or try with importib.metadata
# or try reading pyproject.toml
else:
return distribution.version
Solution to get version when building Sphinx documentation:
# docs/conf.py
from pathlib import Path
import toml # yes, you'll need to add toml to your docs/requirements.txt
metadata = toml.load(Path(__file__).parent.parent / "pyproject.toml")["tool"]["poetry"]
version = release = metadata["version"]
# project = metadata["name"]
# etc.
Another take on extracting the version from the "pyproject.toml" file:
from tomlkit import toml_file
def _get_project_metadata():
project = toml_file.TOMLFile("pyproject.toml").read()
return project.get("tool", {}).get("poetry", {})
project_metadata = _get_project_metadata()
version = project_metadata.get("version")
The version
variable will beset to None
in case the file is broken.
Bonus: basic tests for it!
from unittest.mock import patch
import pytest
import tomlkit
from XYZ import _get_project_metadata
@patch("XYZ.toml_file")
def test_get_metadata(tomlfile_mock):
config = [
"[tool.poetry]",
"version = 'v-test'",
"",
"[tool.other]",
"ignored = true",
]
doc = tomlkit.parse("\r\n".join(config))
tomlfile_mock.TOMLFile.return_value.read.return_value = doc
metadata = _get_project_metadata()
assert metadata["version"] == "v-test"
tomlfile_mock.TOMLFile.assert_called_with("pyproject.toml")
tomlfile_mock.TOMLFile.return_value.read.assert_called_with()
@pytest.mark.parametrize(
"config", [[""], ["[tool]", "version = 'incomplete-table-name'"], ["[tool.poetry]", "version_missing = true"],]
)
@patch("XYZ.toml_file")
def test_get_metadata_with_invalid_config(tomlfile_mock, config):
doc = tomlkit.parse("\r\n".join(config))
tomlfile_mock.TOMLFile.return_value.read.return_value = doc
metadata = _get_project_metadata()
assert metadata.get("version") is None
Again, extracting from pyproject.toml doesn’t help at run-time, because the pyproject file is not installed.
Again, extracting from pyproject.toml doesn’t help at run-time, because the pyproject file is not installed.
Any solution that requires adding a python dependency to my project to read a simple version string is a non-solution. This includes setuptools
and especially pkg_resources
, which is very slow.
IMHO, version semantics and bumping and SCMs could all be considered secondary features. The most important thing is to ensure that the version in pyproject.toml
drives/generates the version found in mypackage.__version__
.
Let me summarize (not expressing my opinion, only my understanding of opinions above -- including mine; please correct any omissions or misunderstandings and I will edit accordingly). Specs:
Solutions (scm issue dealt with separately):
init.__version__
maintained by poetry: it's a duplication but it is managed. In case of conflict pyproject.toml wins, __version__
is overwritterninit.__version__
initialized at load time using importlib.metadata or alternate mechanism: checks the first three boxes As to the alternate mechanism to initialize __version__
, some people are wary of an additional import so here are three options
__init__
As to SCM:
version = {use_scm = true, scm=git}
or some suchSolution 3 above checks all the boxes for me, particularly coupled with a choice of alternate mechanisms 1,2,3, which don't have to rely on poetry and can be chosen by the dev to satisfy their constraints. This way __version__
remains outside the scope of poetry and poetry doesn't have to implement the somewhat fragile search and replace logic as bumpversion does. People who don't think the version number should be available at run time for their package can still use poetry. Not sure why that would be important, but I like the invariant "poetry doesn't tamper with your source code, doesn't make dev choices for you".
As to SCM: it's a strange situation where everyone is using it but we can't assume it. I think it would be hard to replace tools like bump2version, with which poetry version overlaps, if it ignored SCM. My suggestion would be to tag the version if git is present, move on otherwise.
Another potential solution: imitate setuptools_scm (see last part of https://github.com/pypa/setuptools_scm#pyprojecttoml-usage) and let the developer specify something like
# pyproject.toml
[tool.poetry]
version = {version = "0.1.0", write_to = "pkg/_version.py"}
(Add _version.py
to your SCM's ignore file so you don't check it in)
Then you can import it like
# pkg/__init__.py
try:
from ._version import version as __version__
except ImportError:
__version__ = "0.0.0+unknown"
No external deps necessary.
@madig If I understand you proposal right, that doesn't require an SCM at all, right?
Yas.
Thanks. Incorporated as solution 3 in my write up. Let me know if I didn't describe it properly.
Added an option to make SCM primary source of true, controlled in pyproject.
Any news on this matter?
I'd describe it as
Pyproject.toml plus a Python file that
poetry install
generates with a__version__
variable that can be imported elsewhere: no duplication to manage in most cases as you don't need to touch the generated file except when jumping around versions
I don't know if it's intended or not. A way to do that safely is to parse the root
__init__.py
, detect__version__
, back it up, extract the ast, bump the version string and replace it, then extract the new ast and compare the result. If both ast are the same, except for the version, the file semantics have been preserved. Otherwise, rollback the change and display an error message stating we can't bump the version safely.