build: add PEP-517 build config with Poetry

[staticdev]

Closes #509

[adrienverge]

Hello, thanks for tackling this!

The use of Poetry doesn't seem mandatory: could we use something minimal, even if files are not automatically generated? (For example @DimitriPapadopoulos mentioned keeping setuptools: is it an option?)

Also I see some problems, e.g. inconsistencies in Python version dependencies (3.6+ vs. 3.7+), no more command to build documentation, the duplication of the yamllint version in multiple files.

Finally, the impact of this to distro packaging (.rpm, .deb, etc.) should be kept as fluent as possible.

[staticdev]

Hello, thanks for tackling this!

The use of Poetry doesn't seem mandatory: could we use something minimal, even if files are not automatically generated? (For example @DimitriPapadopoulos mentioned keeping setuptools: is it an option?)

Also I see some problems, e.g. inconsistencies in Python version dependencies (3.6+ vs. 3.7+), no more command to build documentation, the duplication of the yamllint version in multiple files.

Finally, the impact of this to distro packaging (.rpm, .deb, etc.) should be kept as fluent as possible.

I tried in this PR to make a minimal change to be compliant with new PEP-517. But you have all valid points.

About setuptools.. It is possible to keep, but did not understand, why do you want to keep it?

About duplicated version.. New packages only use pyproject.toml to control the version. I did not remove it from init file for a simple reason that you still support python 3.7. From 3.8 it is just add importlib-metadata and you are good to go with version. There is also a backport for python 3.7 but all projects I work on already dropped 3.7 since it is 2 months for EOL https://endoflife.date/python

For documentation build I am not sure how you want to go. I usually just add a simple requirements.txt on my docs folder and build from there on github actions. You can also see as example any of my python repos.

I did not dig yet how you are doing other packages. Do you have any suggestions regarding the changes I did regarding that topic?

[ssbarnea]

I also dislike poetry and use setuptools-scm instead, I could probably make a PR that make uses of it.

[staticdev]

@adrienverge just in case I addressed in this PR the issue regarding docs build and version management (now it is only part of pyproject.toml as recommneded - it will only install importlib_metadata as dependency for Python <3.8). This is basically how it is working in all projects I am currently maintainer.

If setuptools is must then setuptools-scm from @ssbarnea suggestion would be the other approach.

[DimitriPapadopoulos]

I am under the impression that setuptools is not considered more standard than poetry nowadays. Yet, I would prefer setuptools too, together with _setuptoolsscm to extract the version.

[staticdev]

Actually I don't understand the preference on continuing to use setuptools. Can any of you enlighten me with a list features setuptools/setuptools_scm has that poetry does not have? Poetry is built from scratch upon the new standards of Python packaging defined:

• https://peps.python.org/pep-0517/
• https://peps.python.org/pep-0518/

Pros of poetry for me:

• it is much more actively developed today than setuptools
• it is very stable since version 1.0.0 (I am basically using it professionally for the last 3 years)
• configuration is simpler than setuptools (you don't need to create setup.py and setup.cfg)
• it's tooling it excellent for initializing, building and publishing with a simple CLI interface (works fine with proxies, internal pypi repositories such as Nexus/Artifactory) - just do: poetry init, poetry build, poetry publish
• it creates identical virtual environments of all developers of the project with the lock file avoiding version conflicts again with just a poetry install
• the dependency resolver is rock solid and fully compatible with latest pip resolver - you can simply add/remove, update and relock dependencies with: poetry add package [version], and poetry update [package], poetry lock..
• it is flexible since you can use dependency groups for different stages/workflows in your CI/CD
• it is safe (all types of hashes can be used to verify integrity of downloaded tarballs)
• it is compatible with all versions of python

I don't really have cons.

[DimitriPapadopoulos]

Speaking for myself, I guess I am getting tired of the continuous changes and instability around Python. As if the necessary Python 2 to Python 3 migration was not enough, I now have to spend excessively much time adapting to continuous changes in the Python development and packaging environment. It is a complete mess – although I understand this is an effort to move from the current mess to a better future. As a result, I think I have a strong personal bias against such continuous breaking changes, and arguments in favour of such changes. That said, changes do happen nevertheless out there, I cannot ignore them for ever :smile:

• According to The Basics of Python Packaging in Early 2023, poetry has a PEP 517 build backend but is not PEP 621 compliant. Is that still the case?
• I have read other articles such as Python Poetry vs PIP. Does poetry replace pip for developers only, or will this change affect end-users too?

[adrienverge]

Hello, and thanks for these valuable inputs!

I have more or less the same fears as @DimitriPapadopoulos (but maybe I'm wrong): Poetry is actively maintained, but what about in 10 years? I prefer stability over features. Yamllint dependencies are kept simple on purpose (just PyYAML + pathspec, always the last version): could a fully-featured dependency management tool like Poetry be overkill? That being said, I haven't followed the progress of each tool recently, so I may be mistaken.

In any case, the next packaging system should:

• Be stable and plan to be maintained for years.
• Be lightweight and foster easy maintenance: - no more dependencies, either with pip or with distros (.rpm, .deb...) - easy distro packaging - the least extra files (e.g. auto-generated poetry.lock that is huge, and seems to paraphrase pyproject.toml)
• Make it easy to: - run flake8 - run tests, including coverage - build documentation - package - publish

Would setuptools + setuptools_scm allow lighter configuration? Is it possible to avoid using poetry.lock (the project dependencies are simple and we stay up-to-date)? Is it possible to put .flake8 data inside pyproject.toml (like we did with setup.cfg)?

[ssbarnea]

Flake8 config is the only one that cannot be moved, mainly because current maintainer did not want to add support for it. Anyway, I think that flake8 days are numbered, as ruff is getting traction at huge speed. I started using it few days ago and I was amazed. I will be using it in parallel with flake8 for at least two months before removing flake8. Anyway, that is bit diverging from the subject but I imagined that you could find the information useful. And, yep, that one uses pyproject.toml.

[adrienverge]

@ssbarnea good to know, thanks :+1: I just tried it but it doesn't support configuration in setup.cfg. So it could be a good idea to switch from flake8 to ruff, but after the upgrade to pyproject.toml.

[staticdev]

Also interesting for me.. i will keep an eye on ruff.

UPDATE @adrienverge I fixed errors on CI/CD of missing dependency and also a replace.

[adrienverge]

Thanks for the updated code @staticdev :+1:

It looks like tests still won't start. In the meantime, I'll work on a setuptools version of pyproject.toml so we can compare more easily.

[adrienverge]

I opened https://github.com/adrienverge/yamllint/pull/565 and https://github.com/adrienverge/yamllint/pull/566 to upgrade from setup.py to pyproject.toml, using the setuptools way. I didn't include setuptools-scm for automatic versioning, to keep things more straightforward.

First of all, thanks for this Poetry version @staticdev, I appreciate the work done. But after testing with setuptools, I see multiple reasons to use the latter:

• It's less diff (7 files changed, 68 insertions(+), 85 deletions(-) for #566 vs. 11 files changed, 234 insertions(+), 111 deletions(-) for #557), and avoids the huge poetry.lock file, that would need to be updated often.
• The Poetry handling of version makes APP_* and __version__ not very consistent.

• 557 still doesn't check documentation build.

• Unit tests and coverage don't work on GitHub Actions for #557.
• In #557 pyproject.toml lacks the description.

You can see how it looks here: https://github.com/adrienverge/yamllint/pull/566.

[staticdev]

@adrienverge I have rebased based on your changes. Could you run CI again?

[coveralls]

Coverage: 99.196%. Remained the same when pulling b23f67361a128c139e988876d0a67ea3477898f6 on staticdev:feature/pyproject.toml into 16eae28a50047960b8841468ac7b8822c8d1cc66 on adrienverge:master.

[staticdev]

I think when you remove setup.py from your PR you will also need coverage[toml] otherwise it will not fetch config @adrienverge