Migrate to Poetry and modernise

[tombh]

Poetry is currently the most popular Python dependency manager. Whilst it is ostensibly a replacement for the requirements.txt convention it also encourages more general best practices such as; dependency lockfiles, oneliner releasing, task running, and so on.

Note that the lockfile isn't used to force locked dependencies in the officially published packages to PyPI. Wheels stilly only use the conventional versions (Eg ~1.0.3, ^2.3.12, >=3.8.1,<4, etc) specified in pyproject.toml. The lockfile will mostly just be used for reproducibility in CI

New

• Introduces Ruff, a modern Python linter, that in our case replaces our dependencies on flake8 and bandit. This is also a nice community hat tip as Ruff uses Pygls for its LSP: https://github.com/astral-sh/ruff-lsp
• Introduces Poe for standardised task running. Eg poetry run poe lint
• CI now has a separate lint job, so tests can pass but lints can fail, giving better visibility to the critical qualities of PRs.
• 2 new updates (poe and I can't remember the other right now) require that we have a minium Python version of 3.8.1. Python 3.7 is end of life, so I assume that's ok.
• Linter to check that the Pygls client has been autogenerated to its most up to date state. This means that the Last Modified: section of the autogenerated file is no longer dynamic, it just references the git command to get the last commit date for the file. When the date is dynamically added, it makes it difficult for git diff to detect if actual relevant code has changed.
• Duplicate CI jobs are skipped, namely jobs are no longer run for push and pull_request events at the same time, only one is chosen. Thanks to https://github.com/fkirc/skip-duplicate-actions
• CONTRIBUTORS.md is automated
• Automate releases by making a Github Release
• CHANGELOG.md autogenerated during Github Release

TODO

• ~[x] Remove Python 3.7 required check from branch protection rules /cc @dgreisen~
• [x] Add PyPI API token to Github Secrets for automated releases, see: https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/ /cc @dgreisen
• [x] I assumed that the lock file would now fix the Pyodide flake error: ImportError: cannot import name 'Any' from 'typing_extensions'. But Poetry doesn't build wheels with the lock file, there's some discussion here about the default behaviour, they recommend using a plugin like poetry-plugin-export to achieve it.

Code review checklist (for code reviewer to complete)

• [ ] Pull request represents a single change (i.e. not fixing disparate/unrelated things in a single PR)
• [ ] Title summarizes what is changing
• [ ] Commit messages are meaningful (see this for details)
• [ ] Tests have been included and/or updated, as appropriate
• [ ] Docstrings have been included and/or updated, as appropriate
• [ ] Standalone docs have been updated accordingly
• [ ] CONTRIBUTORS.md was updated, as appropriate
• [ ] Changelog has been updated, as needed (see CHANGELOG.md)

[tombh]

Thanks 🤓

I'd not actually used poe before either. I was actually a little disappointed that I had to introduce it, because the only reason it's needed is because Poetry doesn't currently support running arbitrary scripts, say like npm run does. poetry run only allows running entrypoints into the main application. There's a lot of discussion about it here: https://github.com/python-poetry/poetry/issues/2310 So I'm very open to other approaches here.

Karthik Nadig is saying that he'd like to make a more experimental release next week with lsprotocol==2023.0.0a3. So I'm wondering if we could make a release ourselves to pin lsprotocol==2023.0.0a2 using this PR to test out the new Github Release workflow. I know we'd talked about waiting to complete LSP 3.17 compliance before making a new release, so what do you think?

[alcarney]

I know we'd talked about waiting to complete LSP 3.17 compliance before making a new release, so what do you think?

That was perhaps me being over-optimistic :sweat_smile:, happy to go for a release now.

[tombh]

Ok, just waiting for @dgreisen to do those Github Settings chores then.

I tested the release,yml on Super Glass and found a tiny bug, so pushed that same fix here as well, hence the new review request.

[karthiknadig]

2 new updates (poe and I can't remember the other right now) require that we have a minium Python version of 3.8.1. Python 3.7 is end of life, so I assume that's ok.

FYI, on the python tools repos we keep EOL python support for three months, with a note in our releases asking users to switch to new versions of python (for python 3.7 till Sept 27th). This means that if pygls switches to 3.8 as minimum then tools that we maintain will not update to it till end of September.

[tombh]

That seems totally reasonable @karthiknadig, I'm sure I can find a way to bring this PR back down to 3.7 👍

[tombh]

Turns out it was really easy to maintain Python 3.7 support, just needed to change poethepoet from 0.20 to 0.19 and sphinx from 6 to 5 ✅

So @dgreisen can you add back the 3.7 check requirements please?

Also, even now, with pinned dependencies, we got the ImportError: cannot import name 'Any' from 'typing_extensions' Pyodide error again! So I don't know what's going on there 🤔

[tombh]

Ok, so I've got this back down to Python 3.7. I think we can merge it now.