Upgrade tooling on the project.

almet commented 1 month ago

Replace black by ruff, as it's quicker ;
Drop python 3.7, since it's end of life since last year ;
Use uv wherever possible as a replacement for pip, as it's way faster to run, add an uv.lock file which will be synced before the releases and published here ;
Remove tox, it's too complex for this project and can easily be replaced by uv ;
Apply ruff formatting ;
Update the makefile accordingly ;
Update the CI accordingly

I'm not sure why, but pytest-libfaketime is currently not working when running inside uv, at least on an apple silicon machine. I uninstalled it otherwise the test couldn't run, failing with the following error:

uv run --extra dev pytest .
re-exec with libfaketime dependencies
dyld[16935]: terminating because inserted dylib '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' could not be loaded: tried: '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file)
dyld[16935]: tried: '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file), '/Users/alexis/dev/ihatemoney/.venv/lib/python3.8/site-packages/libfaketime/vendor/libfaketime/src/libfaketime.1.dylib' (no such file)

almet commented 1 month ago

Tests are failing with python 3.10 + sqlite, but I cannot reproduce locally with the following command:

TESTING_SQLALCHEMY_DATABASE_URI="sqlite:///budget.db"; uv run --python 3.10 --extra dev --extra database pytest .

Edit: It's fixed after a re-run.

almet commented 1 month ago

Curious about your feelings on this @zorun ?

zorun commented 1 month ago

Keep in mind that I tend to be conservative for this kind of reviews ;) I once inherited a project using poetry and I really didn't like it.

Here are general questions:

What would be the main advantage and disadvantages of this change? (I don't buy the argument "it's newer fancy tooling"). Do we have performance issues with the current tooling?
What guarantee do we have that the new tooling will still work reliably X years from now?
Does it affect releases? Would the project still be installable via pip? If a user installs the project with pip, do they get the same dependencies that we have in the lockfile? Can we run tests for that situation?
Does it affect users directly or indirectly in other ways?
Does it create a higher bar for entry for new contributors?

almet commented 1 month ago

No problem, thanks for the questions.

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

With that out of the way, let's discuss the points you bring here 👍

What would be the main advantage and disadvantages of this change?

I believe the main pro is speedup. I personally also like to be able to install different versions of python in a simple way with uv python install in just a few seconds, as it makes it easier to run tests without having to install these manually with another tool.

Do we have performance issues with the current tooling?

I don't believe we have performance issues with the current tooling, no.

What guarantee do we have that the new tooling will still work reliably X years from now?

My feeling is that uv is here to stay, but that's a reasonable question to ask. The postscriptum on the announce covers it a bit:

What does that mean for the future of these tools? Here is my take on this: for the community having someone pour money into it can create some challenges. For the PSF and the core Python project this is something that should be considered. However having seen the code and what uv is doing, even in the worst possible future this is a very forkable and maintainable thing. I believe that even in case Astral shuts down or were to do something incredibly dodgy licensing wise, the community would be better off than before uv existed.

Does it affect releases?

No, not really. It can affect how we release, because the tools we use would be installed by uv.

Would the project still be installable via pip?

Yes.

If a user installs the project with pip, do they get the same dependencies that we have in the lockfile?

When using pip, the dependencies are not compiled by the uv lockfile, but by what's defined in the pyproject.toml, so I believe we're covered here.

Can we run tests for that situation?

I don't think there is a need for it :-)

Does it affect users directly or indirectly in other ways?

It's mainly a DX change.

Does it create a higher bar for entry for new contributors?

I believe it's actually the other way around, as it makes things simpler and faster.

zorun commented 1 month ago

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

Thanks, I wanted to hear your motivation. If it motivates you, that's good, but let's not break things for free.

Actually, thinking about it, I think my motivation comes from repairing broken things, so maybe we make the perfect combo :D

What would be the main advantage and disadvantages of this change?

I believe the main pro is speedup. I personally also like to be able to install different versions of python in a simple way with uv python install in just a few seconds, as it makes it easier to run tests without having to install these manually with another tool.

This is a good argument.

If a user installs the project with pip, do they get the same dependencies that we have in the lockfile?

When using pip, the dependencies are not compiled by the uv lockfile, but by what's defined in the pyproject.toml, so I believe we're covered here.

Can we run tests for that situation?

I don't think there is a need for it :-)

Let me be more clear, because this is a general grip I also had with poetry.

Lockfiles are very good when working between developers: we are sure we have the exact same version of dependencies. However, users will have different dependencies because they don't use the lockfiles, they use pip.

If we run tests with the dependencies from the lockfile, we don't test what the actual users will install, and this is quite bad.

We could run tests using the dependencies from pyproject.toml instead of the lockfile, but then, what would be the point of the lockfile? In that case, let's just use pyproject.toml also for development.

What do you think? Did I miss something obvious?

almet commented 1 month ago

Thanks, I wanted to hear your motivation. If it motivates you, that's good, but let's not break things for free.

Actually, thinking about it, I think my motivation comes from repairing broken things, so maybe we make the perfect combo :D

🙌 🤭

Let me be more clear, because this is a general grip I also had with poetry.

Lockfiles are very good when working between developers: we are sure we have the exact same version of dependencies. However, users will have different dependencies because they don't use the lockfiles, they use pip.

If we run tests with the dependencies from the lockfile, we don't test what the actual users will install, and this is quite bad.

We could run tests using the dependencies from pyproject.toml instead of the lockfile, but then, what would be the point of the lockfile? In that case, let's just use pyproject.toml also for development.

Oh, that's nice, and I understand better now the reason why we have the "minimal" set of dependencies in our tests. I actually think this is a good idea.

I see the .lock file as a "known working set" of dependencies. I don't believe we need to ship it to other users, but that could be something documented, and also part of the release, for those who want to have more control over the environment they are running the project into.

For instance, for the Docker image, we could ensure that it ships with exactly the versions that were present at the time of the build, and I believe that would be a good thing. Not a requirement, but something useful for some people.

But, to be honest, these .lockfiles are just starting to make sense to me.

almet commented 1 month ago

Also: thanks for the tone of the discussion, I told you already but it's been a pleasure to disagree with you, and find out how we can better understand each other and come to the same conclusion.

(This is probably because we're not trying to push our decisions onto each other, but really care about what we think, and why)

Glandos commented 1 month ago

I believe part of the motivation behind this change is personal: putting back this project as a way to experiment with "new shiny tools". It gives me energy, and as such I believe it will be net-beneficial for the project in the end. More fun means it will get easier to maintain (at least for me, but you can have a different opinion).

It's also my point-of-view. Since I knew nothing about uv, I checked it. It's pretty new. It's pretty cool. The only reassuring stuff is that it's made by the same team doing ruff which is here to stay. Otherwise, uv would have been the "15 competing standard" regarding Python dependency management.

Oh, that's nice, and I understand better now the reason why we have the "minimal" set of dependencies in our tests. I actually think this is a good idea.

I remember taking part in this, and this was a really good move, because it also tests installation in every cases. I also found myself in situation where I needed to install something (hello Ansible) at not the very last version, and was happy it worked :)

almet commented 1 month ago

Hey, @zorun, after some rest, what's your feeling on this? Should we merge or close?

almet commented 3 weeks ago

did you check that all instructions in docs/contributing.md still work after this change?

No, not yet. I will if we decide to go this way.

what's the position on the uvlock? As I said, I'm a bit dubious about it because users will install IHM via pip, so I believe we shouldn't use the uvlock at all. Does uv use the lock by default? Once we reach a consensus, please document the result somewhere in the doc.

I believe uv.lock files will allow us to ship a known working set of dependencies, that can be used for instance in the Docker file.

But, it won't be used for installations via pip. I think we should keep it, but it won't have any effect on the distribution via PyPI.

spiral-project / ihatemoney

Upgrade tooling on the project. #1329