fix(cli): improve messaging on renku installation update

[jsam]

When, local installation of renku is outdated, user is greeted with the following message:

You are using renku version 0.8.2, however version 0.9.1 is available.
You should consider upgrading ...

The suggestion we received from users is that we add to this message a command which they could use to do this. We should default to pip usage and recommend it as a default package manager for updating.

[rokroskar]

pipx not pip :)

[emmjab]

because it's kind of an important detail -- the experience is different if in notebooks or local;

in notebooks:

• image provides pipx
• if you try to install new version with pip instead and you don't update paths, you'll still be using old renku from pipx
• if you open a new notebook, you'll still be using old renku
• solution should involve updating the FROM statement in the docker file to use a later renku
• solution? can spit out the FROM line, path to the dockerfile, and link to the docs to help people update this
• maybe something more automated later, but still an issue if people are using their own base images

local:

• if your original installation was in pipx and you try to install new version with pip instead and you don't update paths, you'll still be using old version
• pip vs pipx caused me & others a lot of headaches, we should get our story straight and update the readme & docs

[rokroskar]

I would discourage anyone from installing it with pip if all they want is to use the CLI. If they want to use it as an sdk then it can be installed with pip but probably in a virtual environment.

[jsam]

@rokroskar interesting, could you explain your reasoning about this a bit more?

I see it completely opposite and we should not encourage people to use pipx since its not sponsored by PSA (PyPA) or PSF, it's a relatively small project, most of the community is not using it and on top of that, it's notorious for causing installation trouble. I see no good reason for keeping this around as part of our official support. If they want and need to isolate installation, then I assume they know what they are doing and how to achieve it.

Imho, our official support for distribution channel should be the one which is fully supported and not some small community-driven project which is popular in one specific area of Python community.

edit: I do say above things with complete understanding of what pipx does and I agree that it is the completely nice thing to do, but it's solving a problem which is beyond renku, and since there is no direct dependency between those 2 project we should default to one which is fully supported, instead recommending one which is not.

[rokroskar]

yes - renku has a lot of dependencies, many of them are pinned to a specific version. This tends to cause unresolvable conflicts for users very quickly when they install it into their generic python environment. Our default installation channel was pip and at our first hackathon people's python environments were blowing up all over the place. For command-line tools, pip is a very bad choice imo, and your assumption about users who want to isolate tools in their own environments knowing what they are doing is not valid in my experience.

Our users typically have no idea what a python environment is, where python is installed, or what isolation even means. pipx is simply a wrapper for venv and pip.

[jsam]

I understand, and I agree that on paper this does seem as a better idea - it's just that my experience so far showed the opposite case. I've check out other CLI tools (gcloud, yt-download, ...) based on Python and none of them is recommending pipx.

Considering that so far I had more than 3 cases where I had to troubleshoot user's issues which were all related to pipx and none of them were renku related - it seems that it just adds technical debt and not really viable long term solution. To me, it seems like a good short-sighted solution that works well for hackathons, but is potentially a quite bad solution for LTS (due to amount of problems we have already encountered with it just being in our readme).

I'm happy to be proven wrong on this, but until pipx becomes officially supported distribution method and general community accepted solution, my vote goes to pip (since I see no problems with having renku versions per project - you don't have to pollute globally if you don't want to).

edit: In all 3 cases (I mentioned above) when I was helping a person upgrade renku via pipx, we managed in doing so by removing the pipx completely and then installing renku via pip normally on to machine (and yes I've tested the update and it worked as well). Troubleshooting pipx problems was almost always related to upgrading renku which for some reason does not work when new release contains a different version for transient dependencies. In case you insist on this decision we should probably look into this problem to build better LTS.

[rokroskar]

What were you troubleshooting that was pipx-related? What ended up being the problem?

What do you mean by "since I see no problems with having renku versions per project - you don't have to pollute globally if you don't want to"?

pipx uses pip to install packages - so whatever dependency resolution issues you were having are related to pip and not pipx. All pipx does is create a virtual environment, usepip to install the package there and then make a symbolic link to the executable script from ~/.local/bin.

[jsam]

The usage pattern I've noticed is that users install renku globally and then if they have renku project sometimes they install it in a project environment which depends on renku. This creates a usage problem since they get confused about which renku are they using in the end. In this case, if you upgrade through pipx, even if you upgrade successfully you will end up upgrading the wrong renku installation which ends up in more confusion.

I understand that internally it relies on pip (even though it does not ship with it), but upgrading packages which in new version adds more package ended up in incorrect environment set up (and this happened on multiple times on both linux and osx).

I did not inspect it in detail so I cannot say if the problem was exactly due to transient dependencies or simply a failure in adding additional deps (which would be even weirder). I've try instead of upgrading the package reinstalling it, which ended up in corrupted virtual environment at which point I simply opted for deleting everything and setting it up with plain pip - which ended up in results which user wanted in the first place.

Either way, if we decide to support this, we should look into how to resolve all of those issues we are buying into. Also editing readme files with explaining most common pitfalls would be a good idea.

edit: quick browsing through the code of pipx reveals to me that they are not using exactly the same pip install or upgrade with additional virtual env magic but the approach is slightly different as well, which could result in different transient dependency resolution.

[rokroskar]

It would be great if you can reproduce the problem and document it - I've been using pipx happily for 2 years without problems. They did change their CLI a bit recently to be more in line with pip which was annoying but not a deal-breaker for me.

[jsam]

Sure. I think for starters, we should improve our docs on that part since most of the usage confusion comes from the case I mentioned. For example, pipx section could be moved into a section called "global installation" which explains global vs local installations and then adding a section which explains what and how to use renku if you install it as a dependency. I think the latter is completely missing currently.

And then the most important part is how to upgrade those (with all the most common issues which happen).

[rokroskar]

The problem you describe sounds like a generic virtual environment issue. I'm a little surprised this is common with renku users since most don't use renku as a library but strictly as a CLI tool. I wouldn't confuse the docs with "global" etc sections - there should be a super short install section for the most common use case (renku as cli) and maybe another section that mentions specifically usage as a library. This is basically what I wrote in #1098.

[rokroskar]

FWIW pipx has moved much closer to pip in the months since this issue was first opened. They basically follow the pip syntax now so showing this in a brief message as suggested at the top of this issue seems more plausible now.