Python is no longer easy to use for new users

[rbricheno]

PEP668 while likely to reduce the chances of people trashing their system Python also sets up a pretty big hurdle for new Python users to overcome.

The impact of this could be significantly reduced by (for example):

• creating a default per-user venv for each user on the system, and using it be default in their profile
• configuring thonny to use that venv

@gadgetoid propsed something like this for the Pimoroni software, but I suggest that it would be much better to address it at the OS level otherwise everyone will end up having to (learn how to) do this themselves.

There may of course be a better way to address it. Just documenting it doesn't really cut it, venvs are one of the most confusing things you have to deal with in Python. You might say that's a skill issue, but this is supposed to be a computer that the unskilled can learn on.

[Gadgetoid]

My "solution" is ... sort of okay ... for glossing over command-line use of Python within a venv. Unfortunately it falls flat when it comes to making packages installed therein available to Thonny and other IDEs. In these cases the user must always negotiate some tertiary configuration or jump some additional hoops- so no matter how many times we say: "It's easy, just type:"

python3 -m venv --system-site-packages ~/my_venv_dir
source ~/my_venv_dir/bin/activate

We are largely just kidding ourselves that virtual environments are that simple.

Case in point: If venvs are so hip and cool and easy why does Pi OS Bookworm cheat its way around them by letting users loose on an apt-managed, system Python environment via Thonny? Of course anyone who remains a beginner will never really care why basic "blink an LED" Python Just Works, but how are they expected to advance beyond this manicured garden when it's surrounded by a brick wall?

And it's not just that "venvs are hard." It's the dichotomy of "system python" vs "virtual environments". Beginners trying to push the envelope are not just going to wonder why they can't install "Insert Cool Pypi Python Package Here", but wonder even harder why, when they install it to a venv, it stubbornly refuses to import into their Thonny project.

I don't think RPi should push back against this recommendation. But I do think they should lead the charge in papering over this rift. With Ubuntu now being screamingly awesome on Pi 5, what is Pi OS for if not solving these kinds of problems?

The alternative is everyone doing their own thing and causing absolute bedlam.

(ﾉŎꇴŎ)ﾉ✲ﾟ｡⋆

python3 -m venv --system-site-packages --prompt bedlam ~/absolute_bedlam
source ~/absolute_bedlam/bin/activate
pip install thonny
thonny &

[ghollingworth]

The problem with creating a default environment is that it'll also stop the system python3-xxxx applications from working. Also the default environment then won't be valid if you do "sudo pip install" and if it is (because you use sudo -E pip install) then it will end up writing root owned files into the user's .venv directory which will also break the user's view.

There is no solution to this problem which makes it easier without also breaking it in some subtle way which is not the same as any other linux platform.

There are other options than python3 -m venv such as virtualenvwrapper

$ sudo apt install python3-virtualenv
$ sudo apt install python3-virtualenvwrapper

$ cat << EOL >> .bashrc
export WORKON_HOME=~/.virtualenvs
source /usr/share/virtualenvwrapper/virtualenvwrapper.sh
EOL

Now start a new terminal and do:

$ mkvirtualenv myproject
$ workon myproject

And you're done, I was thinking of installing the bashrc changes by default so we could add the last two lines to tutorials...

[Gadgetoid]

The wrapper would be a good call, since venv is a little cryptic and requires you to remember the explicit full path of your “bin/activate” in each case.

Doesn’t much help Thonny users running into the brick wall that divides system and venv Python though- I’d have been tempted to make the Thonny shortcut a shell script that boots it out of a venv named “thonny” and recommend “workon thonny” to users wishing to side load Pypi packages. I’m sure there are just as many different flaws with this approach.

Maybe some GUI view into available projects (which I think Ben suggested on the forums) with a “open a terminal” or “open Thonny” button would be a less destructive approach that would work in concert with venvwrapper. That’s something I could try prototyping, since I’m not about to come in here and rock the boat without offering some proof of concept 🤣 (though I am officially off today for baby’s first birthday)

Bonus points for somehow being able to use the single system wide Thonny and launch it into different venvs, rather than installing it into each.

[ghollingworth]

I was under the impression that Thonny had always used it's own venv to avoid exactly this problem...

Gordon

[aallan]

I was under the impression that Thonny had always used it's own venv to avoid exactly this problem...

That was my understanding as well? I think you can use the system Python but the default Python is inside a venv.

Yes. Can confirm, just spun up Thonny and the default Python distribution is "Thonny's Python" and it lives in /Applications/Thonny.app/Contents/Frameworks/Python.framework/Versions/3.10/bin/python3.10 on my Mac.

[Gadgetoid]

If I switch Thonny into normal mode and try to install a package via it's own UI, I get:

(Yes I've no earthly idea which screenshot utility to use right now :laughing: )

[aallan]

If I switch Thonny into normal mode and try to install a package via it's own UI, I get:

If you look on your screenshot you'll see "Local Python 3 /usr/bin/python" in the bottom right of the Thonny interface, you're trying to install into the system Python.

Poking around, unlike macOS and Windows, on the Pi Thonny defaults to the local system environment, rather than a virtual enviroment. However you can spin up a venv from the settings.

Click on the bottom right interpreter line and choose "Configure the interpreter".

Then select "New virtual environment". After spinning one up it'll be available as part of the drop down in the lower right

(Yes I've no earthly idea which screenshot utility to use right now 😆 )

That'd be grim.

[aallan]

@ghollingworth I think Thonny not defaulting to a venv in a post-Bookworm world probably qualifies as "a bug" at this point. I'm also almost entirely convinced it used to default to a venv, so I'm not sure what happened here?

In any case I'll throw this into the documentation.

[Gadgetoid]

I think Thonny not defaulting to a venv in a post-Bookworm world probably qualifies as "a bug"

YES!

Also, come on. Telling me how to use Thonny? Was there any point at which you were typing that and thinking "well if I have to explain this to a guy with over ten years experience shipping Python libraries to our customers, maybe there's something wrong?" :facepalm:

[aallan]

Also, come on. Telling me how to use Thonny? Was there any point at which you were typing that and thinking "well if I have to explain this to a guy with over ten years experience shipping Python libraries to our customers, maybe there's something wrong?" 🤦

It was more "for the record" than for you. This thread will turn up in Google. 🤷‍♂️

[aallan]

Just added a section to the documentation, and have pushed that to the deployed site. Should be live soon.

[pelwell]

This thread will turn up in Google. 🤷‍♂️

This.

[Gadgetoid]

None of this really fixes the wider issues of the rift between the Thonny environment and all other interactions with the Pi, however.

If Thonny is entirely self-contained than how do users trying to transition away from Thonny's walled-garden:

• Run their script from the command-line
• Run their script on startup
• Install Python libraries into their Thonny environment from command-line (via helper scripts or from git repositories)
• Edit their Thonny scripts in another IDE

And that's saying nothing about those who don't want to use Thonny at all.

It looks like VSCode looks in ~/.virtualenvs and $WORKON_HOME for Virtualenvwrapper.

I'd strongly suggest the following packages are preinstalled-

• python3-virtualenvwrapper
• virtualendwrapper

I've confirmed that installing VSCode allows it to find venvs here, and using "Select Interpreter" will let you choose one. The whole "Pi 5 runs vscode super well" thing is a feather in its cap, so it seems prudent to lean into what with whatever solution we choose.

Unfortunately I don't believe Thonny knows or cares about ~/.virtualenvs and $WORKON_HOME, but if a giant like VSCode is supporting ~/.virtualenvs then it would seem prudent to encourage Thonny to do the same so they are listed when you use "Configure Interpreter"

How close is Pi's relationship to the Thonny devs? Can this kind of change be eked out of them with any kind of urgency, handled downstream?

Finally, if we're agreeing about a standard path for virtual environments and trying to encourage that via distro choices it would make a basic "Manage Python Environments" GUI a lot easier to throw together and a lot more practical.

This thread will turn up in Google

We're having a conversation about an issue that affects multiple thousands customers ability to install software, and greatly affects how I approach developing and shipping our software in future, and you're trying to SEO optimise it? After giving me an "it works like this on my mac" response? :face_with_spiral_eyes:

This.

I honestly have no words.

[pelwell]

I honestly have no words.

Here are a few more then.

Look, Phil, we're all working towards the same goal, trying to make the best of a usability nightmare. We understand the position you are in trying to maintain support for all the Pimoroni products, and we value your input as somebody even closer to the sharp end; please try and remember that when reading the comments here.

[Gadgetoid]

From one Phil to another, thank you. I'm putting a lot of thought into my replies here (and this issue at large), and ensuring I test my assumptions on a Pi 5 as I go. The replies above, that whimsically miss my points entirely, are... frustrating.

Fundamentally our perspectives are wildly tangential. @aallan understands this as a documentation issue, whereupon I understand it as the opening for another project like PiWheels to give an authoritative, top-down and user-friendly solution to a problem.

In the near term, @aallan is right. There's no software fix right now and I don't believe we can just snap our fingers and come up with the right answer.

In the far term, I strongly believe it's a software problem that needs buy-in from Pi to turn what might become an ungodly mess of not-invented-here kludges from each vendor, into a neat, canonical solution. In my experience, relying on users having read any documentation (or even error messages, or even one line of text above a Yes/No dialog) is, at best, extremely optimistic.

I came here to find out "what can we do about these problems?", since I don't have absolute confidence in my own correctness, but perhaps we'd all be better served if I switched to "here's what we can do about these problems" and let anyone who drives by (or arrives here from Google, ha), and knows better, challenge my assertions.

I'm probably taking this a lot more personally than I should, but a large part of my problem is that I need some vestigial hint of a software solution right now and (because I am crazy enough to care) I would like it to be scalable/extensible to fix these other concerns. I have 100+ Python projects that need migrating and between this and the problems with gpio I have barely even touched those.

It doesn't help that Python isn't a bastion of consistency (or that I have largely refused to engage with virtualenvs in the past), but in the near term I'm erring toward:

• Canonical use of ~/.virtualenvs/ by
• Installing python3-virtualenvwrapper and virtualenvwrapper by default and,
• Adding this to ~/.bashrc:

  WORKON_DIR=~/.virtualenvs
  source /usr/share/virtualenvwrapper/virtualenvwrapper.sh

• Above mentioned feature addition to Thonny, adding ~/.virtualenvs support
• Ideally Thonny's default venv lives in ~/.virtualenvs/thonny or similar, so users can workon thonny and have a cohesive environment between GUI and CLI. (issue raised https://github.com/thonny/thonny/issues/2974)
• Documentation including use of workon and mkvirtualenv and how to select a Python virtualenv in Visual Studio Code
• Maybe documentation for lsvirtualenv -b (lists all current virtual environments)
• Some kind of default-installed GUI for listing virtual environments and launching tools in the right context?

In the near term this would let me make software installers that create ~/.virtualenvs/pimoroni right now, with the expectation that it will be fully compatible with Thonny, VSCode, the mythical GUI tool for managing envs and, crucially, all the official documentation going forward.

[pelwell]

perhaps we'd all be better served if I switched to "here's what we can do about these problems" and let anyone who drives by (or arrives here from Google, ha), and knows better, challenge my assertions

Having a concrete suggestion helps to concentrate the mind.

I should add that I have no opinions on this other than:

1. we need to fix it, and
2. thank goodness it's not my job.

[aallan]

Fundamentally our perspectives are wildly tangential. @aallan understands this as a documentation issue…

You're entirely correct, at least to me this is a documentation issue. Once the documentation is updated then this is by definition no longer my job to fix, and I have no other levers I can pull. I've done my best in the face of, as @pelwell rightly called it, a bit of a usability nightmare.

In the near term, @aallan is right.

I'm always right. It is widely known. 😄

[Gadgetoid]

Suggesting the tiniest change to Thonny I could muster to support ~/.virtualenvs without users having to monkey about manually selecting an environment: https://github.com/thonny/thonny/pull/2978

Edit: TIL - https://pypi.org/project/venvipy/

[aivarannamaa]

I'm also almost entirely convinced [Thonny] used to default to a venv, so I'm not sure what happened here?

It used to be like this (in version 2, I think), but I was asked to change it for RPi so that the users could easily import apt-installed Python packages. Later I ditched automatic Thonny-venv creation on other platform as well.

I'm surprised that Thonny's package manager now gives "Externally managed" error on RPi. I was expecting it to use pip install --user in non-venv cases. I'll investigate this.

[ghollingworth]

Yeah, I think that was me, and at the time it made sense because otherwise, the behaviour of Python under Thonny and Python from the terminal was different.

But now it makes sense to go back to using virtual environments by default because they are being forced on the command line as well.

I don't believe --user will help here, since although the user's installation won't break system packages for the root user, it will break for the user...

[aivarannamaa]

I don't believe --user will help here, since although the user's installation won't break system packages for the root user, it will break for the user...

That's right. I had only skimmed PEP 668 and missed that even --user installs are forbidden in externally managed Pythons. I'm not sure it is right decision, though -- for isolated (-I) apps user-site-packages don't matter anyway and non-isolated apps can be messed up anyway (e.g. by running them in the context of an activated venv).

I have some ideas for making management of virtual environments easier inside Thonny:

• Add "Create virtual environment here" to the file browser menu (View => Files => ☰ ) and the back-end switcher (lower-right corner of the main window). It would present a confirmation/progress dialog with the option to override the default venv name (e.g. .venv). After the venv gets created in current directory, Thonny automatically starts using it as the back-end.
• Automatically add virtual environment's executables into the list of the back-end switcher based on current directory. Perhaps even present an indicator proposing to switch to venv's Python when user changes to a directory containing a venv.
• Add an option for automatically switching virtual envrionments when the user changes the current directory (either via file browser or by running a script in another directory) and Thonny spots a venv in the new directory.

Regarding the gap between Thonny's "walled garden" and outside world:

• Thonny does not hide the full path of venv's interpreter, so once the venv exists, the user can copy the full path of the interpreter for running their code outside of Thonny. Unfortunately, many users don't know that you can do this without activating the venv.
• I believe it's not too hard to create shell extensions matching "Create virtual environment here" and automatic activation of a venv when changing directories (in the spirit of direnv or autoenv).

But now it makes sense to go back to using virtual environments by default because they are being forced on the command line as well.

They are forced only if you are not happy with the current selection of apt-installed global packages.

[Gadgetoid]

They are forced only if you are not happy with the current selection of apt-installed global packages.

Hahaha, that's like saying you're only forced to catch a train if you don't want to walk 500 miles :laughing:

I'd love a good, solid, dedicated UI in Thonny for creating and managing virtual environments system-wide. I guess it might not be part of its remit, but I explored VenviPy (https://pypi.org/project/venvipy/) and it was a little rough around the edges. The ~/.virtualenvs dir seems to be a reasonably standard place to store environments - insofar as there is any kind of standard here - so acknowledging its existence would be helpful.

I am currently migrating a bunch of libraries, but I intend to look into a GUI Venv manager again at some point.

Shell extensions and auto-switching would lean quite heavily on Pi to embrace them, but they could be a nice integrated alternative to a dedicated UI.

Thanks for chiming in here!

[Gadgetoid]

@aivarannamaa would be interested to know if you got anywhere with this?

Prodding and poking Thonny on my Pi suggests there's no easy way for me - as a determined work-arounder of this situation - to make Thonny launch with a virtual environment by default. So far I've resigned to just editing ~/.config/Thonny/configuration.ini (with the express intent that the user simply doesn't want to know/care about the details of selecting a virtual env interpreter).

Anything to help me help end-users would be greatly appreciated.

Here are my not-very-complete instructions: https://pimoroni.github.io/venv-python/#thonny

[aivarannamaa]

@aivarannamaa would be interested to know if you got anywhere with this?

Thonny 5.0 will come with a dialog for creating virtual environments, but I don't feel particularly happy about such local optimizations. I think Python package management needs something totally different.

I recently discovered uv and I like it. I'm considering adding uv back-end for Thonny (back-end is something that can be selected from the menu in the lower-right corner of Thonny's main window). This back-end would make F5 to invoke the current script via uv run instead of plain python.

For single-file programs uv offers nice support for inline script metadata. With venv-s, the step from stdlib-only script to a script with 3rd-party deps could be mindblowing, but with inline script metadata and suitable tooling (like uv's) the increase in complexity feels more reasonable. Yes, the format of inline script metadata is not good for writing by hand, but I'll try to make Thonny help with this.

uv's approach of picking up the project definition according to current working directory matches with Thonny's approach of not requiring a fixed project context. I envision that in uv mode, Thonny's package manager (which is a panel in the main window in Thonny 5.0) would update its content automatically as the user switches current working directory. If the current working directory is not part of a project folder, then the active script would be the context for the package manager.

With uv, stepping out of Thonny wouldn't be too hard -- uv run my_script.py is easier to explain than how to manage a virtual environment. Perhaps I can advertise this command line by mentioning it next to Thonny's magic commands, e.g.

>>> %Run hello_world.py           # uv run hello_world.py
Hello world!

[Gadgetoid]

Thank you for responding so quickly!

I had not come across uv and now that's a whole rabbithole I need to delve into, too! Thank you.

I am not particularly staunch on using virtualenvwrapper and ~/.virtualenvs/whatever, so it's always been making the best of a bad situation. If we can come together around something straight-forward like uv (and if it works in VSCode and other reasonable places too) then it could be a path forward.

So far I've been inclined to hide as much of this complexity from the user as possible, but I'm not sure that's really feasible- as you suggest the step up from a single file venv to a reproducible project is... steep to say the very least. Since inline metadata is a Python feature that seems almost intended for the kind of projects a Pi user might put together, it seems reasonable to embrace it.

uv run my_script.py feels significantly more portable. Though I suspect it makes my complaints about Python packages being shipped exclusively via Pi's apt repository more salient.

Anyway, I like where you're going with this!