Facilitate installation of 3rd party packages for use with Mu

ntoll commented 6 years ago

As a (beginner or teacher) user, I want to be able to install 3rd party packages from PyPI in an easy to use/configure way. Furthermore, the packages I install must be isolated from any other user of Mu on the system.

This issue has been created because of the "rule of three" (3+ users have asked for this sort of feature). A typical request comes via Al Sweigart (@asweigart): https://twitter.com/AlSweigart/status/1051960899982872576

My initial thought is to add a pane to the admin widget containing a textarea into which users write the names of the packages they want in their environment, much like a requirements.txt lite. Upon confirmation of any changes we use pip (perhaps like this: https://pip.pypa.io/en/stable/user_guide/#user-installs) to install the packages in a known isolated-but-for-mu path.

I'd like this to be in the Autumn/Winter 1.1 release, I also believe making the simplest possible solution (and then iterating) is the way to go... i.e. this feature is considered "beta" upon release (and could be subject to change upon user feedback).

illume commented 6 years ago

Yes! This would be a very good addition :)

Lots of questions... A big issue is dealing with breakages. And you'll get breakages with every pip release at least(not to bash them, it's hard). But also breakages with third party packages. Scoping this down to just an admin text area might be a good idea. But also risky in breaking installations. Where do you install the packages? Is mu installed in a virtualenv, system wide, in user (currently broken on homebrew/pip by the way). Are you going to support different versions for different projects? It's sort of a can of worms. Does mu itself use the extra packages? To install the packages, will there be a discovery mechanism? Or is browsing pypi ok?

ntoll commented 6 years ago

@illume oooh... you star... these are all great questions.

So, we need to think carefully about this. My aims are for simple and easy. I'm also of the Homer Simpson school of thought that "if something is hard to do, it's probably not worth doing" when it comes to this sort of thing. I.e.: If you're trying to manage packages in any way above and beyond "just install this thing" then you're of a level of understanding and development that's beyond that of the typical end user we're writing for - in which case, you're on your own (or you shouldn't be using Mu). IYSWIM... this is a rather useful "get out of jail" card... ;-)

Off the top of my head, concerning the really great points you raise:

This is really only for (the majority of) users who have installed Mu via the official installer for Windows or OSX... the ones that come with Python and the rest of the kitchen sink already wrapped up into the installer.
If you're running Mu in an existing virtualenv because you've installed it from PyPI then you should just pip install the packages into the current virtualenv. I wonder if there's a way to detect if a Python app is being run from within a virtualenv and let Mu handle it?
The target directory should be within the user's "home", and probably hidden from view (since we only want Mu to manage it).
I think just putting the target directory containing the "installed" packages onto the Python path for scripts run in Python 3 mode is really all that is required. I guess it means we can have a "RESET" button which just deletes and then re-creates an empty target directory, for times when the environment is broken.
Since we want to do the simplest possible thing, we cannot support multiple versions of packages (although you should be able to pin a package version a la requests.txt syntax).
Mu itself will NOT use the extra packages... however the path containing the extra packages will be in scope for imports when running and debugging scripts and also when the iPython based REPL is run.
There should NOT be a discovery mechanism. We're assuming the teacher knows to tell their students (or will have pre-configured things) to install the required packages. If you're any more technical than a complete beginner, you probably already know enough to use Google/search PyPI.

Thanks for the questions... as I hope my answer shows, they really got me thinking about constraining the scope (and justifying such constraints) in such a way that complete beginners have something simple and easy while also affording them something relatively powerful.

One last thing... Mu already comes with some packages bundled (e.g. PyGame, Numpy, Matplotlib, etc...). I wonder how we manage conflicts / updates when the user says they want Numpy==1.2.3.

illume commented 6 years ago

Sounds like a plan :)

If I'm a teacher or book author, and I want people to have 'Mu, but extended to include A,B,C', how do I tell students to get Mu into this state?

martinohanlon commented 6 years ago

+1 for this...

tjguk commented 6 years ago

Well this is interesting...

I'm not looking at the code right now, but one obvious avenue is precisely to have Mu create its own local venv into which we can programatically pip install whatever we need.

I think the canonical way of detecting that you're in a venv is to look for the VIRTUAL_ENV env var.

I am more than a little concerned about possible conflict between the "system" and the "runtime" packages (eg pygame, numpy etc.) as @ntoll touches on.

tjguk commented 6 years ago

NB the supported way of interacting with Pip programmatically is via the command line:

https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program

JoshuaLowe1002 commented 6 years ago

Yes I'd love this, as for where to install the packages (with PortaMu in mind) it would have to be in either the main folder running on the pendrive (normal installation that would be the program files one) or in Mu_Code.

Unless you can think of any other place it could be installed.

asweigart commented 6 years ago

To deal with Pygame/Numpy/etc conflicts, I'd say, at least for the initial beta version, just hard code a message saying "that package is already included in XXX mode" or something.

This would be great to have for an Autumn\Winter release. I'm working on the 2nd edition of Automate the Boring Stuff with Python right now, and I'd like to switch it from featuring IDLE to Mu. But being able to install modules (even if it was through some hacky workaround with several steps) is a hard requirement for that. I expect to have the draft for the 2nd edition done by the end of the year.

-Al

On Tue, Oct 16, 2018 at 12:47 PM Joshua Lowe notifications@github.com wrote:

Yes I'd love this, as for where to install the packages (with PortaMu in mind) it would have to be in either the main folder running on the pendrive (normal installation that would be the program files one) or in Mu_Code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mu-editor/mu/issues/675#issuecomment-430374657, or mute the thread https://github.com/notifications/unsubscribe-auth/AADL_LS8VQXHez-NSStW-jmeI5TIwdxTks5ulje1gaJpZM4Xd137 .

tjguk commented 6 years ago

Thinking aloud a little. Disregarding my previous suggestion about using a venv, it looks like a pip --user install might meet our needs reasonably well.

https://pip.pypa.io/en/stable/user_guide/#user-installs

The key points are:

We can define the install location with an env var
There is a policy for dealing with system-wide install clashes which /seems/ to match our requirements (I think)
It's established pip functionality so we don't have to monkeypatch / reinvent

What I don't know is whether the particular situation of our frozen environments will interfere with pip, eg because sys attributes are set unexpectedly.

martinohanlon commented 6 years ago

Just adding a couple of comments to this....

First "excellent feature", I think I might have been requestee number 2 :)

I quite often "force" a package installation or upgrade into Mu on Windows using:

pip install guizero --target "C:\Users\Martin O'Hanlon-LT\AppData\Local\Mu\pkgs"

I have found this to be the most reliable method.

tjguk commented 6 years ago

Thanks, @martinohanlon. For some reason I didn't spot that when checking the pip docs! Although I would like to check how it interacts with system-wide (ie pre-installed) packages

martinohanlon commented 6 years ago

@tjguk definitely check, but I am pretty sure it doesnt interact with system-wide packages, however this is based on experience not by actually looking / testing!

asweigart commented 6 years ago

Thanks @martinohanlon for that workaround, that's exactly what I was looking for.

If there's anything I can do to help bring this feature to fruition, let me know. I'll start diving into the developer docs today.

tjguk commented 6 years ago

@asweigart feel free to start looking. I was going to make a start but I also want to work on #654 so I'll leave this one to you or others for now.

asweigart commented 6 years ago

It'd probably take longer for me to get up to speed developing with Mu (I only started using it a couple weeks ago). I meant if there was any testing work I should do, I could handle that.

tjguk commented 6 years ago

Ok, thanks. I've just submitted a PR for #654 so I'll start to poke around this one. That said, we'll probably need an up-front discussion between devs to agree the best way forward.

osupk8 commented 5 years ago

Hi, I am very interesting about adding new modules too. I'd like to use microbit whith grove module pi.py from raspberry if I can If you have any suggestion

asweigart commented 5 years ago

Any updates on this? I'm nearing the completion of the second edition of Automate the Boring Stuff with Python and wanted to switch from IDLE to Mu, but I need a way to install third party modules. I can use the workaround methods if this feature can't be added soon.

Current workarounds:

Windows: pip install guizero --target "C:\Users\Martin O'Hanlon-LT\AppData\Local\Mu\pkgs"

Linux: pip3 install guizero

macOS: pip3 install guizero --target /Applications/mu-editor.app/Contents/Resources/app_packages

Thanks to everyone for their hard work on Mu!

martinohanlon commented 5 years ago

As a strategy for this functionality, how do you feel about.

"If a 3rd party package is installed using Mu's package installer functionality, it is only available for use by Mu."

I think this would simplify the functionality and potentially get around other blockers such as needing to get elevated access to install python packages.

In its simplest form you could install the packages directly into the mu_code folder where Python would pick them up. < Fyi - I dont think this is the right solution and it would make more sense to store them elsewhere but somewhere the user had write access too, I just use it as an option.

asweigart commented 5 years ago

Oh, I assumed that's the way to go about it. Mu has its own Python environment that's separate from any other Python installations on the system, right?

carlosperate commented 5 years ago

It does when using the Windows/macOS installers. When using pip the user would have to manage the virtualenv, but more importantly on Raspberry Pi with a deb installer it would be using system Python.

ntoll commented 5 years ago

Hi folks... a quick heads up. I intend to start implementing a solution tomorrow with a view to a working PR early next week, for folks to pile in and give constructive criticism, feedback and suggestions.

tjguk commented 5 years ago

Excellent. I look forward to picking holes in someone else's work. It's what I do best :)

ntoll commented 5 years ago

Quick heads up... I've made good progress this afternoon... assuming I don't get a deluge of email to handle (like I did this morning) there should be a PR for folks to review by EOD UK time tomorrow. In the meantime, here's a screenie of the admin dialog with the simplest package management list... ;-) package_preview

merwok commented 5 years ago

I would recommend pip-compile from https://github.com/jazzband/pip-tools to transform the requirements into a fully resolved set of packages with compatible versions (or a clear error), and pip-sync to handle the install + remove step!

ntoll commented 5 years ago

Still some work to do... but "it works" in a "I've spiked a solution" sort of a way. Tomorrow I'll solidify and ensure it's well tested and the PR contains all my thinking, mistakes and other considerations. In the meantime, this GIF shows it in action. Enjoy. :-)

mu_modules

osupk8 commented 5 years ago

Great. I enjoy to test this new functionnality Thaks for the job ;)

ntoll commented 5 years ago

OK Folks... sorry for the delay (I've been ill with the flu). The fix for this is in #749

ntoll commented 5 years ago

Heads up folks... I just merged the PR to add this feature (#749) into master. It'll be in the imminent alpha 1 release.

MikeTheWatchGuy commented 5 years ago

I don't see the "Third Party Packages" tab when I click the gear in the bottom right corner. I only get the other 3. Do I need to do something to turn this on?

For now I have copied the 2 pip-installed folders from my Python3 environment on Windows into the Mu pkgs folder and was then successful in importing my package.

ntoll commented 5 years ago

@MikeTheWatchGuy hi, please make sure you're using the latest "alpha" version of the upcoming 1.1 version of Mu. You can download it from the "downloads" page (see the red box) here: https://codewith.mu/en/download

siraj-samsudeen commented 4 years ago

Hi I am new to python and following the advice given by Al Sweigart in his Automate the boring stuff with Python book, I started using Mu Editor. I enjoy the fact it is quite easy to use. But I am not able to use third party packages. When I try to add any package, I get the following error - how do I resolve this? pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available.

carlosperate commented 4 years ago

Hi @ssamsudeen,

Are you using macOS? If so it is likely the issue discussed here: https://github.com/mu-editor/mu/issues/910 We are aiming to have it fixed by the next Mu release. In the meantime you can try to install openssl with brew:

$ brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/663e4792e1f355ef2571a06a9101606027cfedf9/Formula/openssl.rb

paulbaumgarten commented 4 years ago

Is there any indication when this feature will be available in the "stable" download? Or is the alpha considered stable enough for use in a classroom setting on student controlled laptops? Thanks

tjguk commented 4 years ago

@paulbaumgarten the alpha is definitely stable. (Even our master on Github is pretty stable). That said, we are actively -- this week and the next -- working towards the fabled 1.1 release which will include the stuff in the alpha plus other bugfixes, definitely including 3rd party package support. And some new modes.

dybber commented 4 years ago

I guess it should also be mentioned that it's hard to say exactly when a 1.1 release will be ready, as we still have to figure out some things to get Mu running on Mac OS Catalina (signing, notarizing, etc.)

mu-editor / mu

Facilitate installation of 3rd party packages for use with Mu #675