OpenSSL not statically linked to the packaged Python in macOS

krswan commented 5 years ago

Original title (edited by @carlosperate): Third Party installation (pypercard) error on OSX

Install on most current (just installed) Mu on OSX 10.14.5 failed. Here is the text:

pip is configured with locations that require TLS/SSL, however the ssl module in Python is not available. Collecting pypercard Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/pypercard/ Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/pypercard/ Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/pypercard/ Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/pypercard/ Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)': /simple/pypercard/ Could not fetch URL https://pypi.python.org/simple/pypercard/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.python.org', port=443): Max retries exceeded with url: /simple/pypercard/ (Caused by SSLError("Can't connect to HTTPS URL because the SSL module is not available.",)) - skipping Could not find a version that satisfies the requirement pypercard (from versions: ) No matching distribution found for pypercard

FINISHED

ntoll commented 5 years ago

Could you please attach the logs from Mu to this issue..? Thank you!

(Click on the cog, highlight the text window in the resulting popup, type CTRL-A and the CTRL-C to select all and copy all, then paste the result into a text file somewhere and upload it here)

krswan commented 5 years ago

Here you go:

errorlog.txt

seantibor commented 5 years ago

I was experiencing this same bug earlier this week on student machines. Thinking we may need to include openssl in the macos packaging script? I can reproduce this error if I remove openssl from my computer with the following:

brew uninstall --ignore-dependencies openssl

As a workaround @YMLkris, you should be able to make the third party packages work if you install openssl with homebrew.

brew install openssl

@ntoll Want me to try adding it to the packaging script and submitting a PR for review?

martinohanlon commented 4 years ago

I just came across the same error when installing Mu Alpha on a new macOS installation

martinohanlon commented 4 years ago

@seantibor I installed openssl with brew but I got the same error. Presumably this is because the packaged version of Mu cant access it?

ntoll commented 4 years ago

Hi folks, AFAICT this is because the way we compile Python for Mu on Mac is done without static openssl (i.e. it's compiled into Python). That's the normal way to do it, but I thought (or perhaps imagined) that @carlosperate had figured out how to make it compile. This whole story around packaging up Mu is a rather large blip on my radar at the moment (I'm currently focusing on Windows packaging atm) so I'll try to get to look into this very soon. Sorry that it's causing such problems. :-(

carlosperate commented 4 years ago

That's the normal way to do it, but I thought (or perhaps imagined) that @carlosperate had figured out how to make it compile.

Yeah, I remember compiling openSSL and configuring it to be statically linked to (or included with) the Python bundle. I'll have a look.

ntoll commented 4 years ago

As ever @carlosperate, you're a star. Might be useful to update the build to the latest 3.7 version of Python (which is what I've been working on for the Windows installer). Infinite thanks as always. :-)

seantibor commented 4 years ago

Thank you all! We really appreciate it over in our classroom. Students want to play with requests and bullet and arcade and... and... and...

(as it should be)

Best,

Sean

On Tue, Nov 26, 2019 at 10:48 AM Nicholas Tollervey < notifications@github.com> wrote:

As ever @carlosperate https://github.com/carlosperate, you're a star. Might be useful to update the build to the latest 3.7 version of Python (which is what I've been working on for the Windows installer). Infinite thanks as always. :-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mu-editor/mu/issues/910?email_source=notifications&email_token=AJVTC7EAZCBXIWQJ4XGXE2LQVVANDA5CNFSM4IKI2NGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFGPGPQ#issuecomment-558691134, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVTC7FWRMDW4B5BVXJ66DDQVVANDANCNFSM4IKI2NGA .

-- Sean Tibor Computer Science & Technology Specialist Pine Crest School sean.tibor@pinecrest.edu

carlosperate commented 4 years ago

!Warning Don't brew uninstall openssl from your system if the last time you've brew installed (or pyenv install, or manually installed from source) Python (or any other software that depends on openssl) was over 6 months ago. This is because brew has moved to v1.1, and everything compiled with v1.0 will break after reinstalling openSSL. I've learnt that the hard way 🙃.

Also, brew makes it unnecessarily difficult to install old versions.

To install the old v1.0: brew install https://raw.githubusercontent.com/Homebrew/homebrew-core/663e4792e1f355ef2571a06a9101606027cfedf9/Formula/openssl.rb

Right now I am trying to find a way replicate, as even without openSSL Mu alpha 2 works fine on my system (macOS ships with libreSSL, but I really don't want to uninstall that one, as thing in the OS might depend on it).

carlosperate commented 4 years ago

@martinohanlon if you still have this problem, could you try this in the Python REPL to see if it also throws an error?

>>> import ssl
>>> import _ssl

seantibor commented 4 years ago

Which macOS version are you using? Does Catalina fix this issue?

On Tue, Nov 26, 2019 at 5:45 PM Carlos Pereira Atencio < notifications@github.com> wrote:

@martinohanlon https://github.com/martinohanlon if you still have this problem, could you try this in the Python REPL to see if it also throws an error?

import ssl import _ssl

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mu-editor/mu/issues/910?email_source=notifications&email_token=AJVTC7F3GMWQLFCQ2MQX5KTQVWRJVA5CNFSM4IKI2NGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFHV7KA#issuecomment-558849960, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVTC7FUA3MKXAPKZNKGMGDQVWRJVANCNFSM4IKI2NGA .

-- Sean Tibor Computer Science & Technology Specialist Pine Crest School sean.tibor@pinecrest.edu

martinohanlon commented 4 years ago

@seantibor This was with Catalina.

@carlosperate I am away for a couple of days and without my test MacBook. As soon as I'm back I'll send you the information.

carlosperate commented 4 years ago

I'm on Mojave still, like the original report (Catalina drops support for 32-bit programs, so I'm likely going to stick with Mojave for a while).

@seantibor what OS version were you using?

carlosperate commented 4 years ago

@carlosperate I am away for a couple of days and without my test MacBook. As soon as I'm back I'll send you the information.

Thanks! I'll see if I can find a way to create a macOS VM, but getting a macOS iso looks almost impossible 🤔

seantibor commented 4 years ago

I'm on Mojave still. Same reason: 32-bit app support...

On Wed, Nov 27, 2019 at 5:57 AM Carlos Pereira Atencio < notifications@github.com> wrote:

@carlosperate https://github.com/carlosperate I am away for a couple of days and without my test MacBook. As soon as I'm back I'll send you the information.

Thanks! I'll see if I can find a way to create a macOS VM, but getting a macOS iso looks almost impossible 🤔

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mu-editor/mu/issues/910?email_source=notifications&email_token=AJVTC7FB5HDLTATULS2HTMLQVZHAHA5CNFSM4IKI2NGKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFJD5IY#issuecomment-559038115, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVTC7BJFYDQWI6CYGXXUPDQVZHAHANCNFSM4IKI2NGA .

-- Sean Tibor Computer Science & Technology Specialist Pine Crest School sean.tibor@pinecrest.edu

martinohanlon commented 4 years ago

@carlosperate re ssl imports and python.

I wasnt sure whether you meant the REPL in Mu or Python so I have done both.

In Mu

I get the error:

ImportError: dlopen(/Applications/mu-editor.app/Contents/Resources/python/lib/python3.6/lib-dynload/_ssl.cpython-36m-darwin.so, 2): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
  Referenced from: /Applications/mu-editor.app/Contents/Resources/python/lib/python3.6/lib-dynload/_ssl.cpython-36m-darwin.so
  Reason: image not found

Python

I dont receive an error and it imports correctly.

ntoll commented 4 years ago

@martinohanlon so, at first glance this appears to confirm SSL isn't properly compiled / configured in the version of Python that comes with Mu. I defer to @carlosperate on this since he's the OSX expert. ;-)

Thank you, as always, to all of you for contributing to fixing this problem..! :+1:

carlosperate commented 4 years ago

Thanks @seantibor for the additional info (love your podcast btw!) And thanks @martinohanlon for all the outputs and screenshots, definitely helps a lot!

I've managed to generate a macOS ISO image using some tools, so I was finally able to create an environment to replicate the issue.

I can confirm that while Python itself was being compiled statically, libraries from some of the standard library packages were not, like the ssl module and OpenSSL.

After attempting quite a few things this is proving difficult to achieve without editing the Python build scripts directly, which is not ideal.

I have a couple things more I could try, but to be honest I don't think we should be patching the Python build scripts, as we would have to maintain the patches for every new Python release we want to use (surprisingly there are a couple of projects already doing this, and it looks a bit tedious).

Apart from that my current efforts are targetted towards openSSL, and there are other modules with C libraries that would need patching as well that we haven't found about yet (there is a list somewhere, so if needed we could at least know what else is not statically linked without manually searching for object files and checking their dependencies, but feels like a moot point).

I think we should probably try again to use the briefcase images, as that project has already gone through all this trouble and maintains a very complete collection of build scripts designed specifically to do this. And any issues we encounter we should try to patch them upstream.

@ntoll last time we used the briefcase image some users reported issues, which is why we went back to the image built from mu_portable_python_macos, do you remember what it was? I'm having trouble finding the GH issue.

carlosperate commented 4 years ago

Btw, something like this would make our lives a lot easier:

But it looks like it's going to get rejected.

tmontes commented 4 years ago

I think we should probably try again to use the briefcase images, as that project has already gone through all this trouble and maintains a very complete collection of build scripts designed specifically to do this. And any issues we encounter we should try to patch them upstream.

Exactly my thoughts. I gave this a try and, at first sight, it seems to work nicely. I packaged two macOS variations of Mu, from the current master, built on my system, running macOS 10.12.6 Sierra:

The current, obtained with make macos, using Mu's tailor made Python support package.
A simpler, briefcase-default variation, also obtained with make macos, after removing the --support-pkg=https://github.com/mu-editor/mu_portable_python_macos/... option from the Makefile, which uses briefcase's default Python image from here.

Observations on my dev / build system

Both current and simpler run ok on my system, which has brew's openssl installed.
In particular, both current and simpler happily import ssl at the REPL.

Observations on a freshly installed macOS 10.14.6 Mojave VM

current seems to work, but fails at import ssl pretty much as above: the _ssl extension is dynamically linked to brew-installed openssl which isn't found.
simpler seems to work, and runs import ssl successfully: the _ssl extension and, I suppose, the openssl libs, are statically linked into the underlying Python binary.

current vs simpler comparison

	current	simpler
.app bundle size	331 MB	366 MB
.app bundle file count	10605	9656
`python3` binary size	3007216	9109492
`python3` binary dynamic linked to	2 system libs	6 system libs
shared libs in `lib/python3.6/lib-dynload`	64	10
non-system dependencies in shared libs	2 (`_hashlib` and `_ssl` linked to `brew`'s `openssl`)	1 (`_gdbm` linked to `brew`'s `gdbm`)

Final Thoughts

I hand-trimmed the simpler packaged bundle to bring it down in size:
- Removed duplicate binaries: kept one of python3, python3.6, and python3.6m, sym-linked the others.
- Removed most tools in python/bin like 2to3, pydoc, etc.
- Removed "development" files: include files, statically linked python library, and a few others.
- Deleted all tests in the standard library.
- Compiled the standard library and deleted all *.py files.
- Compiled all of Mu's and its dependencies' modules and deleted the *.py files.
- Ran python -m ensurepip to, you know, have pip available.
Brought it down to 276 MB in 7790 files.
It still ran:
- The minimal REPL-based import ssl test.
- Successfully fetched a web based resource over HTTPS using the standard library's urllib (had to set the SSL_CERT_FILE to a CA file, otherwise cert validation would fail -- not sure which, if any, list of CAs is being considered).
- Installed 3rd party pypercard successfully: failed running minimal program due to underlying OpenGL issues with sdl and the VM environment -- I did not explore further but do believe that if OpenGL "things" were "ready", whatever that means, it would have run successfully.

So, all in all, unless there is some horrible limitation, I'm in favour of using briefcase's Python support package -- which seems to have a 3.7 branch available -- and contributing back any improvements we deem relevant for the wider community.

:)

seantibor commented 4 years ago

@tmontes - do you have instructions for making a build of 1.1.alpha.2 using the briefcase method? I've got a number of students that want to use Mu for distance learning and need to install some additional packages like arcade and numpy. Can you help?

tmontes commented 4 years ago

@seantibor,

I suppose you are referring to the simpler variation I mentioned back in December. Here it goes:

Checkout the current master.
Edit the Makefile and change line 109, reading:

        python setup.py macos --support-pkg=https://github.com/mu-editor/mu_portable_python_macos/releases/download/0.0.6/python3-reduced.tar.gz

...to a simpler:

        python setup.py macos

Note that the whitespace leading to python must be a TAB character -- make requires it to be so.

Once done, ensure you have a freshly created and activated Python 3.6 virtual environment: 3.7 might work, 3.8 probably won't.
Install docutils==0.15.2 (explained later) with pip install docutils==0.15.2.
Install Mu's development dependencies with pip install -e ".[dev]"
Create the packaged macOS Mu bundle with make macos.
Tests and other checks will be run and briefcase will be invoked to package mu.
The resulting mu-editor.app application bundle will be found under the macOS directory.

Keep in mind that distributing this unsigned and non-notarized application bundle might require your target audience to jump through some hurdles in order for their systems to run it -- macOS is getting increasingly picky about running arbitrary binaries.

PS: The docutils==0.15.2 installation step avoids a constraing combination that pip doesn't know how to solve. Somewhat recently, botocore, a library indirectly used by briefcase locked its docutils dependency to <0.16. However, other mu dependencies, also depending on docutils, set other constraints (like >=0.12 and others). The current pip just goes ahead and installs 0.16 (the latest) and later running briefcase leads to an exception otherwise.

Let me know how that goes for you. I've just ran through it on macOS Mojave 10.14.6 and it looks fine up to a point:

Successfully runs import _ssl from the REPL.
Fails at installing third party packages: apparently, briefcase's default Python support package does not include pip... (argh!) :-(

I understand this may not serve your purposes. Keep in mind that numpy is bundled with mu and that you might add arcade to setup.py's install_requires list to have it bundled too, I suppose. :-)

I recall, back in December, exploring this packaging issue for a bit. I know I have a longer set of notes that, IIRC, include: 1. Building CPython from source, 2. Making it relocatable, 3. Installing Mu onto it, 4. Trimming it all down. Not sure if I reached a "deliverable product" by then: maybe the macOS app bundling step could be could be missing... (IOW: you'd get a distributable self-contained mu-editor binary that runs from the CLI on any recent-ish mac, with or without Python, with or without brew, etc.). Let me know if you'd like to take a stab at it -- you'll need to get your hands a little bit dirty, I guess. But might be useful for you. :-)

tmontes commented 4 years ago

@seantibor,

Just after having submitted my previous note I reviewed my December comment and said "wait, we can do better". Or, at least, in a more approachable way.

The procedure I just described ensures that mu bundles a Python distribution that handles secure network transfers -- a requirement to grab packages from pypi.org -- by properly linking to and bundling the underlying bits that the default mu packaging for macOS is missing.

However, like I said, this Python distribution -- briefcase's default -- does not include pip. Hmmm... Interesting! The mu way has pip but might fail at secure network transfers, the briefcase way fails the other way around.

Here's yet another hack that seems to work around the "default" scenarios above. Here's the plan:

Running the procedure in my previous comment will bring in the default Python distribution used by briefcase, caching it in a file at $HOME/.briefcase/Python-3.6-macOS-support.b7.tar.gz.
We will hack that Python distribution and install pip on it.
Then, we'll make clean and run the make macos step again that will now use the "same cached" Python distribution with pip now in place.

Step by step:

Run the procedure in my previous comment.
Launch mu-editor.
Confirm that import _ssl works at the REPL.
Confirm that evaluating _ssl after the import returns <module '_ssl' (built-in)>.
Confirm that installing a third party package fails with No module named pip.
Quit mu.
Run make clean.
Installing pip into briefcase's Python runtime: 8.1. Create a clean working directory: mkdir $HOME/horrible-hack. 8.2. Change the current working directory there: cd $HOME/horrible-hack. 8.3. Extract its contents with tar xzf $HOME/.briefcase/Python-3.6-macOS-support.b7.tar.gz. 8.4. Install pip into that Python distribution: ./python/bin/python3.6 -m ensurepip. 8.5. Repack the Python distribution into the cache: tar czf $HOME/.briefcase/Python-3.6-macOS-support.b7.tar.gz .. 8.6. Cleanup after yourself: cd && rm -r $HOME/horrible-hack.
Change the working directory back to mu's repository root and rerun make macos.
The newly packaged mu-editor.app application bundle should now be able to install third party packages from PyPI on any recent-ish macOS, regardless of whether or not Python/brew/whatever is installed.
Keep in mind the macOS binary distribution constraints I mentioned before: this result is unsigned and non-notarized so your users might need to give it a nudge such that it runs. :-)

PS: Optionally update pip to the latest release with ./python/bin/python3.6 -m pip install --upgrade pip between 8.4. and 8.5.... Should work!

Wrapping it up, I just tested this all on my dev system running macOS Mojave 10.14.6. I successfully installed and imported arcade but did nothing else (NB: had to install dataclasses too, given that briefcase's Python distribution is 3.6).

Let me know of your progress. :-)

seantibor commented 4 years ago

This is AWESOME. I will let you know how it goes. I'll start working on it this afternoon and let you know how it goes. It should be ready in the next few hours.

On Tue, Mar 17, 2020 at 3:43 PM Tiago Montes notifications@github.com wrote:

@seantibor https://github.com/seantibor,

Just after having submitted my previous note I reviewed my December comment and said "wait, we can do better". Or, at least, in a more approachable way.

The procedure I just described ensures that mu bundles a Python distribution that handles secure network transfers -- a requirement to grab packages from pypi.org -- by properly linking to and bundling the underlying bits that the default mu packaging for macOS is missing.

However, like I said, this Python distribution -- briefcase's default -- does not include pip. Hmmm... Interesting! The mu way has pip but might fail at secure network transfers, the briefcase way fails the other way around.

Here's yet another hack that seems to work around the "default" scenarios above. Here's the plan:

1.

Running the procedure in my previous comment will bring in the default Python distribution used by briefcase, caching it in a file at $HOME/.briefcase/Python-3.6-macOS-support.b7.tar.gz. 2.

We will hack that Python distribution and install pip on it. 3.

Then, we'll make clean and run the make macos step again that will now use the "same cached" Python distribution with pip now in place.

Step by step:

Run the procedure in my previous comment.

Launch mu-editor.

Confirm that import _ssl works at the REPL.

Confirm that evaluating _ssl after the import returns <module '_ssl' (built-in)>.

Confirm that installing a third party package fails with No module named pip.

Quit mu.

Run make clean.

Installing pip into briefcase's Python runtime: 8.1. Create a clean working directory: mkdir $HOME/horrible-hack. 8.2. Change the current working directory there: cd $HOME/horrible-hack . 8.3. Extract its contents with tar xzf $HOME/.briefcase/Python-3.6-macOS-support.b7.tar.gz. 8.4. Install pip into that Python distribution: ./python/bin/python3.6 -m ensurepip. 8.5. Repack the Python distribution into the cache: tar czf $HOME/.briefcase/Python-3.6-macOS-support.b7.tar.gz .. 8.6. Cleanup after yourself: cd && rm -r $HOME/horrible-hack.

Change the working directory back to mu's repository root and rerun make macos.

The newly packaged mu-editor.app application bundle should now be able to install third party packages from PyPI on any recent-ish macOS, regardless of whether or not Python/brew/whatever is installed.

Keep in mind the macOS binary distribution constraints I mentioned before: this result is unsigned and non-notarized so your users might need to give it a nudge such that it runs. :-)

PS: Optionally update pip to the latest release with ./python/bin/python3.6 -m pip install --upgrade pip between 8.4. and 8.5.... Should work!

Wrapping it up, I just tested this all on my dev system running macOS Mojave 10.14.6. I successfully installed and imported arcade but did nothing else (NB: had to install dataclasses too, given that briefcase's Python distribution is 3.6).

Let me know of your progress. :-)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/mu-editor/mu/issues/910#issuecomment-600262381, or unsubscribe https://github.com/notifications/unsubscribe-auth/AJVTC7FJHSJGLTTAFNXCCILRH7HGTANCNFSM4IKI2NGA .

-- Sean Tibor Computer Science & Technology Specialist Pine Crest School sean.tibor@pinecrest.edu

dybber commented 3 years ago

This was fixed by @tjguk's work on using a virtual environment in #1072, which is now part of master and will be in the next beta release (coming up soon), so I'm closing this issue.

carlosperate commented 3 years ago

Just a heads up that this issue with the missing openSSL library with the Python interpreter has been fixed in the latest release: https://github.com/mu-editor/mu/releases/tag/1.1.0-beta.2 🎉

seantibor commented 3 years ago

We just tested it in class yesterday and confirmed it's working beautifully.

ntoll commented 3 years ago

Yay @seantibor :tada:

mu-editor / mu