pythonw should be default in OS-X

[ChrisBarker-NOAA]

On OS-X, for a process to access the windowing system the executable must be in a "Framework" or an "Application bundle" or somethign or other.

Anaconda addresses this by providing a "pythonw" command that is a shell script that re-directs itself to start up a copy of python that is inside a provided *.app bundle.

However, this can be really a pain. It's not SUCH a big deal to type pythonw when you need to run a GUI app, nor is it that bad to change the #! line on scripts (though that makes it all less system independent -- does linux even HAVE a pythonw in general?

But when this really kills me is when I'm not controlling the start-up script -- in particular, I'm using nose to run tests, some of which require wxPython to be started up. -- the whole test process fails, as nose is not being run with the right python. And going in an hacking nosetests does not seem like the way to solve this (and is less then trivial, actually, I just spent some time trying to figure it out, to no avail)

The python.org builds solved this many releases ago -- they have a small bianry executable that re-directs to a properly set up python -- and this is the default, so "python" and "pythonw" do exactly the same thing. As far as I can tell, this provides no downside of any sort for command line apps, and makes the whole GUI thing a lot easier to deal with. And the pyton.org build has been shipping that way for years, with apparently no complaints from anyone.

I suggest that Anaconda do the same thing -- it would really make things easier for GUI develoment, testing, using MPL with some back-ends, etc, etc, etc....

Whether you use the current shell script re-direct or python,org's executable, I have no opinion on -- whatever works is fine, but it really would make it easier to use Anaconda on OS-X.

Note: if you search the Anaconda list for pythonw, you see this has created a lot of issues and confusion -- maybe most have been solved with a one-character change to a #! line, but it would have been better not to have it at all, yes?

This technical details have been discussed some on the Anaconda list:

https://groups.google.com/a/continuum.io/forum/#!searchin/anaconda/why$20not$20make$20python$20invoke$20the$20framework/anaconda/-ZAynUQW5HQ/6g4_4v96S_cJ

and the python-mac list:

https://mail.python.org/pipermail/pythonmac-sig/2014-August/024079.html

As kludgy as it is, the bootstrap executable looks quite usable.

I hope we can get his cleaned up.

[asmeurer]

I think the executable would be preferable. There may be situations where the shell script does not work.

[ChrisBarker-NOAA]

I agree -- that seems the more robust approach -- and it's worked for many years and releases in the python.org build.

[ChrisBarker-NOAA]

Ping!!

Could you all please fix this! I guess there isn't a lot of desktop development going on with Anaconda, but really, the python.org build had this right years ago.

BTW -- mostly I've just been using $%^$ pythonw, but right now I'm messing around with a little wxPython app in a package and found:

Another developer did "the right thing" and set it all up with "entry_points" and setuptools -- nice!

But now I can't run the $%^$ app directly from the command line, because relative imports don't work if I do that.

And I can't run it with the start up script setuptools puts in because it doesn't use "pythonw" .

I'll agree that this is probably a setuptools bug, but as you can use "python" on every other platform, and can use it with the python.org build on OS-X -- we really should be able to use it with Anaconda Python too.

sigh.

[ChrisBarker-NOAA]

NOTE:

https://bitbucket.org/pypa/setuptools/issues/410/the-gui_scripts-entry-point-should-use

Seems to indicate that pythonw can not be counted on existing, so setuptools won't use it (though I'm a bit confused, as it really should use it on Windows (don't you get an extra command window poped up if you don't?)... not sure what it does there)

[sbourdeauducq]

don't you get an extra command window poped up if you don't?

On Windows, python pops up a console window and pythonw cannot do any console I/O.

We have a package that has both console and GUI scripts, and only the former should pop console windows. While this sounds innocuous enough, the lousy design of conda/setuptools makes it impossible in reality. Very frustrating...

[ilanschnell]

lousy design of conda/setuptools

Which one? Conda and setuptools are two completely separate projects.

[ChrisBarker-NOAA]

Os-X does not pop up terminal Windows at all. But pythonw is required to access the window manager.

CHB

On Jun 18, 2016, at 10:40 PM, Sébastien Bourdeauducq < notifications@github.com> wrote:

don't you get an extra command window poped up if you don't?

On Windows, python pops up a console window and pythonw cannot do any console I/O.

We have a package that has both console and GUI scripts, and only the former should pop console windows. While this sounds innocuous enough, the lousy design of conda/setuptools makes it impossible in reality. Very frustrating...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/199#issuecomment-226976038, or mute the thread https://github.com/notifications/unsubscribe/AA38YCkiAowRhivTTRGwqXeDwNnS64nAks5qNKwigaJpZM4C2gQ2 .

[ChrisBarker-NOAA]

On Jun 18, 2016, at 10:45 PM, Ilan Schnell notifications@github.com wrote:

lousy design of conda/setuptools Which one? Conda and setuptools are two completely separate projects.

Indeed. And neither of them had design issues with regard to this issue, but rather, specific decisions that are not ideal:

Setuptools should use pythonw appropriately in scripts ( entry points )

Conda ( really Anaconda) should have pythonw and python be the same thing on OS-X.

Not sure what the "right" thing is on Windows.

CHB

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/199#issuecomment-226976168, or mute the thread https://github.com/notifications/unsubscribe/AA38YHG2VBnIX3wNORJDk0sT5o-lDRcrks5qNK1DgaJpZM4C2gQ2 .

[sbourdeauducq]

neither of them had design issues with regard to this issue

Is there any way to select pythonw or python for specific entry_points in meta.yaml?

[sbourdeauducq]

Ping?

[ChrisBarker-NOAA]

On Sat, Jun 18, 2016 at 10:42 PM, Sébastien Bourdeauducq < notifications@github.com> wrote:

neither of them had design issues with regard to this issue

Is there any way to select pythonw or python for specific entry_points in meta.yaml?

entry points are totally a setuptools thing, yes?

though I suppose conda could kludge in a patch after setuptools does it's thing -- ugly but it could work.

-CHB

—

You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/199#issuecomment-226977653, or mute the thread https://github.com/notifications/unsubscribe/AA38YG_kai9CiYpCD0UCVGjhWFCf786Vks5qNLqIgaJpZM4C2gQ2 .

Christopher Barker, Ph.D. Oceanographer

Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception

Chris.Barker@noaa.gov

[sbourdeauducq]

No, http://conda.pydata.org/docs/building/meta-yaml.html#python-entry-points

How do I create a package that has some pythonw entry points, and some python entry points?

[kalefranz]

Is there any way to select pythonw or python for specific entry_points in meta.yaml?

Not that I'm aware of. But it seems like a reasonable request. Maybe @msarahan would consider adding this to conda-build?

Or maybe @ilanschnell has some other ideas?

[sbourdeauducq]

Not that I'm aware of.

Hence my "lousy conda design" comment. This feature is sloppy and not thought through, and now if we want to replace it with something proper, we have to deal with backward compatibility.

[hoechenberger]

Has any progress on this issue been made in the meantime? The current solution causes lots of frustration.

[andreatramacere]

is there any progress on this issue?

[newville]

Any progress on this very irritating and long-standing issue?

[ChrisBarker-NOAA]

No :-(

I think the core problem is that no one has been motivated enough to do the work.

My thoughts on the way forward:

conda-forge is a way to get new packages out to users, and also a potential route to getting something different into defaults -- once porven useful, there is at least a chance of that.

Here:

https://github.com/conda-forge/python-feedstock

is the conda-forge "feedstock". So, if you have the inclination (and time), I would:

fork that feedstock.

update the recipe for os-x to build the pythonw.exe.

do a PR to the feedstock, and see what happens. :-)

NOTE: the challenge for updateing the feedstock is that the python.org build scripts are only set up to do particular build combinations (or were, I haven't looked in a while):

1) plain old *nix builds (what conda is using now)

2)"framework" builds -- which put all of python inside ans OS-X Framework. In addition, the Framework builds were set to build "fat" binaries (32+64 bit) -- not sure if that's still the case, but it was a complication.

The framework builds included the pythonw executable hack. There is actually no reason that the pythonw executable has to be part of a framework build -- those two things are orthogonal, but separating them out will require some build script hacking.

I'd also note that Ned Deily (who has been managing the builds for python.org) is sympathetic -- I imagine he'd help out if someone gives it a try.

[newville]

@ChrisBarker-NOAA thanks for the interesting perspective and ideas.

I was not aware that it might be possible to decouple "framework" and "pythonw". I agree with you that this is probably the right way forward. I very much doubt I have the expertise to do this.

I think that using conda-forge would be less than ideal. This issue really is for ContinuumIO / Anaconda.com to solve. There have been numerous complaints over the past few years, and they continue to ship a faulty product: they claim they support Mac OSX, but their version of Python is crippled on Mac OSX in a way has been solved by others.

So, I think your lead-off comment gets to the heart of the matter: the folks at Anaconda.com are not motivated to properly support Mac OSX.

[ChrisBarker-NOAA]

@newville

Yup, the challenge is that someone with auto cons expertise needs to do it :-( which I’m not either.

As for conda-forge — it IS a way that could lead to getting into defaults. As we can’t task Continuum staff, it’s pretty much the only way :-)

[newville]

@ChrisBarker-NOAA

I have no experience with Frameworks. I hadn't ever really looked at the Mac-specific Python sources before. A quick look suggests that maybe https://github.com/python/cpython/blob/master/Mac/Tools/pythonw.c

is not really that complicated.

One thing that I never quite understood is: Why can't "pythonw" (which Anaconda does provide) just be used in place of "python" for all python needs?

Currently, "pythonw" is a shell script. It appears to be written at install time because it actually hardwires in the installation directory. For one install for me, it is written as

!/bin/bash

export PYTHONEXECUTABLE=/Users/Newville/anaconda3/bin/python /Users/Newville/anaconda3/python.app/Contents/MacOS/python "$@"

I would have thought that they would have been better off not hard-wiring that in and writing that as

#!/bin/bash
TOP=${BASH_SOURCE%/*/*}
export PYTHONEXECUTABLE=$TOP/bin/python
$TOP/python.app/Contents/MacOS/python "$@"

Maybe Anaconda.com has a reason to prefer the first. Bash is subtle, especially regarding symlinks. I am reluctant to suggest they didn't know any better ;).

So, why can't this simply be used as "python"?

Reading the above-linked pythonw.c, I noticed that it also sets an environment variable of '__PYVENV_LAUNHCHER__'. I didn't have to go very far to see that other tools such as pip rely on this env var: perhaps that is why "pythonw" is not suitable for "python"?

Perhaps the Python in python.app could be altered to set __PYVENV_LAUNCHER__ appropriately and then be used as python?

Somehow, I think the Issue should really not be that hard to work out. The lack of effort or attention from Anaconda.com is disappointing.

[ChrisBarker-NOAA]

All I know is that the Anaconda pythonw shell script doesn't work with how setuptools builds script entry points -- so that's a killer anti-feature.

I don't understand the vagaries of shell script forking and re-direction to have any idea if that's fixable.

But I do know that the pythonw.c executable works fine as a replacement for the regular pythonw in, as far as know, ALL places.

And we have build scripts for it, we just need to pull them apart to get pythonw without the Framework.

I agree, I wish Continuum would do this, but apparently they don't have a lot of customers clamoring for it.

-CHB

[msarahan]

Insulting us, making assumptions about us, and telling us what to do aren't going to be successful strategies for getting improvements. @newville please keep your comments less incendiary if you would like us to respond positively (or at all). Keep in mind that you're getting free support for a free product, and that everyone can help improve the system by contributing PRs to recipes either at conda-forge or at https://github.com/anacondarecipes/ which is our fork of conda-forge that we've added our new compilers to. Our team is very small, and we're stretched very thinly.

@ChrisBarker-NOAA, I'm sorry you've been chasing this for so long with no result. If the problem is addressed at conda-forge, we'll be happy to merge in the changes into defaults. In the meantime, I'll try to re-raise this issue. No promises on any timeline, but I'll try to at least keep it on the radar.

[jakirkham]

FWIW a similar discussion on conda-forge has occurred in issue ( https://github.com/conda-forge/python-feedstock/issues/23 ). Had hoped there was something we might learn from this package. I don't know that anyone ever pursued it. Jonathan also tried taking some patches from MacPorts, but it didn't work out in the end.

From skimming the pythonw.c code in CPython, it looks mostly like a bunch of path manipulation to find python. There's some other stuff in there, but it's not obvious to me how it helps.

Does anyone know who the original author or perhaps the current maintainer of that code is? Maybe we can get additional insight from them.

[newville]

@msarahan I am surprised to read that saying things such as "the lack of effort is disappointing" or "the folks at Anaconda are not motivated to work on this" can possibly be characterized as "incendiary". Anaconda Python is crippled on Mac OS X. If you need to portray me as a bad guy for saying that, I'm happy to provide that service.

This Issue is number 199, and is more than three years old. Multiple people have pinged the issue over the years. The problem has been discussed on the Anaconda public mailing lists going back approximately that far too. I don't know who is or is not part of the Continuum/Anaconda team so please correct me if I am wrong, but it sort of appears to me that yours is the first response from anyone associated with Anaconda that is longer than one sentence. Maybe complaining loudly actually works... ;).

You don't have the resources... we understand. We also have limited resources. We are very grateful for the work done by Anaconda, and do our best to support and promote it. Still, having mailing lists and public trackers and then ignoring long-standing issues is going disappoint your users.

FWIW, I have asked for support on the paid support channel as well.

[newville]

@jakirkham I think that the path manipulation to find python in pythonwc.c may all be to set the value of __PYVENV_LAUNCHER__, which appears to be related to the problems referenced for pip and setuptools. Perhaps this is all that is really necessary?

It seems that doing

#!/bin/bash
TOP=${BASH_SOURCE%/*/*}
export PYTHONEXECUTABLE=$TOP/bin/python
export __PYVENV_LAUNCHER__=$TOP/bin/python
$TOP/python.app/Contents/MacOS/python "$@"

is not sufficient. But maybe setting __PYVENV_LAUNCHER__ in the version of Python built as part of Python.app would work? I'm not sure.

I wonder if one can simply add that to the Python that is built as part of python.app, without needing the shell script wrapper? Maybe that's too simple....

[jakirkham]

Do you have an MRE that we can test it on? If so, I think we should take your theory for a spin in the python.app feedstock.

[msarahan]

@newville if you have a paid support contract, then we've definitely fallen down on our support effort. Please email me and I'll look into how that happened. msarahan@anaconda.com

[bryevdv]

@newville The specific comment that ired me personally was:

The lack of effort or attention from Anaconda.com is disappointing.

A very small group of people has put in many 50, 60, and sometimes 70 hour weeks over the last five years, to give Anaconda to the world, for free. If you want to criticize prioritization, thats one thing (tho FWIW as a mac user, I have never needed pythonw, not once, in 4+ years). Saying "lack of effort" however, is hard to accept with a smile.

[mingwandroid]

What got me was:

the folks at Anaconda.com are not motivated to properly support Mac OSX.

In my opinion this is incorrect in two ways:

1. We are motivated to properly support Mac OSX.
2. we do not consider the current Mac OSX to be currently not properly supported. I use macOS as my primary dev machine and run Miniconda on it every day without once in nearly 2 years having wished for pythonw.

and:

Still, having mailing lists and public trackers and then ignoring long-standing issues is going disappoint your users.

This implies that we ignore these things deliberately when the reality is we have far too much to do and consider this lower priority than those other things.

To your specific points.

Before @jakirkham goes ahead and breaks pythonw attempting to fix a non-issue:

#!/bin/bash
TOP=${BASH_SOURCE%/*/*}
export PYTHONEXECUTABLE=$TOP/bin/python
$TOP/python.app/Contents/MacOS/python "$@"

1. Will not work for anyone not using bash and sources their python script or prefixes it with zsh (zsh pythonw -c import os; print(os.environ['PYTHONEXECUTABLE']) gives /bin/python)
2. Will not clean up redundant .. values (e.g. /opt/conda/envs/npy/bin/../bin/pythonw)
3. Will result in a relative PATH if launched from the root dir via ./opt/conda/envs/npy/bin/pythonw
4. The cost of such a tiny shell script is very minor and given the above issues it avoids it is a very sensible approach.

The __PYVENV_LAUNCHER__ stuff could be useful but should be added to what we use already. @newville, I'd like some references to the setuptools and pip issues you hope __PYVENV_LAUNCHER__ will improve. I know that the most major point of incompatibility between pip and conda (which is that pip overwrites hardlinked files breaking all environments .. a fine example of an issue much closer to the top of my priority list than this one) is not related to __PYVENV_LAUNCHER__.

One thing that I never quite understood is: Why can't "pythonw" (which Anaconda does provide) just be used in place of "python" for all python needs?

pythonw (really python.app) links to the macOS CoreFoundation Framework:

otool -L /opt/conda/envs/npy/python.app/Contents/MacOS/python
/opt/conda/envs/npy/python.app/Contents/MacOS/python:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 853.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

.. which links to a load of system libraries:

otool -L /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation:
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1445.12.0)
    /usr/lib/libDiagnosticMessagesClient.dylib (compatibility version 1.0.0, current version 1.0.0)
    /usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 59.1.0)
    /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)

In general, linking to libraries that are not needed is problematic and is called over-linking. Concretely, pulling in these system libraries means that some software might not load Anaconda Distribution libraries because it will call dlsym to see if some symbol they need is loaded and will then avoid calling dlopen. We recently ran into this issue with MKL and Qt.

Regardless, I agree that gui scripts should be called through pythonw.

[newville]

On Sun, Dec 10, 2017 at 3:12 AM, Ray Donnelly notifications@github.com wrote:

What got me was:

the folks at Anaconda.com are not motivated to properly support Mac OSX.

In my opinion this is incorrect in two ways:

1. We are motivated to properly support Mac OSX.
2. we do not consider the current Mac OSX to be currently not properly supported. I use macOS as my primary dev machine and run Miniconda on it every day without once in nearly 2 years having wished for pythonw.

Well, if you don't use some feature or library, you are not likely to see problems with those features or libraries. For sure, not every program has a problem running with "python" instead of "pythonw". But some people do experience the Issue here, and have pinged the issue multiple times. It seems that the principal factor is wxPython. I don't know if that is the only case, or if there is a viable solution for packaging wxPython for Mac OSX in a way that does not require running with "pythonw".

This is not a wish for pythonw -- that exists. The idea is to not have to distinguish "python" from "pythonw" but to have one program called "python" that can run all python programs.

and:

Still, having mailing lists and public trackers and then ignoring long-standing issues is going disappoint your users.

This implies that we ignore these things deliberately when the reality is we have far too much to do and consider this lower priority than those other things.

Well, it implies that ignoring long-standing Issues will disappoint users. I understand that you set priorities and that this Issue has been a low priority for ContinuumIO / Anaconda.com.

To your specific points.

Before @jakirkham https://github.com/jakirkham wastes his time on your shell script fix for a non-bug:

!/bin/bash

TOP=${BASH_SOURCE%//} export PYTHONEXECUTABLE=$TOP/bin/python $TOP/python.app/Contents/MacOS/python "$@"

1. Will not work for anyone not using bash and sources their python script or prefixes it with zsh (zsh pythonw -c import os; print(os.environ['PYTHONEXECUTABLE']) gives /bin/python)
2. Will not clean up redundant .. values (e.g. /opt/conda/envs/npy/bin/../bin/pythonw)
3. Will result in a relative PATH if launched from the root dir via ./opt/conda/envs/npy/bin/pythonw
4. The cost of such a tiny shell script is very minor and given the above issues it avoids it is a very sensible approach.

OK.

The PYVENV_LAUNCHER stuff could be useful but should be added to what we use already. @newville https://github.com/newville, I'd like some references to the setuptools and pip issues you hope PYVENV_LAUNCHER will improve. I know that the most major point of incompatibility between pip and conda (which is that pip overwrites hardlinked files breaking all environments .. a fine example of an issue much closer to the top of my priority list than this one) is not related to __PYVENV_LAUNCHER__.

Yesterday @Chris.Barker-NOAA (who originated this Issue three years ago) suggested that "Framework build" might be decoupled from "pythonw". Perhaps you missed some earlier parts of the conversation?

I had never looked at that possibility and had never read pythonw.c before that suggestion. A quick read suggested that pythonw.c does little except manipulate __PYVENV_LAUNHCER__. This internally-used env var appears designed to unwrap symlinks for the Python executable in virtual environments, and seems to also have (or maybe "had") implications for distutils as well as for Python on Mac. See, for example the discussions in https://bugs.python.org/issue22490 . The issue here seems a bit different from that one, and it might be that the venv trick that pythonw does is a red-herring. I don't know, but it might be worth further investigation, especially if pythonw and Framework are actually separate concerns.

One thing that I never quite understood is: Why can't "pythonw" (which

Anaconda does provide) just be used in place of "python" for all python needs?

pythonw (really python.app) links to the macOS CoreFoundation Framework:

otool -L /opt/conda/envs/npy/python.app/Contents/MacOS/python /opt/conda/envs/npy/python.app/Contents/MacOS/python: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 853.0.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1197.1.1)

.. which links to a load of system libraries:

otool -L /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation: /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1445.12.0) /usr/lib/libDiagnosticMessagesClient.dylib (compatibility version 1.0.0, current version 1.0.0) /usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 59.1.0) /usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0) /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)

In general, linking to libraries that are not needed is problematic and is called over-linking.

The problem seems to be that sometimes those libraries are needed, requiring "pythonw" to be used instead of "python". Perhaps "python" could be described as "under-linked". ;).

Concretely, pulling in these system libraries means that some software

might not load Anaconda Distribution libraries because it will call dlsym to see if some symbol they need is loaded and will then avoid calling dlopen. We recently ran into this issue with MKL and Qt.

I certainly believe that complications can arise with large third-party libraries. You might want to add "wxPython" in that list of third-party modules that cause trouble loading, only it seems to have the opposite issue, requiring "pythonw" and those CoreFoundation libraries that permit drawing to the screen. I don't know how other GUI libraries handle this. wxPython definitely links to these, but I do not understand why the main interpreter program would also need to link to those in order for a wxPython-using program to draw to the screen.

Somehow, other Python distributions (including Python.org) on Mac OSX have been able to provide a "python" that does not require a separate "pythonw". They do appear to link to the CoreFoundation tools. Maybe that is not possible for Anaconda Python. I don't know why that would be the case. I haven't seen anyone respond "no, we cannot do this", only "it is not a priority".

Regardless, I agree that gui scripts should be called through pythonw.

The Issue here is essentially the desire to run all python programs with the program named "python", and to not distinguish "python" and "pythonw".

Hope that helps.

[PythonCHB]

TL;DR --

The pythonw.c solution is a hack, but it's a hack that has worked for years and MANY versions of python and OS-X, without any problems. It's a least worth a try for Anaconda.

We "just" need someone with the time, skills and motivation to try it out.

And here's some more detail ----

Take a look at the original post on this issue -- I link to a couple other discussions for teh technical details.

But to answer a couple questions:

I'm pretty sure Ronald Oussoren wrote the original pythonw.c code, and Ned Deily is maintaining the mac builds for python.org.

They both can be found on the pythonmac-sig and python-dev lists.

I'm pretty sure Ned is considering re-doing the builds as Non-framework builds at some point, so he may be interested in helping out with all this (and would certainly like to get help :-) )

With regards to setuptools:

1) the setuptools devs don't seem to have any interest in supporting "pythonw" on the mac -- they seem to have the attitude of "Apple is being weird, we don't want to deal with it". And the fact that the current situation "works" for the python.org builds means they don;t think they have to (fair enough)

2) The current Anaconda pythonw script does not work with setuptools' launcher scripts anyway -- in the sense that if you simply replace "python" with "pythonw" -- it doesn't work. I do not understand the technical reasons for that -- maybe there is a simple fix, which would at least get us closer.

Is Anaconda "broken" on the Mac?

Of course not -- it is very widely used very successfully for all sorts of things. However, the current situation is not-very-good for doing desktop GUI development on the Mac with Anaconda. However, apparently, there aren't a lot of us doing that, so it just hasn't gotten a lot of attention. (and the python.app package DOES get it working well enough to get by)

Apple's insistence on a particular structure to access the Window Manager is really annoying, and simply not well suited to an interpreted language like Python. But the pythonw.c hack (and yes Ronald does think it's a hack" has worked very well for the python.org builds. So it seems worth giving a try for conda.

Again, the "framework build" and pythonw.c are orthogonal issues -- either can exist without the other. However, the whole point of pythonw.c is to produce an executable that makes OS-X happy by appearing to be in an Application Bundle. And that may require linking to a bunch of stuff that we may not want for non-GUI apps.

[hmm -- looks like there is no longer a "pythonw" in the the python.org builds (3.6) -- interesting]

In the python.org build, this is what python3.6 is linked to it is linked to:

$ otool -L python3.6
python3.6:
    /Library/Frameworks/Python.framework/Versions/3.6/Python (compatibility version 3.6.0, current version 3.6.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)
MacBook-Pro:bin Chris$ 

Seems pretty minimal, and it works fine with wxPython at least.

[PythonCHB]

A couple specifics:

Regardless, I agree that gui scripts should be called through pythonw.

The Issue here is essentially the desire to run all python programs with the program named "python", and to not distinguish "python" and "pythonw".

right -- the "pythonw" solution has two problems:

1) the rest of the community, notably setuptools, doesn't seem to support that idea :-(

2) the current pythonw shell script provided by the python.app package doesn't work with setuptools' launcher scripts anyway :-(

-CHB

[chogg]

Should I understand from this that I still need to use pythonw? I will be very pleased when this issue gets resolved. Thanks to everyone for their efforts. :D

[ChrisBarker-NOAA]

yes, you still need pythonw -- and it comes with the python.app package.

-CHB

[brainstorm]

Is it just me or pythonw is only python2-based?:

(flatcam)$ pythonw
pythonw     pythonw2    pythonw2.6  pythonw2.7
(flatcam)$ pythonw --version
Python 2.7.10
(flatcam)$ python --version
Python 3.6.4

I'm porting FlatCAM to python3 and pyqt5:

https://bitbucket.org/jpcgt/flatcam/pull-requests/75/initial-port-to-python3-and-pyqt5/diff

And the menu focus issue is the latest UI/UX defect I've left to fix:

https://riverbankcomputing.com/pipermail/pyqt/2016-September/038132.html

From the many suggestions I'm seeing in this thread, which is the most recommended/supported nowadays?

[ChrisBarker-NOAA]

I’m pretty sure pythonw is the same with python3

You need to install “python.app” to get it.

-CHB

Sent from my iPhone

On Feb 16, 2018, at 7:57 PM, Roman Valls Guimerà notifications@github.com wrote:

Is it just me or pythonw is only python2-based?:

$ pythonw pythonw pythonw2 pythonw2.6 pythonw2.7

I'm porting FlatCAM http://flatcam.org/ to python3 and pyqt5:

https://bitbucket.org/jpcgt/flatcam/pull-requests/75/initial-port-to-python3-and-pyqt5/diff

And the menu focus issue is the latest UI/UX defect I've left to fix:

https://riverbankcomputing.com/pipermail/pyqt/2016-September/038132.html

From the many suggestions I'm seeing in this thread, which is the most recommended/supported nowadays?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/199#issuecomment-366414296, or mute the thread https://github.com/notifications/unsubscribe-auth/AA38YIyCGeyrt3fYaloIaEvNa9qfc8X4ks5tVk5AgaJpZM4C2gQ2 .

[brainstorm]

Thanks! As advertised conda install python.app does install the corresponding pythonwfor python3 :)

[OffTopic] Next up for me: how to package it all automatically in a bundle, .dmg or similar for easy install 👍

[PythonCHB]

[OffTopic] Next up for me: how to package it all automatically in a bundle, .dmg or similar for easy install 👍

I think py2app works with conda.

Or you can check out constructor

https://github.com/conda/constructor/blob/master/README.md

-CHB

—

You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ContinuumIO/anaconda-issues/issues/199#issuecomment-366422320, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwXFY-mc-2D4auQqorpCmJKbaEvxmJ_ks5tVngRgaJpZM4C2gQ2 .

-- Christopher Barker, PhD

Python Language Consulting

• Teaching
• Scientific Software Development
• Desktop GUI and Web Development
• wxPython, numpy, scipy, Cython

[czzarr]

how do you tell conda to use pythonw ?

[newville]

@czzarr You can install pythonw with conda install python.app. But you then have to run pythonw instead of python. If you have a script (or an entry-point) to run, you can put #!/usr/bin/env pythonw as the first "shebang line".

FWIW, pythonw is nothing more than

#!/bin/bash
export PYTHONEXECUTABLE=/Users/NAME/anaconda3/bin/python
/Users/NAME/anaconda3/python.app/Contents/MacOS/python "$@"

but simply copying that python exectuable to the bin folder will not fix the problem -- it has to be inside the App.

FWIW, Anaconda.com has stated that they will not fix the problem.

[czzarr]

Thanks @newville . Do you know how to use it with ipython so that I can use %matplotlib osx ?

[newville]

@czzarr I believe one has to edit the bin/ipython and/or bin/jupyter-kernel to use #!/usr/bin/env pythonw.

[cbrnr]

Another caveat of having a separate pythonw as implemented by python.app: In my PyQt-based GUI I'm using multiprocess to spawn a subprocess, but when I terminate it my program segfaults with /Users/clemens/anaconda/bin/pythonw: line 3: 3527 Segmentation fault: 11 /Users/clemens/anaconda/python.app/Contents/MacOS/python "$@". This does not happen with the official Python builds for macOS, where I can just run my program with python (there is no pythonw anymore).

[newville]

@cbrnr Also: It appears that the python executable in python.app is also using Python 3.7.0, which is somewhat (~18 months) behind current python (3.7.6) in Anaconda. I don't know that this would cause problems for multiprocessing, but it might cause some surprises.

[cbrnr]

@newville this is strange, I expected that both python and pythonw should point to the same version of Python. In the meantime, we fixed the problem with multiprocessing.Pool by switching to pebble.ProcessPool. Not quite sure why this works, because in theory both modules should use the same basic mechanisms to manage the process pool...

[hoechenberger]

because in theory both modules should use the same basic mechanisms to manage the process pool...

I don't think that's the case: They both use the same type of processes, but have different implementations of process pools. Still begs the question why multiprocessing.Pool segfaults on pythonw...

[hoechenberger]

I expected that both python and pythonw should point to the same version of Python

I did too, but they're actually different! Wow!!

$ which python
/Users/hoechenberger/miniconda3/envs/mnelab-dev/bin/python
$ python -V
Python 3.7.6
###
$ which pythonw
/Users/hoechenberger/miniconda3/envs/mnelab-dev/bin/pythonw
$ pythonw -V
Python 3.7.3

[newville]

Hm, with both Anaconda3-2019.10 and the latest Miniconda3, I get Python 3.7.0 from pythonw. Is that from conda-forge or something?

For multiprocessing, I believe the problem might be that although pythonw runs /Users/<username>/[miniconda3,anaconda3]/python.app/Contents/MacOS/python which is a Framework-built version of Python, it "cleverly" redefines PYTHONEXECUTABLE to be /Users/<username>/[miniconda3,anaconda3]/bin/python. This is what is held in sys.executable (not pythonw) which is used by multiprocessing to spawn new processes. The result is (I think) that those spawned processes will not be using a Framework-built version of Python.

Multiprocessing might work if you launch with /Users/<username>/[miniconda3,anaconda3]/python.app/Contents/MacOS/python.

[hoechenberger]

Hm, with both Anaconda3-2019.10 and the latest Miniconda3, I get Python 3.7.0 from pythonw. Is that from conda-forge or something?

Yes, using conda-forge here, sorry I forgot to mention that.

Multiprocessing might work if you launch with /Users/<username>/[miniconda3,anaconda3]/python.app/Contents/MacOS/python.

Interesting! I'd specifically pointed multiprocessing to the pythonw executable before, but hadn't tried the Python executable inside the .app bundle.