installing formula with python requirement and system python

[NikolausDemmel]

So with the recent python changes, bottles for formulae that require python (depends_on :python). I thought the idea was that when the user does not use brewed python, these formulae are built from source. It turns out this does not happen automatically. Rather, I have to specify --build-from-source manually, otherwise brewed python is pulled in.

This is especially awkward if the python requirement is only a recursive requirement. It turns out, this is not only inconvenient but actually broken:

brew deps gtk+ -1 reveals no python dependency brew deps gtk+ --tree indicates a recursive :python dependency via gobject-introspection.

Assume I have a setup with brewed python and gtk+ installed. I now do brew uninstall python gtk+. Following this with brew install gtk+ would pull in brewed python as a dependecy. However if instead I execute brew install gtk+ --build-from-source, it does not pull in brewed python (but gobject-introspection is still installed from bottle). The fact that I install gtk+ from source should have no influence on the choice of brewed vs system python.

[MikeMcQuaid]

I understand your frustration but there will always be edge cases. I'm trying to bottle against the system Python as much as possible but I haven't finished my work here yet and anything that provides bindings and has a bottle will be using a Homebrew Python unless you manually build that individual package from source.

[MikeMcQuaid]

I'll try and investigate the gtk+ case more though. It would be useful if you were able to try and submit PRs to improve this behaviour, though. I'm more than happy to help you along the way with any questions you may have but ultimately the problem is that I don't actually use Python or Python bindings personally.

[NikolausDemmel]

Thanks for your response. Yes, hombrew and python can still be a PITA, but I believe it is going in a good direction. I am extremely grateful for all the work you put it. I usually don't have time to dig into these issues (I am spending a lot of time on other open source projects already), but I at least want to raise these issues here and point at use cases that are poorly supported. I know PR's, even half-baked, would be more useful :-/

First of all, can we agree on how this should work?

1. Ideally there would be bottles for system and brewed python; but since multiple bottles per formula are not possible, IMO the conclusion reached for now to bottle against brewed python and try to minimize the number formulae that depend on python by default is reasonable.
2. For a user of brewed python things seem to be good. She can just install stuff, using bottles for python dependent packages. The only downside is that the formulae that (now) have an optional dependency on python cannot be installed from bottle with python support. If you need the python binding, you need to install with the accordning option and thus from source. I find that acceptable; the only one I have personally found to be a hassle is boost, because it takes so long to compile. (Someone is looking into breaking out Boost.Python into a serparate formula I believe in #30957.) I personally use brewed python and am quite happy with it, but I cannot fully recommend it because of the broken FindLibPython cmake module (#25118)
3. For a user of system python, it seems that currently, whenever installing something that depends on python (possibly recursively), I have to go and find all the formulae in the dependency tree that actually depend on python, install those manually with --build-from-source, and then install the original formula (from bottle)
4. Scrap 3., it does not even work. In the gtk+ example, if I install gobject-introspection (which depends on python) from source, and then want to install gtk+ from bottle (which does not depend on python directly, but only via gobject-introspection), it still pulls in the python formula as a dependency. So what I have to do as a system python user is build everthing (recursively) depending on python from source. That is quite a list (*)
5. Back to what we would like to have: Is it reasonable to expect brew install gtk+ with system python to automatically install only gobject-introspection from source, and then proceed to use bottles for everything not directly depending on python?

(*)

$ brew uses python --recursive
Error: Failed to import: ignition-transport
No available formula for cppzmq
Please tap it and then try again: brew tap bertjwregeer/compat
Error: Failed to import: pygame
No available formula for smpeg
Please tap it and then try again: brew tap homebrew/headonly
Error: Failed to import: zlib
No available formula for zlib
ansible14               goocanvas
argus-clients               gource
astrometry-net              gpredict
at-spi2-atk             gqview
atk                 graph-tool
atkmm                   grass
atram                   gsettings-desktop-schemas
binwalk                 gsmartcontrol
blast                   gssdp
bochs                   gst-libav
boost-libstdcxx             gst-plugins-bad
boost149                gst-plugins-base
byobu                   gst-plugins-good
cardpeek                gst-plugins-ugly
cattle                  gst-python010
cegma                   gstreamer
cgap                    gtk+
clutter                 gtk+3
clutter-gst             gtk-chtheme
clutter-gtk             gtk-doc
cogl                    gtk-engines
cpyrit-cuda             gtk-gnutella
daemonlogger                gtk-murrine-engine
ddar                    gtkdatabox
diff-pdf                gtkglarea
diffuse                 gtkglext
dnsperf                 gtkmm
easy-tag                gtkmm3
emscripten              gtksourceview
fceux                   gtksourceview3
fragroute               gtksourceviewmm
frescobaldi             gtksourceviewmm3
fsv                 gtkspell3
gabedit                 gtkwave
gal-sim                 gtsam
ganglia                 gupnp
gazebo                  gupnp-av
gazebo2                 gupnp-tools
gazebo3                 gwyddion
gbdfed                  hamsterdb
gcab                    hexchat
gdal                    highlight
gdal-grass              homebank
gdk-pixbuf              honeyd
gdmap                   htqc
geany                   ideviceinstaller
gearman                 ifuse
geda-gaf                ignition-msgs
geeqie                  influxdb
genometools             inkscape
gerbv                   innoextract
ggobi                   insighttoolkit
giira                   ios-webkit-debug-proxy
git-cola                iptux
gkrellm                 itstool
glade                   jigdo
glib-networking             joinx
gmediaserver                json-glib
gmt                 json_spirit
gmt4                    kernagic
gnome-doc-utils             klavaro
gnome-icon-theme            konoha
gnumeric                lablgtk
gobby                   lasi
gobject-introspection           ledger
goffice                 libbtbb
libdnet                 mtl
libetonyek              mysql-connector-c++
libgda                  nanopb-generator
libgee                  newt
libglade                ninja-ide
libglademm              node
libgnomecanvas              nordugrid-arc
libgnomecanvasmm            numpy
libimobiledevice            oath-toolkit
libinfinity             octave
liblas                  ogre
libmagic                ogre1.8
libmongoclient              ogre1.9
libmspub                ola
libnice                 ooni-probe
libodfgen               opam
libplist                open-scene-graph
libpst                  openbr
librasterlite               opencv
librevenge              opencv-libstdcxx
librsvg                 openimageio
libsequence             opensaml
libsoup                 openslide
libspatialite               orfeo
libswiften              ori
libtorrent-rasterbar            osgearth
libvisio                osm-pbf
libwpd                  osm2pgrouting
libwpg                  pango
libwps                  pangomm
libxml2                 pangox-compat
libxmlsec1              paps
libxslt                 pcb
lilypond                pcl
llvm31                  pdal
llvm32                  pdf-tools
llvm33                  pdf2htmlex
llvm34                  pdf2svg
llvm35                  pdfgrep
logstalgia              pdftoipe
luabind                 pdns
macvim                  pdnsrec
maker                   pgrouting
mal4s                   pidgin
mapnik                  pillow
mapnik071               pktanon
mapserver               player
mat                 plplot
matplotlib              plustache
matplotlib-basemap          points2grid
mbsystem                poppler
mediatomb               postgis
meld                    postgis20
metaproxy               postgresql92
midgard2                povray
mira                    pow
mkvdts2ac3              prokka
mkvtomp4                prooftree
mkvtoolnix              protobuf
mkvtoolnix58                protobuf-c
mlpack                  pspp
mlt                 py2cairo
mobile-shell                pyassimp
mongodb-dev             pydoop
monotone                pyexiv2
msitools                pygobject
pygobject3              swatchbooker
pygtk                   swish-e
pygtkglext              sylpheed
pygtksourceview             synfig
pykep                   synfigstudio
pylucene                szl
pymol                   tcpflow
pypcap                  tegh
pyqt                    telepathy-gabble
pyqt5                   telepathy-glib
pyqwt                   telepathy-idle
pyrit                   telepathy-mission-control
pyside                  terminator
pyside-tools                therion
python-dbus             thrift
qgis                    timbl
qiv                 tophat
qscintilla2             trans-abyss
quantlib                transmission-remote-gtk
radare2                 treeline
rcsslogplayer               ufraw
rcssmonitor             upnp-router-control
rcssserver              urdfdom
repeatmasker                usbmuxd
repeatmodeler               v8
retext                  vera++
rethinkdb               viewglob
rmblast                 vim
root                    vips
rpm                 virtualpg
rrdtool                 vowpal-wabbit
sailfish                vpython
scantailor              vte
scapy                   vte3
scipy                   vtk
sdcc                    vtk5
sdformat                wopr
sfcgal                  wps2odt
shiboken                writerperfect
simgrid                 xastir
simple-tiles                xchat
sip                 xml-tooling-c
slimerjs                xournal
smartsim                xplanetfx
snort                   xsane
source-highlight            xulrunner
spatialite-gis              yaml-cpp
spatialite-gui              yarp
spatialite-tools            z3
spdylay                 zbackup
sqliteman               zbar
srecord                 zenity
submarine               zmap
suricata

[NikolausDemmel]

cc @wjwwood fyi

[MikeMcQuaid]

Is it reasonable to expect brew install gtk+ with system python to automatically install only gobject-introspection from source, and then proceed to use bottles for everything not directly depending on python?

No, I don't think so I'm afraid. We're supporting system Python for stuff that doesn't create bindings (i.e. applications like ansible) and Homebrew Python for others where we need the bindings on by default for other bottled packages. We'll review PRs to make depends_on :python => :optional instead of :recommended or always enabled for various packages but we can't handle your particular use-case, I'm afraid.

[NikolausDemmel]

I see, I actually forgot about the bottles against system python for stuff without bindings (good idea IMO).

Assuming I were able to install that combination as described in 3. above. Is there a technical reason that a formula with only indirect dependency on python (like gtk+) would not work if installed from bottle when the dependency that brings in python (gobject-introspection) is installed from source (and thus against system python)?

[MikeMcQuaid]

There's probably not a technical reason but it's probably not something we'll support as it's a pretty niche case.

[NikolausDemmel]

Is it really a niche case? I was not suggesting that 3. is the way it should work. I wanted to know if there are technical hurdles for supporting 5.

My main concern is that people who want to stick with system python have to carefully watch out for formulae that have a recursive dependency on python somehow and make sure to manually specifying --build-from-source. If they don't, brewed python will be installed. There are quite a few of them (see the list above), whereas I counted only 3 formulae with a hard dependecy on the python formula across all my taps. I'm particularly worried about users that might not even know about the choice between brewed and system python. They start of with system python as the default. That's ok. Now at some point they install a formula that has nothing to do with python, e.g. gtk+. Now they have switched to brewed python, probably without even realizing. Do you think this is a niche use case?

[MikeMcQuaid]

If you have system Python installed you can still use Homebrew Python and vice-versa. If you want Homebrew Python to never be installed the only way around this is to always build-from-source.

[NikolausDemmel]

System python is always installed. Of course you can still use it after installing brewed python, but the default python executable has changed, which has all sorts of ramifications for formlae with python bindings (possibly installed from source before the inadverted switch to brewed python) and pip packages.

Do I understand you correctly that the default situation (not-niche case) for homebrew & python is -- and in your opinion should be -- that most users start out with system python only, and then (consciously or not) eventually switch to brewed python as the default, once they install a formula with a (recusive) python requiremnet?

[MikeMcQuaid]

No, I don't agree with that. My opinion is that users who care about what Python is used by an individual package will build it from source.

[NikolausDemmel]

@mikemcquaid: You disagree with that statement, but you only provide your opinion on what people who care about what version of python is being used (in the sense that they make a consious choice about using either system or brewed python) should have to do. For those people, I agree that it is tolerable that they have to jump through some hoops to get exactly what they want. If they go for brewed python, they at some point decide to brew insall python. They will have to make sure they understand the ramifications and set up their system accordingly. That's ok.

The "default situation" I was refering to above are users that don't care about exactly which python is being used. I would say these are the majority (would you disagree?). They may install formulae that are implemented with python (e.g. ansible), they may install formulae that have python bindings (e.g. gobject-introspection via gtk+ or boost --with-python), and they may install pip packages. But they don't develop in python much themselves, and they don't care much about system vs brewed python. What they do care about is that stuff works. Now here is an example of how the current state of affair breaks for these users:

The user installs homebrew, makes no choice about python and therefore starts of with system python only. She goes on to install and use stuff that does not depend on python, and keeps having system python only. At some point she wants to do some C++/opencv development. She brew install opencv. There is no bottle, it gets built from source and the python bindings that come with opencv get compiled against system python. After that, she installs gtkmm to code a GUI in C++. This pulls in brewed python (via gtk+). Now her default python interpreter is brewed python. She wouldn't care, except that now her system is broken. Any software she subsequently tries to install and run that happens to use the opencv python bindings will likely fail, since they are built against system python and she is now useing brewed python (without making a conscious choice about the switch).

Now this problem would not arise if homebrew would not have pulled in python when installing gtkmm. IMO it should have detected that system python is being used, and therefore chosen to not use the bottle for gobject-introspection. What makes this worse is the fact that she has no real way of realizing that this might indeed be a problem, making tracking down the root of this issue very hard.

Does that example make sense? This is not just some made up scenario. These kind of things are actually happening to people.

[MikeMcQuaid]

These people should submit issues if it's affecting them. Until there's a bunch of those issues I don't think we should stop providing bottles to all people unless they have installed Homebrew python first.

We need to make a decision about the best approach to take. Every approach is going to make some people unhappy. I'm afraid I don't want to talk about this any more; I've spent far, far too long fixing all the Python stuff (which I don't use) for almost zero thanks from the community and constant criticism. If you think we should change the way we're doing this: submit pull requests and we'll talk about them.

[NikolausDemmel]

Fine. This is exactly what I wanted to do, discuss how it should work vs how it currently works. You can not expect people to support patches when it is not documented how things are intended to work and you are also not willing to discuss it before hand. How are people supposed to know what is a bug and what a "limitation" of the currently chosen design? At least I have failed to realize the distinction a number of times now.

With all respect Mike, I believe this is mostly an issue of communication. Here and in other discussions I sometimes feel like issues raised are not acknowledged at all (while maybe they are actually perfectly understood). I can live with "yes, we know this problem exists, but the alternative has also downsides which we think weigh heavier because it affects more people" or "yes, we know this problem exists; it would be great if it worked differently, but noone of us has time to implement that", but if you simply reply with "we will likely not support this, sorry; please provide a PR with proposed changes to discuss there" I feel like the issue raised is not being understood and I might start arguing strongly why I think it is a problem. Please don't mistake that as criticism of your work.

I will leave it at that. To repeat what I said above: I am extremely grateful for your work. For my personal usage, the homebrew & python situation has really improved as I can take advantage of an increasing number of bottled python-related formulae. So thank you, Mike, for your effort!

[MikeMcQuaid]

The problem is when speculative issues are raised as actual issues. Stuff like "these kind of things are actually happening to people" is unhelpful, honestly. Which people? When? Have they filed issues on Homebrew? If not, why not? It feels like around the Python stuff in particular there has been a lot of talking about potential issues and very few PRs for actual being submitted. This isn't helpful, really; it means the maintainers spend their time reading walls of text and even after that are unable to work out how to fix a problem.

For example, if you were filing this issue again the best way would be to:

1. Explain what commands you ran and what the output and effects wer.
2. Explain what you expected the output and effects to be and why you expected them to be like that.
3. Explain what, if any problems it causes you and any workarounds you currently use.

If you're happy to restate the original issue in that format I'm happy to try and elaborate more.

[NikolausDemmel]

I will try to improve the way I raise issues according to your guidelines. I can see that this will be more helpful.

I might revisit this issue in that format when I get around to it.

Thank you.

[nerduno]

This issue described by Nikolause is not purely speculative. Today I ran into the exact problematic use case Nikolause described - first installing packages against system python, and having them disappear because brew unexpectedly installed brewed python when installing pyqt. I believe the pyqt formula must have changed since I don't recall this happening last year, and I'd wager that others have run into this issue but not filed a git issue. Is a warning presented when brew installs python as a dependency?

Is brew now less friendly to users of the system python. Is there a way to brew pyqt against the system python and avoid having it install brewed python? If so, would you mind explaining? I'm sorry, but despite looking at the formula, reading the various brew/python help pages, and reading the discussion above about "--build-from-souce", I'm still unclear how to use brew with the system python.

Thanks, Aaron

[MikeMcQuaid]

Is brew now less friendly to users of the system python.

Insofar as we use the brewed Python for some things now without users requesting it: yes.

Is there a way to brew pyqt against the system python and avoid having it install brewed python?

brew install pyqt --build-from-source

[nerduno]

Thank you for your help. I ran: brew install sip --build-from-source And then: brew install pyqt --build-from-source

Based on the discussion above, I believe it was necessary to avoid letting pyqt install the sip dependency because this would have installed brewed python, correct?

The relationship between --build-from-source and whether or not brewed python is installed is unintuitive to me (probably because I don't understand the inner working of homebrew/bottling) and can result in strange behavior. Is this the documented somewhere? I don't see this information on the Homebrew and Python page: https://github.com/Homebrew/homebrew/blob/master/share/doc/homebrew/Homebrew-and-Python.md

Perhaps it be possible to print a warning when brew installs python as a dependency and masks the system python? The warning could mention that --build-from-source can be used to avoid this.

Also, I realized today that the gstreamer Formula does not list python as a dependency, but none the less installs python and hides system python.

Thanks Aaron

[MikeMcQuaid]

The docs should probably be updated, yes. I would also say that the system Python can still be used, you'll just need to put it before the Homebrew Python in your PATH.

[nerduno]

Regarding system python still being usable, the problem will be that the installed package (e.g. pyqt) will have been built against the brewed python. In my experience if you try to import this into the system python, there is a decent chance you'll end up with a segmentation fault.

[tdsmith]

A note about that: segfault on import is often an artifact of libraries improperly linking against a Python framework at build time. Homebrew Python and Apple Python should be binary-compatible for extension modules. Things that are imported from Python (as opposed to embedding Python) should be (but often aren't) built with -undefined dynamic_lookup so that symbols are resolved at load time instead of at build time. Fixing this across Homebrew is on my to-do list but it'll be a slow process. Some things, like wxpython, already just work IIRC.

[nerduno]

Thanks for the information - that will probably come in handy in the future.

[MikeMcQuaid]

@tdsmith I worked on this a bit and gave up so might be worth sharing notes. They should be binary compatible but I've not found out a way of consistently making it so.

@nerduno Yes and unfortunately when I had the opposite I had the Homebrew Python users complaining. There's no solution that will make everyone happy, unfortunately.

[nerduno]

Thanks for all your help.

As we discussed above, my two-cents is that documentation could help a lot. It is somewhat unintuitive (at least to me) that --build-to-source has any effect on whether brewed python gets installed. <why is this? could it be explained somewhere>

Thus adding a warning or message like: "Brewed python will be installed and build against ..., do you want to continue? If you would prefer to use and build against MacOSX system python see ."

If the functionality were changeable, I might consider adding a --without-brewed-python flag that in essence invoked --build-from-source in some recursive fashion.

-a

On Wed, Oct 29, 2014 at 6:14 AM, Mike McQuaid notifications@github.com wrote:

@tdsmith https://github.com/tdsmith I worked on this a bit and gave up so might be worth sharing notes. They should be binary compatible but I've not found out a way of consistently making it so.

@nerduno https://github.com/nerduno Yes and unfortunately when I had the opposite I had the Homebrew Python users complaining. There's no solution that will make everyone happy, unfortunately.

— Reply to this email directly or view it on GitHub https://github.com/Homebrew/homebrew/issues/31229#issuecomment-60921008.

[MikeMcQuaid]

As we discussed above, my two-cents is that documentation could help a lot. It is somewhat unintuitive (at least to me) that --build-to-source has any effect on whether brewed python gets installed. <why is this? could it be explained somewhere>

I'm going to try and add some when I get the chance.

Thus adding a warning or message like: "Brewed python will be installed and build against ..., do you want to continue? If you would prefer to use and build against MacOSX system python see ."

Perhaps just say "building against /usr/bin/pythonor/usr/local/bin/python`.

If the functionality were changeable, I might consider adding a --without-brewed-python flag that in essence invoked --build-from-source in some recursive fashion.

Sadly the more people try to convince me this is a good idea I'm increasingly thinking we should never use the system Python for anything that requires bindings. It's really, really, really error-prone the way Python bindings are handled and we've gone back and forth many times trying to work out how to do it better and fail every time. Obviously none of this was probably known to you or your fault but just explaining why things are the way they are.

[MikeMcQuaid]

@tdsmith Open to your input here.

[tdsmith]

I don't know if you wanted more half-baked ideas but has Homebrew tried only pouring :python bottles if the python in PATH is Homebrew python? Building pyqt from source isn't a great user experience but I think it's less surprising than having a new Python appear in PATH, especially if brew install ohai's an explanation of why the user isn't getting a bottle.

[MikeMcQuaid]

@tdsmith We did something like that previously and people got even more confused, unfortunately. Perhaps a drastic solution here is we depend on Python for more things but make it keg-only?

[tdsmith]

Yeah, I thought about it but I don't think that helps in either major case. If /usr/bin/env python -c "import foo" needs to work, bottling against a keg-only Python can't be helpful. If importing with the python in PATH doesn't need to work (because we're just installing foo as a dependency of bar and can control the environment when we launch bar so that we can find foo), bottling against system Python is more straightforward.

For users who are installing pyqt because it's a dependency of something else, I guess we could bundle pyqt into libexec. :trollface: (If pyqt were cellar :any, I might not even be joking.) I guess the long-term solution here remains variant support for bottles; I wonder about a keg-only pyqt-dependency package always bottled against system Python as a stopgap.

We did something like that previously and people got even more confused, unfortunately.

I don't mean to be tendentious but was that confusion really widespread? I'm happier defending fail-to-pour in principle but I agree that doing something that works for users is more important than being principled. I went looking for confused-user stories on the tracker and only came up with #28281, where the complaint seems to be about the message rather than the behavior.

[MikeMcQuaid]

Yeah, I thought about it but I don't think that helps in either major case. If /usr/bin/env python -c "import foo" needs to work, bottling against a keg-only Python can't be helpful. If importing with the python in PATH doesn't need to work (because we're just installing foo as a dependency of bar and can control the environment when we launch bar so that we can find foo), bottling against system Python is more straightforward.

I'd rather bottle everything against the system Python but then Homebrew Python people complain the opposite.

For users who are installing pyqt because it's a dependency of something else, I guess we could bundle pyqt into libexec. (If pyqt were cellar :any, I might not even be joking.) I guess the long-term solution here remains variant support for bottles; I wonder about a keg-only pyqt-dependency package always bottled against system Python as a stopgap.

I'm against variant support for bottles. That bottles currently use the default configuration is a feature rather than a bug; they handle the majority case and reduce errors in that case. If people have special needs they should build from source and be able to handle the errors that may result from doing so.

I don't mean to be tendentious but was that confusion really widespread? I'm happier defending fail-to-pour in principle but I agree that doing something that works for users is more important than being principled. I went looking for confused-user stories on the tracker and only came up with #28281, where the complaint seems to be about the message rather than the behavior.

I guess we read differently; most of that thread to me reads as "I want to use Homebrew Python and the Boost bottle".

Perhaps the long-term solution is you set an environment variable "HOMEBREW_PYTHON_DEVELOPER" and then everything is built from source against the Homebrew Python and otherwise everything is built against the system Python (although I suspect people will still complain that they want to enable that and e.g. still use a boost bottle). That seems more Homebrew'y as we don't force an extra dependency or linked system duplicate on people who don't want it (which I'd guess is the majority case).

By this point though I really hate anything including the words "Homebrew" and "Python".

[HughP]

Greetings, sorry that I am adding to an old issue here. I will create a new issue #38559 for getting retext working but I wanted to add to this issue because using --build-from-source is not intuitive to me - in fact I am not really sure why it is required. (So I am just saying I am one of the confused users out there...) where the default brew install xyzabc did not work.

I found this issue because it is one of 2 issues which mention retext. I tried brew install retext and the build of sip failed. So I started reading and someone said something about --build-from-source. So I tried that. It worked. Then pyqt failed just like sip but then I tried --build-from-source again and it worked but said that Phonon support is broken. I’ll report that separately. Currently working on OS X 10.9.5 with Xcode 6.2. I am really green at all this. I do have a system python(2.7) and brew python(2.7.9) installed (plus a brew python3).