boost - provide option to pour from bottle?

[mistydemeo]

Boost currently always builds from source if the user has a Homebrew python installed at all:

  def pour_bottle?
    # Don't use the bottle if there is a Homebrew python installed as users
    # will probably want to link against that instead.
    not Formula.factory('python').installed?
  end

As far as I can tell, this is unconditional, e.g. there isn't a --pour-bottle option of some kind to override this.

Personally I don't care about boost and it just gets installed as a dependency of other stuff, and I really hate compiling boost. (I also use it with python approx. never.) This was pretty surprising behaviour to me. I'd like to see this get changed.

Ping @mikemcquaid @samueljohn

[jacknagel]

+1

[mistydemeo]

Incidentally, having some messaging on why the bottle isn't being poured in either the verbose build or when HOMEBREW_DEVELOPER is set would be great, I honestly thought the "bottles compatible?" logic was broken until I looked at the formula.

[jacknagel]

Yeah, "building from source because f.pour_bottle? is false" would be helpful.

[mistydemeo]

Or promote pour_bottle to a DSL block that lets us track extra information:

pour_bottle do
  optional true

  pour_when { not Formula.factory('python').installed? }
  cause "building boost from source will cause it to link against your custom python."
end

To produce:

brew install boost -v
==> boost will be built from source instead of from prebuilt binaries.
This is because building boost from source will cause it to link against your custom python.
If you would like to install the prebuilt binaries anyway, pass --pour-bottle.
==> Downloading http://downloads.sourceforge.net/project/boost/boost/1.54.0/boost_1_54_0.tar.bz2

[samueljohn]

Agree that a message should be printed as you suggest, Misty.

I think we have to decide to either bottle boost with system python or with brewed python. I don't know how to make it build/run with both. Completely disabling python support in the bottle is not beneficial, too.

[manphiz]

+1 for system python. Personally prefer less dupe by default, and not following macports in this regards.

On Sun, Aug 11, 2013 at 11:16 PM, Samuel John notifications@github.com wrote:

Agree that a message should be printed as you suggest, Misty.

I think we have to decide to either bottle boost with system python or with brewed python. I don't know how to make it build/run with both. Completely disabling python support in the bottle is not beneficial, too.

— Reply to this email directly or view it on GitHub.

[mistydemeo]

I agree, I don't think we should ever bottle anything against anything but the system python.

[mistydemeo]

(Bottle, that is, not build.)

[MikeMcQuaid]

Sorry, I added this on request without thinking about it too hard. At the very least, I agree, I need to improve the messaging here.

I think it's probably worth asking if this is a sensible thing to do anyway? Is having the non-system Python installed enough to block bottle installation (regardless of a message)?

I think an extra DSL is probably overkill given this is the only formula where this is used and we might not even want it here.

Finally, and perhaps controversially, stuff like this makes me wonder whether even supporting the non-system Python well is something we even want to do.

[samueljohn]

That request came from me, I guess, because otherwise cryptic run time errors can come up. But a message would be good.

I think I want to support a non-python well, for a lot of reasons, but I am fine with having system python as the default (as it is already right now).

[MikeMcQuaid]

Can you show some of the cryptic run-time errors?

Without meaning to be too much of an ass here: you want to support a non-system Python but do our users? Is it worth the added complexity to both core and formulae?

I wonder if simply having a non-system Python installed is enough to signal intent that you'd rather use it instead.

[samueljohn]

The original issue was #17142. There are also somewhat related: #11633, #20114.

Yeah, @mikemcquaid, the cryptic error is this one and it is quite non-telling:

Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6

I documented that in the wiki some months ago in "Common Problems".

If the user has a brewed python and he has set up the PATH correctly, that python is upfront and will be used by scripts (that don't have hard-coded the she-bang), potentially leading to the above mentioned "Fatal Python error". So, for me, having a brewed python is the "flag" that other things in homebrew should use that Python. We could also start to introduce a --with-brewed-python flag for this but that would add another layer of complexity in its own.

At least it should be possible to use boost with a brewed python, IMHO. Either by --with-brewed-python or by just having a brewed python. I still prefer the latter.

I don't know how many users prefer boost with system python vs. boost with a brewed python.

[MikeMcQuaid]

@samueljohn Should we perhaps be using a different Python module PATH for system and Homebrew pythons? That would avoid this problem, no? I think we've got the problem in reverse here; the problem isn't that the Boost formula is doing anything wrong but that the Homebrew Python is trying to import something that will fail.

At least it should be possible to use boost with a brewed python, IMHO.

Is this something you personally use or just think should be possible?

[samueljohn]

@mikemcquaid I'll have to think about the consequences of using different directories for system and brewed python... Given the PythonInstalled Requirement, it should be fairly easy to implement that in a central way. But I don't want to make a decision head over heels.

At least it should be possible to use boost with a brewed python, IMHO. Is this something you personally use or just think should be possible? --@mikemcquaid

Both :-)

[manphiz]

With --with-brewed-python user can choose which python to use at least. Maybe adding a caveat when building against system python when homebrewed python is also present, or when using bottles.

[samueljohn]

I am not sold on passing with-brewed-python for every formula with python bindings, I install. For people who don't care about a brewed python it just works™ as it is right now.

So my argument is that brew install python (or should we reduce to having python linked) is the "flag" that tells, I want the python bindings for that one.

[manphiz]

Well, I understand your argument, but to extend this issue, what if both python2 and python3 are installed in homebrew? Boost support both python versions, for instance, though there is currently no option to build python3 binding at the moment.

On Mon, Aug 12, 2013 at 12:15 PM, Samuel John notifications@github.com wrote:

I am not sold on passing with-brewed-python for every formula with python bindings, I install. For people who don't care about a brewed python it just works™ as it is right now.

So my argument is that brew install python (or should we reduce to having python linked) is the "flag" that tells, I want the python bindings for that one.

— Reply to this email directly or view it on GitHub.

[samueljohn]

For PyQt and some others I implemented simultaneous Python 2.x and 3.x support. I don't know if that is easily possible for boost. Depends on how boost handles it.

[manphiz]

For boost, it is possible to have multiple python bindings at the same time, which is what Debian is doing. See also the package file list and the build log which explains their way to handle this situation (python 2.7/3.2/3.3 at the same time):

http://packages.debian.org/sid/amd64/libboost-python1.54.0/filelist https://buildd.debian.org/status/fetch.php?pkg=boost1.54&arch=i386&ver=1.54.0-2&stamp=1373225896

Simply put, what they are doing is to build python and MPI bindings multiple times against different python versions and carefully name the built libraries. So theoretically both system python and homebrew python can be supported at the same time, given that user is using the correct dylib. But I'm not a heavy boost-python user, so I'm not sure how many homebrew users will actually need this.

And then I digressed. Back to the topic, IMO it is still worth providing options for user to specify which one to use. IIUC, the complication here about duplicated python installation is that it also depends on the user to correctly set $PATH and $PYTHONPATH for their preferred python to work. For instance, similar to those issues, if I installed homebrew python, then boost-python binding will be built against that one, but if I still have /usr/bin with higher priority than brew --prefix/bin, then system python will be picked, which is prone to breakage as well. So it is also important to guide user to correct/optimal configuration for everything to work.

On Mon, Aug 12, 2013 at 1:31 PM, Samuel John notifications@github.com wrote:

For PyQt and some others I implemented simultaneous Python 2.x and 3.x support. I don't know if that is easily possible for boost. Depends on how boost handles it.

— Reply to this email directly or view it on GitHub.

[samueljohn]

For boost, it is possible to have multiple python bindings at the same time

Cool, than we (I) can do that, too, perhaps. Thanks for the pointer.

Simply put, what they are doing is to build python and MPI bindings multiple times against different python versions and carefully name the built libraries

This is how I do it for PyQt, PySide, sip, shiboken, too. Python Modules (and compiled .so modules) go into different site-packages and binaries get appended -py3 mostly. We have the python do ... end block for that, which is executed for once for each python (2.x vs 3.x).

So theoretically both system python and homebrew python can be supported at the same time, given that user is using the correct dylib.

However, I assume, most software has no option we can use to tell which of the different boost-python-mt libs to use (if we add different suffixes).

if I still have /usr/bin with higher priority than brew --prefix/bin

...then brew doctor will complain. This is not supported and other things will break, too.

But the problem arises no matter against which python we build, because people use Python from anaconda, from python.org, from pythonbrew (this is not homebrew), from pyenv and so forth...

The original question here was if just having a brewed python is enough to disable the bottle and build the python bindings against that one, or if an extra flag has to be provided. Other package managers have a kind of global flag, they call "meta"-packages. I think we don't want to go down that road...

[MikeMcQuaid]

I like the approach @manphiz mentioned. I think the actual bug(s) are Python looking in the wrong place. I don't agree that doing brew install python signals intent.

[samueljohn]

I don't understand what you mean by "Python looking in the wrong place", since it is about other software using the boost-python.dylib (or similar named) to generate their C++ bindings.

However, it is unfortunate that Python or boost-python does not error out with a more helpful message like "Hey, you have built boost with another Python that is not binary compatible to this one." or even better building the bindings with a wrong python binary should fail early.

We've always had the "first python found in PATH" as the way tell which python to use.

We have to think harder to find a better solution here. Perhaps I am no longer so vehemently against --with-brewed-python ... but I have not thought about this long enough.

[MikeMcQuaid]

Ok, I'm not sure I understand the problem but if there's imcompatible APIs I guess my point is they should require names (if not different locations) to allow software to look in the right place. Given --with-brewed-python is sticky (and we could do different options for brewed Python 2 and 3) it seems like a good solution to me. The Python infrastructure, although neat, perhaps is overkill for the relatively small number of users who are going to be using a brewed Python (which, as a system duplicate, longer-term we probably want moved to a tap anyway) and software which builds Python bindings and actually using those Python bindings.

[manphiz]

if I still have /usr/bin with higher priority than brew --prefix/bin ...then brew doctor will complain. This is not supported and other things will break, too.

While I understand the rationale to enforce this, sometimes users (including me) are forced to keep /usr/bin at the beginning of $PATH.

One example: Back in the days I was using macports (and I put macports path in the front), I happened to come across a 3rd party product (binary) which links against system openssl, and when I tried to write some code and build against it, it linked against openssl in macports, which in turn became a huge chaos until I found out the cause. Not exactly the same issue for multiple Python here, but maybe worth noting.

Admittedly my understanding of Python is primitive, but I kind of have the feeling that by carefully setting $PATH, $PYTHONPATH, $PYTHONHOME, may be some envvar as well, one can choose exactly which Python to use in an environment with multiple Python installations. Do correct me if I'm wrong. If this is the case, IMHO it may be worth providing options for user to choose from, while carefully pick a sensible default.

[manphiz]

Oops, I seems to have missed sitecustomize. So @samueljohn have implemented the mechanism for "python sandboxing", great work.

[samueljohn]

@mikemcquaid

• what do you mean by "sticky"?
• I will look into implementing a --with-brewed-python option, okay. But I need time to get it right.
• I don't think the number of users of brewed Python is all that small :-) I'd love to have python in core. Let's discuss this for brew2.

@manphiz yep, you might want to look at https://github.com/mxcl/homebrew/blob/master/Library/Homebrew/requirements/python_dependency.rb and https://github.com/mxcl/homebrew/blob/master/Library/Homebrew/python_helper.rb. With those, it should be possible to implement a --with-brewed-python option.

[MikeMcQuaid]

@samueljohn By sticky I mean you only need to call it once.

[samueljohn]

@mikemcquaid globally? Like a hypothetical brew config --with-brewed-python?

[MikeMcQuaid]

No, per formula. However, that's how we indicate all preference based things generally.

[samueljohn]

I am not opposed so vehemently against --with-brewed-python any longer. So that we kinda hard-code system python and disallow any other external python.