Portable bottles for Linuxbrew

[sjackman]

Started from this discussion.

[sjackman]

I would really like Homebrew-Science to be that central repository of bioinformatics software. It is ported to both Mac and Linux. It already has formula to build tons of bioinformatics software from source, 173 at last count. There are binary packages (called bottles, in Brew-ease) of 141 of these for Mac OS! The missing component is bottles for Linux. Building the bottles is easy: brew install --build-bottle bwa && brew bottle bwa. The trick is making this bottles portable across multiple distributions of Linux.

The two key issues:

• dynamic library dependencies (including glibc and libstdc++)
• Installed location in the file system. Some bioinformatics software, once it's installed, doesn't like to be moved to a new location in the file system.

I've create an issue on GitHub if anyone wants to discuss bottling for Linux: https://github.com/Homebrew/linuxbrew/issues/380

The good news, a lot of bioinformatics software doesn't suffer from either of these issues and can be bottled today! We need only get up and running a CI tool, such as TravisCI or CircleCI or Jenkins, to bottle each new release of the bioinformatics software as is already done for Mac by the BrewTestBot.

[iggyvolz]

Not sure if you would want to give me push access to the repository, but I have a virtual private server with Jenkins running on Ubuntu 14.10 (working on getting it up to 15.04) that I'd be willing to offer to share.

[sjackman]

Hi, Iggy. I would very much like the help of another contributor/committer to Linuxbrew. You've contributed two commits (7344ef24ebacdac841c2dc7fb74904bdd72a034a and 93471b0b873e290f73b2f1b085fbdd029e736765) and lots of support to Linuxbrew. I'd like to see a history of a few more commits before giving push access. Could you go through the open issues and find say three that you can fix and submit pull requests to fix them? #369 is an easy one.

[sjackman]

Oh, and thanks for the Jenkins offer! I was planning on using the hosted TravisCI or CircleCI, but BrewTestBot uses Jenkins, so it might be easier to get that up and running. I don't have a strong preference which, as long as it works.

[sjackman]

I've bottled homebrew-science/bwa for Linuxbrew and uploaded the bottle to Bintray as a proof of concept. See https://github.com/Homebrew/homebrew-science/commit/e4861ac28a601729f8c7e17650a6a14a1b0715f6

[andreabedini]

good stuff guys, I'm loving this. Question, is x86_64_linux going to be enough? shouldn't we include something more specific (like glibc version?) to make it work on all platforms? otherwise we could use a distribution/release scheme like "CentOS_6_6", it would include a lot of bottles to build but it would also be rock solid. my 2c :)

[sjackman]

I'd rather not make a separate bottle for each distribution. There are simply too many. The Standalone Installation of Linuxbrew is my solution to this problem. The glibc formula provides an up-to-date glibc regardless of your distribution.

[andreabedini]

it makes sense @sjackman, I tried building my own bottles with

docker run sjackman/linuxbrew-standalone sh -c 'brew install --build-bottle hello && brew bottle hello

pretty cool stuff!

[sjackman]

I agree! It is cool stuff. That's exactly how I plan to build bottles, using the linuxbrew-standalone Docker image. I just need to get it rolled into a CI system.

[andreabedini]

@sjackman what's the plan regarding dependencies? if I do brew install --build-bottle python can I then get bottles of python's dependencies? they get built after all ... If not we need to build a bottle for each dependency prior to building the main package to speed up things and maximum bottling :)

[andreabedini]

happy to give a hand with CI if you need

[sjackman]

If not we need to build a bottle for each dependency prior to building the main package

That's the approach that Homebrew/Homebrew takes. For Linuxbrew I added a wee undocumented hack: export HOMEBREW_BUILD_BOTTLE=dependencies

See https://github.com/sjackman/docker-linuxbrew/blob/master/linuxbrew-bottle/Dockerfile#L11 and https://github.com/Homebrew/linuxbrew/blob/455e3b814e1b8226dc20548217d205582a1323b7/Library/Homebrew/formula_installer.rb#L360

[sjackman]

happy to give a hand with CI if you need

Happy to accept a PR! I'm pretty busy writing up my thesis proposal, so you can expect delays in my responses and in merging, but I am interested and will do my best.

Which CI were you thinking? The options (that I know of) are Jenkins, Travis and Circle. BrewTestBot uses Jenkins. Linuxbrew/Homebrew-science uses Travis. @iggyvolz offered to help set up a Linuxbrew BrewtTestBot on a Jenkins machine.

[iggyvolz]

I haven't forgotten about this - it's prep time for final exams so I'm swamped with homework and can't dedicate time to other things right now... hopefully this weekend I can start looking into this.

[sjackman]

No worries. I may not be able to seriously look into this until after my comprehensive exams, which are end of July.

[andreabedini]

what's the trick to pour bottles? I compiled some bottles as above and copied them on a different machine. I'm trying to do

brew install ./doxygen-1.8.9.1.x86_64_linux.bottle.tar.gz

but it goes on installing everything from source as I just did brew install doxygen. What am I missing?

[sjackman]

Is it installing the dependencies from source? A bottle includes only that package and does not include its dependencies.

[sjackman]

I've got a Raspberry Pi. I'm going to start bottling for arm-linux. Should the default HOMEBREW_PREFIX for bottles be /home/pi/.linuxbrew or /usr/local?

The biggest benefit of /home/pi/.linuxbrew is that it's completely independent of the rest of the system. The biggest benefit of /usr/local is that it's the same as Homebrew.

[andreabedini]

if it's only a default /usr/local should be ok. As I see it, default means you own your computer and can do chown user /usr/local if needed; while users on a system they don't administer can set HOMEBREW_PREFIX= /home/pi/.linuxbrew. my 2c

[sjackman]

It is only a default, but it's a default that matters a bit more than usual, because bottles are compiled for a particular prefix, and if the bottle is not relocatable, it can only be installed in that prefix.

[andreabedini]

ouch, I get it now. Very hard choice then :)

[sjackman]

My biggest reservation about installing libraries in /usr/local/lib is that they may mess up system executables installed in /usr/bin if they load the wrong libraries. There's no risk of that if they're installed in /home/pi/.linuxbrew/lib.

[andreabedini]

good point, on OS X this is avoided by keg only formulas right? on gnu/linux distribution everything is more messy. :+1: for .linuxbrew then.

[andreabedini]

I still run into problems with building bottles for dependencies, see this log. libevent should get built as a bottle because it's installed as a dependency of tmux but that doesn't happen. Bug or I am missing something?

[andreabedini]

It's because I'm using linuxbrew-standalone rather than linuxbrew-bottle. It's not clear what's the difference between the two.

[sjackman]

I have a special undocumented hack to deal with that.

export HOMEBREW_BUILD_BOTTLE=dependencies

See https://github.com/sjackman/docker-linuxbrew/blob/master/linuxbrew-bottle/Dockerfile#L4

[DoomHammer]

Are those bottles usable in master Linuxbrew? When I try to install one of them specyfing full path, brew still tries to build it from source (the package, not dependencies).

[sjackman]

It checks that you have both patchelf and glibc installed, or it'll refuse to use the bottle and build from source instead. You can use --force-bottle to override. See https://github.com/Homebrew/linuxbrew/blob/master/Library/Homebrew/formula_installer.rb#L85

[DoomHammer]

OK. I got it now. But it means that you still need to have some kind of usable CC in order to build glibc first, right? I have a system that both lacks compiler and package management (Gentoo sans Portage to be exact) and I wanted to (ab)use Linuxbrew to install on-demand software there.

[sjackman]

You can build custom bottles specific for your Gentoo installation of the toolchain (binutils, glibc and gcc) and use those on other Gentoo systems.

[DoomHammer]

I tried more hackish approach.

1. I exported ~linuxbrew/.linuxbrew from sjackman/docker-standalone.
2. On a Vagrant guest I ran LD_LIBRARY_PATH=/vagrant/.linuxbrew/lib /vagrant/.linuxbrew/opt/glibc/lib/ld-2.19.so /vagrant/.linuxbrew/opt/patchelf/bin/patchelf --set-interpreter /vagrant/.linuxbrew/opt/glibc/lib/ld-2.19.so --set-rpath /vagrant/.linuxbrew/lib /vagrant/.linuxbrew/opt/patchelf/bin/patchelf. This gave me usable patchelf.
3. In a more than silly approach I tried to change RPATH and interpreter of whatever binary I could find: find /vagrant/.linuxbrew -type f -executable|while read binary; do /vagrant/.linuxbrew/opt/patchelf/bin/patchelf --set-interpreter /vagrant/.linuxbrew/opt/glibc/lib/ld-2.19.so --set-rpath /vagrant /.linuxbrew/lib "$binary"; done. This gave me usable ruby (sort of).
4. Trying to run (export PATH=/vagrant/.linuxbrew/bin; /vagrant/.linuxbrew/bin/brew --help) results (as ruby isn't reconfigured for the new system) in:

<internal:gem_prelude>:1:in `require': cannot load such file -- rubygems.rb (LoadError)
        from <internal:gem_prelude>:1:in `<compiled>'

Is it at all possible for me to reuse linuxbrew-standalone in another directory?

[sjackman]

When you install a bottle, as long as you have patchelf and glibc installed, it handles setting the interpreter and changing the RPATH for you. Any formula marked as cellar :any should work. Note that ruby is not cellar :any, so it is not expected to work. :grimacing:

[DoomHammer]

I am still not quite comfortable with all the new terminology. I understand this means that ruby can't be installed from a bottle?

[sjackman]

It can be installed by a bottle, but only to the HOMEBREW_PREFIX of /home/linuxbrew/.linuxbrew.

[DoomHammer]

I should then try to substitute the brewed ruby with Travelling Ruby. Maybe this will allow me to port the whole deal to "pristine" box.

[sjackman]

Cool. Yes. Travelling Ruby looks like it would be very useful to bootstrapping Linuxbrew.

[hartzell]

Back in the thread that inspired this one, @sjackman answered a question that "lh3" asked by saying

Can a user package a bottle by him/herself, or packaging has to go through linux/brew?

Yes, a user can bottle any formula, but that bottle can't be uploaded to the official Homebrew Bintray account, which is done only by the BrewTestBot.

I've done a few experiments based on your Dockerfile and have ended up with bottles and bits of ruby.

I'm not sure what to do with them though. I'm interested in building a local cache of bottles that I've built (trust issues, etc...) so I can't add the ruby snippets into the Forumula and push them back upstream.

I supposed that I could merge them into my local repo and just keep on top of it if/when I pull.

On the same theme, can I get linuxbrew to use my bottles w/out modifying the formula?

Thanks for all of the work on this!

[sjackman]

Cool. I like the idea. Modifying each formula is unfortunately the most straightforward tack, like so:

bottle do
  cellar :any
  root_url "https://your.server/bottles"
  sha256 "…" => :x86_64_linux
end

If you wanted to try hacking Linuxbrew to avoid needing to modify each formula, change the default root_url over here https://github.com/Linuxbrew/linuxbrew/blob/master/Library/Homebrew/software_spec.rb#L307 and then you'll have to hack up the logic to disable needing the sha256. That may be tricky.

[sjackman]

I've just discovered that it is possible to override the bottles in your own fork of Homebrew using https://github.com/Homebrew/homebrew/blob/master/Library/Homebrew/hooks/bottles.rb I don't know anything about this mechanism other than it's intended for your purpose.

[sjackman]

I had planned on building bottles on the sjackman/linuxbrew-standalone Docker image using the glibc installed by Homebrew (2.19). An alternative plan would be to build the bottles on an older distribution. The compiled bottles should work on Linuxbrew Standalone, but should also work on older systems. The dependencies used from the host system should be glibc and libstdc++. All the other dependencies should be provided by Linuxbrew. That way perhaps Linuxbrew users could use bottles without having to install glibc and gcc (for libstdc++).

Distro	glibc
Linuxbrew	2.19
Ubuntu 14	2.19
Ubuntu 12	2.15
Ubuntu 10	2.11.1
Ubuntu 8	2.3.6
Ubuntu 6	2.3.2
CentOS 7	2.17
CentOS 6	2.12
CentOS 5	2.5
CentOS 4	2.3.4

[sjackman]

what do we do about apps that hardcode information at build time, like the local lisp path from the emacs configure step:

Homebrew solves this problem by installing to a fixed prefix /usr/local. Linuxbrew on the other hand installs by default to ~/.linuxbrew, which is a different path for each person. One solution is to install to a fixed prefix like /usr/local or /opt/linuxbrew or /home/linuxbrew/.linuxbrew. Note that this only affects formula that store hard-coded paths in the binary executable. If that path is in a text file, Homebrew knows how to edit that config file for you when the bottle is installed. Formula marked with cellar :any do not have any paths hardcoded in binary executables and can be bottle for Linuxbrew. Roughly half (1870 / 3472) of formula are cellar :any.

[sjackman]

The dependencies of glibc are bottled: zlib binutils linux-headers The dependencies of gcc are bottled: gmp isl libmpc mpfr glibc and gcc themselves are not yet bottled, due to them not being cellar :any.

[sjackman]

I'm manually bottling a few key formula, particularly formula that are common dependencies and formula that take a long time to build. I'm using sjackman/linuxbrew which is running Ubuntu 14.04.3 LTS, glibc 2.19 and gcc 4.8.4.

[sjackman]

Until I figure out how to bottle gcc, I'm going to bottle up libgcc_s.so.1 and libstdc++.so.6. These libraries are normally provided by gcc. On Debian/Ubuntu, these packages are called libgcc1 and libstdc++6 respectively. I was thinking of putting them in the same package, perhaps called libgcc.

[sjackman]

I've bottled GCC for Linuxbrew! I'm very excited by this feat. With both glibc and gcc easily installable by users, we've nearly achieved the desired portable bottles for Linuxbrew. That's great progress since this ticket was opened almost a year ago. Now we need to get Linux bottles building automatically on a CI system, such as Travis CI or Circle CI, and uploaded to BinTray. See https://github.com/Homebrew/homebrew/issues/48832

[tseemann]

This is pretty neat! FYI - the equivalent packages on RHEL/Centos are libgcc and libstdc++.

[sjackman]

Yes, it is! I may not need to package these libraries separately, since they're included in the gcc bottle.

[sjackman]

I wish I could bottle glibc, then we wouldn't need a host compiler at all to bootstrap Linuxbrew, but glibc doesn't like to be moved to a different directory once it's installed.

[zbeekman]

How do I get/install gcc from bottle? Or build and post my own gcc bottle? Do I need to build glibc from source first?

[zbeekman]

(To elaborate a little more, I'm trying my hand at mimicking some of your docker builds using vagrant and ssh with Ubuntu precise... I want an easier way to get GCC 5.3 and 5.4 (once it rolls out) on Travis-CI)