Adding IPFS support to Homebrew

[andrew]

As mentioned in https://docs.google.com/document/d/1-HtUiRpMzYq9to56ShCGyCr-NCZ6TR49Zl-b5HHJdm0/edit#heading=h.m17gf4vdw67, adding IPFS support to "portable registry" package managers is fairly easy due to the fact that you can fork the whole registry and add hashes to the metadata.

As an experiment, I thought I'd give it a try with Homebrew.

I picked a simple "Formula" and added the tarball to ipfs:

$ ipfs urlstore add https://github.com/mtoyoda/sl/archive/5.02.tar.gz

Which gives you the hash: zb2rhYgkf18LAi1FjyZPZfU43UrxwBZgV59XrpKR3mCXfNAyX

I then added that hash to the formula in my fork of homebrew-core: https://github.com/andrew/homebrew-core/commit/50b776ce8f2841457610a232a815267910a45f5c

This extra field won't just work with the brew command, so I forked and patched homebrew to add support for the ipfs method: https://github.com/andrew/brew/commit/17492b84b5da49181eb9a6c601747adc5ebafc96

This quick hack makes a few of assumptions:

• you have the ipfs command installed in /usr/local/bin/ipfs, which is the default on mac when installed using homebrew
• you have ipfs daemon running, it doesn't try to start it for you
• you want to build from source, it doesn't work with prebuilt binaries, which homebrew calls bottles

Luckily with homebrew, you can install and run it from any directory, so it's easy to download my fork:

$ git clone https://github.com/andrew/brew && cd brew

Then install my forked sl formula via IPFS:

$ ./bin/brew reinstall andrew/core/sl --build-from-source

Also of note, homebrew checks the file that was downloaded from IPFS against the exist sha256 integrity hash and it matches 🎉

Then you can run it:

$ ./bin/sl 

IPFS hashes could easily be added to most of the other ~4,700 formula homebrew core using a script to find the url of each formulas tarball, add it to ipfs, update the formula with the hash and commit, and then have a script to watch the homebrew repo for new formula updates.

I also briefly mentioned it to @mikemcquaid over SMS, overall he thought it was very neat, adding:

FYI to avoid disappointment: we’re unlikely to add a new download strategy to Homebrew/brew until it’s required by a core formula as the main download method.

Some other homebrew notes:

Homebrew is very heavily used in the mac community for installing databases, languages and other system level packages, they also have public install analytics: https://formulae.brew.sh/analytics/install/30d/

They're also moving towards offering more precompiled binaries (bottles), currently hosted on https://bintray.com/homebrew

Homebrew is also preinstalled on all Travis CI and Circle CI MacOS projects

[mikeal]

This is awesome! This kind of exploratory code is super useful and we learn a ton from it.

IPFS hashes could easily be added to most of the other ~4,700 formula homebrew core using a script to find the url of each formulas tarball, add it to ipfs, update the formula with the hash and commit, and then have a script to watch the homebrew repo for new formula updates.

I spent a little time working on a Lambda function that would return you the IPFS hash for the resource of any URL and cache the result based on the etag.

https://github.com/mikeal/ipfs-http-file-cache/blob/master/lib/file-blocks.js

It isn’t near complete yet, it doesn’t cache the result or store the blocks yet. One problem we have is that there isn’t a good way to pin and manage all the blocks. My plan right now is just to write them into S3 so that they are available in some way, even if not actually accessible in the IPFS network.

This is a very generic problem, and one that we should break off as an independent work item and build something that works universally.

[jimpick]

This seems like something I'd use. :-)

[andrew]

As hinted to me by @warpfork this morning, adding support for "Bottles" may be restricted right now due to the lack of exec bit file permissions in unixfs at the moment

[andrew]

A related discovery I made whilst poking around in homebrew formulas: https://github.com/Homebrew/homebrew-core/pull/37426

[andrew]

Got some stats from @mikemcquaid on the bintray storage and bandwidth used by homebrew for prebuilt binaries:

January 2019: 1.4TB storage/167TB bandwidth

February 2019: 1.4TB storage/173TB bandwidth

That's an average of 5.5TB per day.

Also of note, all that hosting is provided for free to Homebrew by Bintray.

[andrew]

Also spent a little time researching Linuxbrew, which is now officially supported in Homebrew 2.0.

According to @mikemcquaid:

Linuxbrew had 15k active users in February and Homebrew has 1.2M

Linuxbrew basically works exactly the same as homebrew, so the same ipfs method can be added to formula and the same patched version of homebrew should work there too, including on windows within the linux subsystem.

It looks like most formula build from the same source on both mac and linux, so they would use the same hash, the only real difference is that bottles for linux brew come from a different bintray namespace: https://bintray.com/linuxbrew

[meiqimichelle]

@andrew what's the status of this experiment? (<-- ...you may be sensing a theme in my latest round of comments <3 )

[andrew]

This experiment is pretty much done for now, by the time we get back to it much of the actual code will need to be redone.