Installer script for IPFS?

[mitra42]

Version information: All

Type:Feature

Description:

I'd like to see an install script for IPFS, something that a package that wanted to build on IPFS (like dweb-mirror could use to install IPFS as part of another installation process

The current methods dont really lend themselves to automation, at least on the platforms we are interested in.

ipfs-update works well in some circumstances, but it depends on installing golang > 1.9 and then builds from source. The problem with this is that on for example ARM the apt-get current install is 1.7 so ipfs-update fails with the maths/bits problem documented in https://github.com/sahib/brig/issues/13

[https://dist.ipfs.io/#ipfs-update] does a pretty good job for manual installation, but the links to packages are version specific e.g. [https://dist.ipfs.io/go-ipfs/v0.4.18/go-ipfs_v0.4.18_linux-arm.tar.gz] so any installation script that tries to grab them is going to be fixed (immutable :-) at a particular version of IPFS.

I'm sure I'm not the only one banging my head on how to install IFPS automatically in a variety of circumstances, so it would be great if there was a supported way of doing this rather than a hack.

On the topic of which ... in the absence of a supported version, I'll stick a hack into dweb-mirror based on whatever I can get to work on my three current platforms (OSX, Rachel3+, RPi3) PR's for other platforms will be welcome.

[djdv]

[https://dist.ipfs.io/#ipfs-update] does a pretty good job for manual installation, but the links to packages are version specific e.g. [https://dist.ipfs.io/go-ipfs/v0.4.18/go-ipfs_v0.4.18_linux-arm.tar.gz] so any installation script that tries to grab them is going to be fixed (immutable :-) at a particular version of IPFS.

I wonder if it would be sensible to generate and provide a "latest" URL in https://github.com/ipfs/distributions. e.g. https://dist.ipfs.io/go-ipfs/latest/go-ipfs_linux-arm.tar.gz would be a copy or link of https://dist.ipfs.io/go-ipfs/v0.4.18/go-ipfs_v0.4.18_linux-arm.tar.gz until dist updates and gets republished.

[mitra42]

I think that would be a huge improvement.

The other thing that would make this easire would be using info in the links that is automatically detectable.

For example

https://dist.ipfs.io/go-ipfs/latest/go-ipfs_`uname -k`-`uname -m`.tar.gz

would give us packages ending in names like go-ipfs_Linux-armv71.tar.gz, there would be some duplication of links, but it would save having to maintain a comprehensive mapping in any install script between what the running OS thinks it is, and what name dist.ipfs.go thinks it is.

[Stebalien]

ipfs-update works well in some circumstances, but it depends on installing golang > 1.9 and then builds from source.

ipfs-update only requires go to build ipfs-update. You should be able to download the latest version here: https://dist.ipfs.io/#ipfs-update.

Once you have, ipfs-update install latest should automatically:

1. Find the latest version.
2. Install it.

Of course, it rarely picks a good location to install ipfs.

---

One solution (if someone has time) would be to do what rustup does:

1. Ship an ipfs-update package that includes both an ipfs-update and an ipfs binary.
2. Instead of including the real ipfs, the ipfs binary would just be a launcher for the real ipfs binary downloaded by ipfs-update. This way, we can put it anywhere we want without having to worry about the path.

---

I wonder if it would be sensible to generate and provide a "latest" URL in https://github.com/ipfs/distributions.

IIRC, the only reason we don't do that is a lack of symlink support in the gateway. In the past, this has been blocked on security concerns but I think we've thought that through pretty well and could just implement it now.

[mitra42]

Yes @Stebalien , you can always download the latest version of both ipfs and ipfs-update from dist.ipfs.io as linked in the opening comment, but that cannot be used from a script, as the links to actually get the files are always different.

Maybe, if IPFS can't do symlinks, then you could put the symlinks outside of IPFS (old-fashioned HTTP server :-) or if you wanted to be able to fetch the file entirely in IPNS and the gateway supports IPNS then would something like this work?

http://dist.ipfs.io/go-ipfs/latest/go-ipfs_Linux_armv71.tar.gz -> gateway.ipfs.io/ipns/q12345/go-ipfs_Linux_armv71.tar.gz -> gateway.ipfs.io/ipfs/q12345/xxxx

(Edited - last url was wrong)

[Stebalien]

Do you really need the latest version of ipfs-update or can you just pick a version and hard-code it? My point is that ipfs-update is, itself, the "install script".

[mitra42]

That's actually a good question - I was worried about fixing the version of ipfs, but maybe fixing the version of ipfs-update wouldnt be too bad. You'd still need a reasonably complex install bash script because you have to map the version numbers from what uname -k and uname -m returns to the (arbitrary, I think) file names that dist.ipfs.io uses, but thats an order of magnitude easier (and therefore less likely to fail) than trying to get Go to install automatically, and then ipfs-update and then ipfs which is the only way currently without specifying (and therefore freezing) version numbers.

Any thoughts on that http -> ipns -> ipfs pathway above. (I just edited it, last line was wrong).

[Stebalien]

You'd still need a reasonably complex install bash script because you have to map the version numbers from what uname -k and uname -m returns to the (arbitrary, I think) file names that

So... unfortunately, those are the architecture names as returned by go env. That's why we use them. If you really wanted, you could install go (any version) using your system's package manager, then use dist_get to fetch the right ipfs-update archive, then use ipfs-update to get IPFS. But yeah, that kind of sucks.

Any thoughts on that http -> ipns -> ipfs pathway above. (I just edited it, last line was wrong).

With respect to "use a plain-old web server": Given our tooling (use ipfs for everything), supporting a special-purpose webserver for dist.ipfs.io is probably going to be significanlty more work than just adding symlink support to the gateway.

For context, the dist.ipfs.io actually uses the go-ipfs gateway using DNSLink. (cloudflare has better documentation on this than we do: https://developers.cloudflare.com/distributed-web/ipfs-gateway/connecting-website/).

or if you wanted to be able to fetch the file entirely in IPNS and the gateway supports IPNS then would something like this work? http://dist.ipfs.io/go-ipfs/latest/go-ipfs_Linux_armv71.tar.gz -> gateway.ipfs.io/ipns/q12345/go-ipfs_Linux_armv71.tar.gz -> gateway.ipfs.io/ipfs/q12345/xxxx

Are those HTTP redirects?

---

Note: You can also:

1. Fetch https://dist.ipfs.io/go-ipfs/versions.
2. Read the last line of this file (the latest version).
3. Fetch https://dist.ipfs.io/go-ipfs/$version/go-ipfs_$version_$os-$arch.$extension

That's what ipfs-update does internally.

[mitra42]

That's unfortunate that go invented its own way of describing the processor :-(, removing a dependency on go (for ipfsipdate would be good on small machines (like Raspberry Pi's). That means that I think whatever happens we need something in a bash script that maps portable ways to detect processor type etc (uname) with go-specific names.

Yes -I'd assumed dist.ipfs.io was doing something like that ... sorry, my abbreviated suggestion was cryptic. I can think of several ways to do this, but am not current on the state of IPNS.

For example .... I think there could be an IPNS mapping from gateway.ipfs.io/ipns/<SOme link that never changes> to gateway.ipfs.io/ipfs/<hash of directory containing all releases for a particular version> that IPNS link would be updated each time a new release went up.

Then either an http server could do a redirect from https://dist.ipfs.io/go-ipfs/latest to gateway.ipfs.io/ipns/<SOme link that never changes> OR if that was too hard to do then the script would just need to know gateway.ipfs.io/ipns/<SOme link that never changes>

Reading the last line of "versions" is also a reasonable way to do it in a script, I didn't know that was possible. I can work that into [https://github.com/internetarchive/dweb-mirror/blob/master/install_ipfs.sh] next time I do a revision (which will be for the next platform I try and install it on).

[mitra42]

I just updated that script at https://github.com/internetarchive/dweb-mirror/blob/master/install_ipfs.sh to handle the Raspberry Pi's I've got my hands on correctly. As far as I know there is still no shell script in the distro that can be used. It seems to work in a number of cases as long as the dist.ipfs.io site is up (and that site is currently down about half the times I try it)

[ThomasFreedman]

I seem to always run into trouble with ipfs-update. I use it every other month or so and every time my install script fails when I use ipfs-update to install ipfs.

I've seen this error before as well: could not find good install location. My workaround last time was to copy ipfs from where the tests were run to where I want it to live.

This time I'm using latest golang version (1.13) on arm61 arch (Raspberry Pi 3 / Raspbian 9.1), with ipfs-update version 1.5.2. ipfs-update successfully downloads and reports: success! tests all passed., followed by install failed, reverting changes..., then could not find good install location. What's wrong with $HOME/go/bin?

I see above Of course, it rarely picks a good location to install ipfs., so at least the problem is recognized. When will the anticipated fix for this be available, any ETA?

Any other workarounds besides the manual copy I described?

[mitra42]

Did you try my script linked above - if so, and it fails then would appreciate an update.

One thing I'd change if I reworked it (and we aren't using IPFS on RPI's any more for various reasons) would be to replace ipfs-update install latest with something like ipfs-update install latest || ipfs-update install latest || ipfs-update install latest || ipfs-update install latest since ipfs-update fails a lot of time due (I believe) to the gateways being overloaded.

In our dockerfile installs on the larger platforms we try 10x with a 5 min delay in between so that we wait out the single-point-of-failure.

[ThomasFreedman]

Not sure if this will help you or not but I wrote a bash script to install IPFS on Raspberry Pi3s (or most any Debian Linux OS, tho only tested on Raspian Stretch and a dedicated cloud host running Debian 9.9)

You can find it in my git repo at: https://github.com/ThomasFreedman/ipfs

Good luck.

[mitra42]

Great - looks pretty comprehensive were there any cases that my script handles that you didn't ?

[ThomasFreedman]

I believe so. For one thing my script has only been tested with Debian Linux variants and yours works with other systems.

Mine has more cmd line options, such as waiting between each step, setting maximum storage size and a few other things.

I've been using / tweaking mine for several months, and I find it doesn't always work for reasons typical of fast paced dev efforts like blockchain and ipfs. It's difficult to write a stable installer when the product is in flux.

Last bug I found (still not fully understood or resolved) had nothing to do with ipfs but is a bug related to cmd line arg processing. I hadn't used getopt(s) in a bash script before, but I tested all variations of each option when I created the script. The bug is that the descriptor for the -g option should allow for an optional parameter, however when no version arg is provided it doesn't detect that condition. Thus if you want to use go you must specify -g because -g without it fails.