search

hashicorp / go-getter

Package for downloading things from a string URL using a variety of protocols.
Mozilla Public License 2.0
1.62k stars 227 forks source link

IPFS getter #165

Open bigs opened 5 years ago

bigs commented 5 years ago

Is there any interest in adding IPFS support to go-getter? Over at protocol labs, we're experimenting with Nomad as a cluster manager for a test bed (i.e. launch many different p2p nodes and run simulations) and are interested in fetching artifacts off of IPFS. We do run HTTPS gateways to IPFS, so it's not immediately blocking, but I think it'd be great to have support added to the library. I'm happy to take this on if there is interest in merging it.

I envision two possibilities, and would support both:

1) go-getter interfaces with a local IPFS daemon. 2) go-getter uses IPFS as a library, creating a temporary peer and fetching from IPFS in-process.

Edit: PS: the rough first draft of this testbed will be on github soon!

apparentlymart commented 4 years ago

Hi @bigs! Sorry for the long silence here. I just happened to spot this while I was looking at the issues for another reason and it piqued my interest because I'm an IPFS fan. :grin:

With that said, I think having IPFS support embedded in this codebase would be a bit awkward since if memory serves the IPFS Go implementation has quite a heavy set of dependencies to impose on every go-getter caller, which includes Terraform and Packer in addition to Nomad.

Given that, my initial instinct here would be to start an issue about IPFS support in the Nomad repository to discuss it as a Nomad feature specifically rather than as a generic go-getter feature, and then if the Nomad team is amenable to it (and thinks go-getter is the best avenue to get there) then the actual detector and getter implementations could potentially live in a separate codebase imported only by go-getter callers where IPFS support would be useful, without imposing that additional functionality on other callers.

Interfacing with a local IPFS daemon would likely require fewer dependencies (maybe just net/http?) but I suspect it still couldn't be included as a default getter because it would need configuration for the location of the daemon, right? I could see it supporting a mode where the gateway address is part of the "URL" passed to go-getter, but at that point it would just reduce to the generic http getter, right? So it'd be better if the gateway were configured out of band and the detector/getter could just accept /ipfs/<cid> and /ipns/... as input.

I know it's been a long time since you opened this, so perhaps your interest in it has changed in the meantime, but hopefully if you're still interested then this reply will be useful.