Start a conversation about ipfs/gx-style imports in dep

[sdboyer]

hi! i'm one of the people behind dep, the "official experiment" package manager for Go. i've had an eye on gx for a while and been meaning to reach out, but things have been busy. i'm just back from vacation, though, which means i have a bit more mental bandwidth.

so, i've only read gx' documentation, i've not actually tried it (let alone used it in anger), but based on what i can tell from the docs, i have a few impressions:

1. gx is clearly not tied to Go specifically, so the use case overlap is only partial (read: i'm not here to try to assimilate or challenge gx's existence at all - i come in peace! 😄🤗🎉).
2. the way that gx/ipfs conceives of names/versions seems, at least from what i can tell on the surface, to be quite compatible with the way dep thinks of them.
3. it seems like dep should be able to add support for ipfs-shaped imports. doing that would entail teaching dep to speak with ipfs directly, and could well mean transparently working with existing projects that use them.
4. it also might be feasible to think about ipfs-driven registries (which would hide the details of ipfs from dep). we've got a few folks working on a prototype/spec for registries, as well as a lot of discussion, some extraneous, in golang/dep#286.

(3 & 4 aren't mutually exclusive, of course)

if this sorta stuff is interesting to to y'all, i'd love to chat about it more!

[whyrusleeping]

Hey @sdboyer, Thank you for reaching out! This all sounds very interesting, especially giving dep the ability to 'just deal with' gx and ipfs imports. I would love to chat with you some time about moving forward here, and I know some of the rest of the team would also enjoy taking part in the discussion.

cc @Kubuxu @noffle @Stebalien @lgierth

[sdboyer]

yay, awesome!

probably the easiest thing would be to set up a time for a prelim voice discussion, just because there's too much to cover easily in an async context like this. on that note, lemme invite everyone to package.community. we don't necessarily need to talk in the discord - though it may be a good place to coordinate details for voice - but it's something you'd probably be interested in, anyway.

[Kubuxu]

One of the reasons for creation of gx was last or build reproducibility without vendoring. As we started splitting out libs from main codebase, simpler vendoring was no longer an option. dep seems to solve that problem.

---

go-ipfs recently has gained ability to directly address and fetch git objects (using the same hash as the git object/commit itself) it would be interesting to explore it usability with dep. It leads to possibility of a (optional) system where Gopkg.lock could reference and fetch dependencies via IPFS in fully distributed manner.

[sdboyer]

build reproducibility without vendoring...dep seems to solve that problem.

a lot of it, yes, though dep itself still doesn't do anything about left-pad type problems. that's one thing that registries and/or ipfs could certainly help with.

go-ipfs recently has gained ability to directly address and fetch git objects (using the same hash as the git object/commit itself) it would be interesting to explore it usability with dep. It leads to possibility of a (optional) system where Gopkg.lock could reference and fetch dependencies via IPFS in fully distributed manner.

interesting! and possible, though complicated by the fact that we have to service the same four vcs that go currently does - git, bzr, hg, svn. which is to say, it could be useful for the git case, but it's not sufficient to cover everything dep has to cover.

[Kubuxu]

This is where IPFS could solve the left-pad problem. For the cases of the other vcs, something we use currently can be useful. The hash of the package is quite arbitrary (it is specific to ipfs) but it is unique identity of this version of this package. This hash could be stored in the Gopkg.lock.

[sdboyer]

This hash could be stored in the Gopkg.lock.

yeah, this is what i thought you were thinking, and it's not impossible. we're actually on a trajectory to something very similar right now: we just merged logic for doing tree hashing. those digests would be stored in Gopkg.lock, and providing us a single, normalized way of verifying across all the underlying VCS types, and act as a vital component in doing vendor/ verification (golang/dep#121). further, they would - i hope - eventually be used as the basis for a content-addressable storage area. very, very similar to ipfs principles, for sure. (it's "i hope" because that's a question for how things work after dep has been integrated into the go toolchain, which is, of course, not something i control, though there's generally a lot of agreement between the relevant folks on the Go team and i about this)

my immediate concerns about it are that the hashing algorithm we use has some quirks:

1. we normalize CRLF to LF unconditionally (so, more aggressively than e.g. git, which tries to differentiate between binary and non-binary files) so that file trees with platform-native line endings have the same hash digest
2. we skip vendor, .git, .svn, .bzr, and .hg dirs

these aren't things we're wedded to, but they are important considerations that we can't ignore.

[Kubuxu]

we just merged logic for doing tree hashing. those digests would be stored in Gopkg.lock, and providing us a single, normalized way of verifying across all the underlying VCS types

That is great to hear, it was one of the concerns I had with Gopkg.lock files

There is quite a big step from directory tree hashing to having something be content addressable (you can allow for much more destroyed data during hashing) especially in case of trees.

my immediate concerns about it are that the hashing algorithm we use has some quirks:

Given that I would say it would be best (if we were to move forwards with something) to keep verification hash and content address separate.

---

I think those are the most important use cases for why gx was created:

• completely reproducible builds, I might be wrong but dep currently doesn't provide a way to do that if package is to be used as library (example: go-ipfs is being used as library by OpenBazaar, we both want to be sure we are using the same exact code, I think go-ipfs' Gopkg.lock would be ignored in this case).
• content addressable packages:
  • no central point of storage that can lead to disruption if one DNS flips upside down (famous: I can't work because GitHub is down and I can't download those packages).
  • works great offline or in remote location - packages are cached locally and can be shared in local network seamlessly
  • great caching properties
  • easy to backup
  • no left-pad alike problems, once you have a package you can keep it alive it doesn't matter what happens with its origin

There are probably some more (@whyrusleeping will know).

I have been playing with dep over last few hours, if it was about to work/cooperate with gx in those two areas it would be just awesome.

[vielmetti]

A note that this problem came alive today with the go-bindata problem, about which more see https://github.com/a-urth/go-bindata/issues/1 (very similar to the left-pad issue).