Discussion of go-libp2p file bloat, vendoring issues

(Raw notes from discussion 8-Oct-2018 @lgierth @raulk @mgoelzer @Kubuxu @vyzo +add yourself)

History

go-libp2p extracted from IPFS initially
- more can be extracted (the host, identify, tests)
- bitswap relies on go-libp2p package which pulls in everything
- raulk: godepth and gx --deps or (gx --tree)

What about pulling interfaces out into an interfaces package?

Arguments against: (1) just a huge bunch of work (2) interfaces will change a lot (Kuba, but Lars disagrees)
- Explanation of (2): we would have to update the hashes
Raul: but this is weird in any language? will feel unfamiliar?
Each set of repos (like all muxers) has a go mod dedicated to it
- Them implementers can pull in what they need
- Lars: ok, say I'm going to use only yamux, will it pull in all the other muxers?
  - Kuba: no, bc a module is defined by a text file, so it would pull in only yamux

What about go modules?

We're not dumping gx anytime soon
gx could live on as a lock file, but go modules could be used to modularize the library
- go mod stable is 2 go versions away (~9 months from now)

Mono repo - would it help downstreams?

Could Ethereum Whisper exclude all non-dotgo files?

Yes, because we are building with go build so compile process cannot pull in anything that is not a .go
Could also exclude all *_test.go

Competing libraries on HN:

(forget name) - a one-off p2p library with no upgradability, muxing
need to convince community that the modularity we've created is worth the bloat

What about just having a go-libp2p-interface package?

Central location for how to use libp2p for different use case

Problem: there is no central place that explains "what does libp2p provide for me?", "what repos do I need to pull in to get just {transport, identity provider}

What is eth's goal?

Whisper v6
Two versions:
- devp2p (default)
- libp2p (behind an experimenrtal flag)
  - But just this tiny test option is forcing them to pull in 590 files!

Short-term actions we could take

We don't want to force them to change how they do their build (ie, force them to remove non-.go files)
Fix this: root go-libp2p package still has dependencies in there
Going through dependencies are completely unusused
- Which ones are that use only a tiny fraction of a big library (could just copy the code)
Putting interfaces into one place
go modules - figure out how this would look with go modules

I will quickly add a few points I raised with @raulk over lunch in Glasgow.

What about just having a go-libp2p-interface package?

Technically, this already exists, in part, in the specs repo, they obviously need further detail and I think the should fill the role of the place to go to figure out what libp2p module to use for different use cases. I believe that placing the concrete go definition of those interfaces in a single repo on serves to separate them further from the code that is the reference implementation for that interface.

At the end of the day, eth maybe vendoring a lot of files but that is the trade-off for the modularity of libp2p which as another user of libp2p (ipfs-cluster), it is super useful knowing what is being used by each package and only having to bring in what is required.

libp2p / notes