Reconsidering GX

[Stebalien]

Current State

The current plan is to:

1. Make our libraries usable without gx by #4831 and by adding go.mod support.
2. Make it possible to update a transitive dependency without modifying other packages depending on that transitive dependency.
3. Make gx defer to the language's package manager for dependency resolution (when requested by the user). Take a look at https://github.com/whyrusleeping/gx/issues/179 starting at https://github.com/whyrusleeping/gx/issues/179#issuecomment-408243162.

There has also been some more recent discussion (https://github.com/whyrusleeping/gx/issues/200) about potentially creating a go-mod proxy that uses IPFS and using that instead or in addition to gx.

The issue currently blocking all of this is https://github.com/whyrusleeping/gx/issues/215. Once that is resolved, we can adopt both systems in parallel.

Unfortunately, that's only the first step. The next step would be implementing the improvements to gx mentioned in https://github.com/whyrusleeping/gx/issues/179#issuecomment-408243162. Without that, we can technically use gx along side go modules but we'll still have to manually keep the two systems in sync.

Why GX?

The primary arguments (that I know of) to use GX are:

1. Security: everything is content addressed. However, go modules have a go.sum file that we could check in to the repo to provide the same security, as far as I can tell.
2. Performance: GX uses content addressing so we can avoid fetching files that haven't changed and can perform fresh builds in CI without complicated caching. While go.mod doesn't provide the former, it does provide the latter.
3. Dogfooding: That is, gx uses IPFS. We'd like to eventually make gx into a versatile package manager that can be used by multiple languages. Using IPFS also brings along perks like being able to work in disconnected networks.

Proposal

So, this brings me to my final point. Given that:

1. We're already massively over-extended and have repeatedly failed to dedicate time to fixing gx.
2. Go modules appear provide equivalent security to gx through go.sum (as gx exists today, given that we're not authenticating gx's "lastpubver" file in any way).
3. gx is still a large productivity sink.

I'd like to seriously consider:

1. Adopting go modules as per https://github.com/ipfs/go-ipfs/issues/5848.
2. Dropping gx until it can live (usably) along side go modules.

The key questions we really need to answer is:

1. Does the go module system (assuming we check-in the go.sum file) provide equivalent functionality and security as gx (as it exists today)?
2. Can and should we somehow find the time to fix gx (given everything else we're working on)?

I know this will be a polarizing topic but please be constructive and rational.

[Stebalien]

cc @ipfs/go-team @libp2p/go-team

[Stebalien]

@jbenet pointed out that go modules don't help if a dependency gets removed from github. Supporting that use-case without gx would require something like https://github.com/whyrusleeping/gx/issues/200.

This is somewhat mitigated by the fact that we cut full source releases but it's still not ideal.

[magik6k]

go modules don't help if a dependency gets removed from github

We can maintain forks in some org, like we already do in https://github.com/gxed.

As for the proposal itself I'd say it's worth trying given how much time I waste on fiddling with gx deps, evan with all the tools. And then there is the fact that gx is rather unapproachable for outside contributors.

[Stebalien]

We can maintain forks in some org, like we already do in https://github.com/gxed.

While we can maintain forks, we'd have to make go-ipfs use these forks in imports which will cause problems for packages not using these forks (unless I'm misunderstanding the suggestion).

[djdv]

Can and should we somehow find the time to fix gx (given everything else we're working on)?

I believe so.

We'd like to eventually make gx into a versatile package manager that can be used by multiple languages

May align with our upcoming efforts on package managers.

Why gx

I am a fan of gx's concepts, but the implementation has been something to wrestle with previously. It's been improving though, and I think we have a foundation of a useful tool with good constructs, covering multiple languages.

Using a single interface to manage packages for multiple languages is a big plus in my book. And content addressable package repos, needs no defense.

That being said, I'm not attached to gx specifically. If we can achieve what we need with something else, I'm not opposed to dropping gx for it, until gx itself has matured.

Highlighting this:

have repeatedly failed to dedicate time to fixing gx

[hannahhoward]

This is perhaps a "side" issue to this question, but I am curious -- right now, the use of gx de-facto prohibits us from using a lot of outside libraries -- or at minimum, it discourages their use cause then you have to put it on GX, and then maintain our own versioning.

How do people feel about the use of 3rd party librarys? For example, as someone who likes to write tests, I was tempted to reach for https://github.com/stretchr/testify -- but didn't want to go the rigamarole of the debate about using it and bringing it into the project with gx. It doesn't actually support go modules yet but it looks like it will very soon now.

As someone coming from Ruby, JS, Elixir and other scripting oriented languages, I'm used to bringing tools and packages I didn't write. (and no one on the team wrote) But also, I know go is systems programming oriented and people generally use external 3rd party libraries less in these languages.

Anyway, the use of go modules would unlock the ability to use 3rd party libraries more easily, so I wonder how folks feel about this?

[magik6k]

We might be able to use the gomod replace directive with gx - from https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive:

The replace directive allows you to supply another import path that might be another module located in VCS (GitHub or elsewhere), or on your local filesystem with a relative or absolute file path. The new import path from the replace directive is used without needing to update the import paths in the actual source code

[Stebalien]

but didn't want to go the rigamarole of the debate about using it and bringing it into the project with gx.

We can pretty easily import external packages into gx using the gx-go import command. This will automatically generate package.json files for a package and it's dependencies and publish them with gx. The primary difference is that these files won't get committed to any git repo.

The tricky part here is really dealing with version conflicts. When you import a package into gx this way, you have to use the --map flag to tell gx about transitive dependencies that have already been imported into gx.

---

Just to be clear, the new gx lock system is designed to completely fix this issue. With the gx lock system, you should never have to explicitly import a dependency into gx. Instead, you're supposed to be able to run a command to:

1. Ask some other system (e.g., go get or the go module system) to resolve all dependencies.
2. Import these dependencies into gx on demand.
3. Write a single lockfile in the top-level package specifying the IPFS paths to all dependencies.

The key part here is that dependencies don't have to be modified.

However, I submitted this issue because the current gx lock system is missing some key components at the moment:

1. We only have a gx-go lock-gen command that builds the lock file from gx package.json files instead of building it using the go module system.
2. The story around linking dvcs dependencies gx-go link is still in flux (blocking us from using the current lockfile system in go-ipfs as it exists).
3. The UX/documentation needs a bunch of TLC.

To change this situation, we'll need someone completely dedicated to making gx awesome. Someone who:

1. Wants to own gx and lead the effort. Simply wanting to make gx less painful isn't enough.
2. Isn't split between gx and other high priority projects.

At the moment, I don't know of anyone who can fill that role.

[bigs]

i completely agree with @Stebalien that this is a full-time job, and is something we should hire for if we want to make it a focus. i also agree @djdv that this is something we should hire for. imo our lives would be made easier by:

• moving to go mod entirely for the foreseeable future
• attempting to hire or re-org someone into working on gx
• focus the efforts of gx on either:
  • integrating cleanly as a backend for serving data for go mod or
  • being fully featured enough as to include helper commands for building projects, updating dependencies, etc

[anacrolix]

gx is an obstacle to potential contributors, and no longer necessary with Go modules. Go modules now provide the versioning and security needed for the project. The other features gx provides are nice but at this point are just showcasing at a cost to the complexity and unfamiliarity of the project. I think gx should be revisited down the track, when Go modules use is more ubiquitous, and features like proxies and source archives are more certain (and by extension, gx has more licence to tap into IPFS). In the short-term, user uptake and ease of contribution are much higher priority.

Code being removed from GitHub is an issue that can strike any project, and reasonably exceeds the scope of project development I think (despite this project's long-term goal to make this a non-issue, it remains a goal and not a requirement).

The PR #5849 is on track to demonstrate that Go modules can operate and pass in all the CI environments.

[Stebalien]

Code being removed from GitHub is an issue that can strike any project, and reasonably exceeds the scope of project development I think (despite this project's long-term goal to make this a non-issue, it remains a goal and not a requirement).

I agree with everything except this. The argument "everyone else does this" doesn't really hold when there's something we can (and currently do) to avoid it.

On the other hand, frequent source releases and/or forking projects we depend on (linking them in with the "replace" directive) may be sufficient.

[hannahhoward]

There seems to be broad consensus that:

1. gx in its current state hinders development speed, significantly
2. To get gx to the point of being not a hindrance requires investment of an FTE working only on gx

It seems to me there are two separate decisions at this point:

1. What is right for go-ipfs to support go-ipfs development moving as quickly as possible?
2. What is right for the larger IPFS project -- especially in terms of adoption -- and does Protocol Labs, the company that can potentially sponsor an FTE to work on GX, want to invest in its development so that it can be a 1st class Go package manager? (And potentially be the package manager for go-ipfs, if not right now then in the future)

The answer to question 1, for the moment, seems relatively clear -- go-ipfs for the foreseeable future is best served by switching to go modules, as it's like to increase dev speed for existing developers and reduce cognitive load for new developers. (with an initial speed bump to make the switch)

The answer to question 2 is less clear, because at minimum, IPFS's overall roadmap puts package manager adoption as our top 2019 priority, so abandoning GX feels contrary to that. The more I think about it though, the more I think that question 2 is a decision that needs @momack2 or @daviddias input, cause it's more of a "overall goals for IPFS + Protocol Labs" question than a technical question.

(it might also be that GX just get's prioritized in our OKR planning in a way it hasn't before, and doesn't need a seperate FTE? I dunno... but want to also @eingenito in this discussion since he's the OKR guru ATM)

[bonedaddy]

This is a fantastic change! Go modules are amazing. gx is a nice idea but I've burnt many hours trying to work around GX issues. For example anytime my project updates dependencies for even non IPFS projects, I'll end up burning about 2 hours just getting GX to behave. It's gotten to a point where I simply do not update my core code base dependencies unless I absolutely have to because of the headache GX causes

I think gx as it stands is a massive barrier to adoption of the IPFS project as a whole. I would agree that yes code can dissapear from GitHub, but is mitigating that via a package manager really worth potentially stopping adoption of IPFS completely?

You all (protocol labs) have built up a very solid community and perhaps this can partially be solved by reaching out to the community for assistance with this issue? You all have a lot on your plate, and I'm sure there's people in the community that would be willing to help. I know I myself would be happy to help in whatever way.

[hsanjuan]

Ok, I'll break a lance in favour of Gx, despite all the painfulness that it causes to maintainers on libp2p and go-ipfs land, myself included. At certain level, it would make me a bit sad to drop Gx because:

• Gx is the IPFS package manager. It is counter-intuitive that we have package managers as 2019 priority and at the same time drop ours because it doesn't work well for our workflows. IMHO it gives a horrible signal even if Gx works very well (the Gx workflow is the problem).

• We are trying to demonstrate that decentralization works and things can be done differently. The possibility of fetching our depedencies from IPFS is not just a feature, it's at the core of what is important for us. We can't say "Put your websites on IPFS", "Put your datasets on IPFS" if we just decided to not "Put your code on IPFS" and defer to Github.

• The but we'll go back to it when it works safeguard makes me raise an eyebrow. It is contradictory that we haven't managed to dedicate enough time to Gx during the years that it was essential to the Go ecosystem, and we claim that once it becomes completely superfluous, we'll actually fix it and overcome any resistances to switch back to it after having changed everyone's workflows.

• Much of the perceived pain in go-ipfs (by users and contribs) was that it had (and has) rewritten dependency paths. imho the gains out of this policy has been way smaller than the losses and we should change this.

Moving everything and everyone to Go modules and dropping Gx will not be a light task. I don't have an idea of how much it will cost to "fix Gx", but we might as well prioritize it so that we can have Go modules and Gx side to side without having to drop Gx first? If we did not have to "bubble up" Gx deps, would Go modules provide any advantage to consider their adoption (the inflexible dependency resolution provided by Gx is also a powerful glue which provides a clear snapshot of the whole ecosystem at the time of a build).

[Stebalien]

I agree, I'd much prefer to keep GX as it dogfoods our own tech. However, the reality is that someone needs to become the GX maintainer for it to be usable. That's not just dedicating a little time to make it better. We need someone to really own it.

Really we don't actually need someone to own gx itself, we just need someone to own creating a content addressed package manager (that works with go, and hopefully other languages).

Much of the perceived pain in go-ipfs (by users and contribs) was that it had (and has) rewritten dependency paths.

That's only half of it; that's the pain seen by people trying to integrate go-ipfs. The other half is the pain and wasted time we feel whenever we have to bubble an update through the entire dependency tree.

IMHO it gives a horrible signal even if Gx works very well (the Gx workflow is the problem).

I think this will be entirely eclipsed by happy users/contributors. Creating a good package manager with a good UX is hard but GX has demonstrated that content addressing in package managers works.

would Go modules provide any advantage to consider their adoption

Regardless of what we do with gx, we need to support go modules.

the inflexible dependency resolution provided by Gx is also a powerful glue which provides a clear snapshot of the whole ecosystem at the time of a build)

Go modules actually provide the same guarantee. Once you update a version constraint on a dependency or a transitive dependency:

1. Go will deterministically compute the minimum versions that satisfy the version requirements. This means that, unless the go.mod file is changed, the dependency versions never change.
2. Go will hash every dependency and transitive dependency, storing these hashes in the go.sum file. This gives os the exact same snapshot guarantees.

[ghost]

From the libp2p project perspective, our main goal is gaining broader adoption. Every non-standard tool or concept that users or aspiring contributors encounter raises their cost to adopt libp2p. So, speaking purely from the perspective of getting libp2p adopted by 10+ major open source projects in 2019, go.mod seems preferable to gx.

[whyrusleeping]

Its exciting that we're finally at the point where recommended go tooling can give us most of the same guarantees as gx. In an ideal world, I'd like to be able to use gx to fulfil dependencies for go modules (my favorite feature is not having to download the same code multiple times), but given that I have no time to dedicate to maintaining gx and giving it the features it needs to not be oboxious, switching to go modules seems like the right approach.

Onwards!

[anacrolix]

I suggest #5849 be reopened, and that git tagging of releases be resumed. The tagging, and go.mod support in dependencies are not blockers to immediate adoption.

[Stebalien]

I'd like to fix the dependencies first. Without go-mod, we have no way to specify minimum version constraints which will make reliably updating dependencies painful and error prone.

[momack2]

@warpfork and @daviddias who I think might either have ideas on folks/communities who might make good dedicated gx maintainers or other ways we can support the work to mitigate this velocity damper in the short term.

[warpfork]

I have tricky feels on this.

On the one hand, I 100% am on board with the idea that it's important to aim for a future where we have package managers that DTRT while being powered by content-addressable decentralized forget-nothing secure goodness.

On the other hand, I think that's an incredibly complicated design space. (Right off the bat: "package" and "package manager" are powerfully ambiguous terms. I just came from a roundtable at the Reproducible Builds Summit about trying to define the word "package": we couldn't; it carries a lot of concepts and means different things in different contexts... and our experience so far in discovering that "UX is hard" with gx is also reinforcement of this.) So, I agree with everyone who's said that doing something that fits and is impactful in the long run requires a lot of resources -- both for very careful design, as well as the grinding details of implementation.

On the third hand, it's exciting that the Go community tools are starting to offer snapshotting and integrity guarantees. Given this, and given the new-contributor-friction argument... on the whole, I'm swayed to agree that yes, perhaps we should give go modules a shot.

On the gripping hand: I have to confess, I use git submodules for this on my personal projects and continue to regret nothing. Source control workflows for controlling source are surprisingly adroit.

---

In summary: I'm reserved about whether the Go modules system will be perfect for ages (probably not; nothing is), and definitely not effervescent with joy about Yet Another for-some-reason-language-specific package manager being part of my life... but those are reservations, not refutations. If the consensus is that using the Go modules system is on the whole going to be best for go-ipfs at this present moment, I'm onboard.

I really want to continue the research into content-addressable-native and reproducible-by-design package managers. If we step away from using gx here, let's not view it as a concession that this isn't important. At most, it's a concession that "we're not done yet" -- and the silver lining is that it's possible that we'll have a better time carrying out more research and development on this topic in a less center-of-the-stage place than the go-ipfs repo.

[phritz]

From the perspective of go-filecoin: we would love to move away from gx. It's a significant drag on velocity and clearly not something that we are capable of supporting at the level it deserves/requires.

[eingenito]

Are we ready to close this issue? To summarize I'd say there's general consensus that, all factors being considered, we should switch to go modules, and begin the work to do so asap.

[raulk]

From the libp2p perspective, we have discussed this on several occasions with a general consensus that we should experiment with go mod, and find the right hooks to enable go mod to pull packages from IPFS via (proxies were suggested), and to use merkle hashes to refer to releases and authenticate releases.

Reading the go mod plans for 2019, I feel even more confident about this: https://blog.golang.org/modules2019

It’s worth keeping an eye on how module authentication and mirroring progress. We could integrate IPFS via the mirroring API, create a multicodec to encapsulate their hashes, and use their indices to push all public modules to IPFS, much like the js team has done with npm.

It would be great to designate a person to actively participate in the design process upstream, and to advocate for what we need in terms of APIs, hooks, etc.

[jvsteiner]

rather unapproachable for outside contributors

This was my teams biggest worry when we started working with libp2p. when we realized we could use dep in our project, despite gx being used upstream, it was a big relief...

[eingenito]

Definitely not ready to close this issue yet. Thanks for the input Raúl and jvsteiner.

[b5]

Thanks to @Stebalien for setting a tone on this potentially-contentious topic. I've talked with @whyrusleeping on this issue at length, and lived with gx outside of PL for a little over a year now.

When I encountered gx during the tech audit phase, hashes in import paths had me briefly consider not using ipfs at all. In the end I plunged maybe 25 hours into making gx work for our organization. It hurt. Today we import go-ipfs as a gx-dep, and 6 of our 15 CI steps involve gx in some way. We've written makefiles that have allowed others to contribute to and build our project from source without really knowing what gx is or how it works. It's hard to handle, and every release of go-ipfs costs me roughly a day to update our various codebases. I have had to alter our technical roadmap because having gx in our ecosystem has created impossible builds (for example, try importing gx'd go-ipfs and github.com/google/go-cloud in the same codebase).

I do want to point out one thing though: gx is the second service built on IPFS that I take for granted, (the first being the hosted HTTP gateways at ipfs.io). I take it for granted because yes, the tooling is rough, but it's technology that we rely on every. single. day. I can't say that about many things, particularly about decentralized things.

Big, ambitious projects like IPFS derive great benefit from needing to solve real problems for real people. For what it's worth, I think there's only one major problem with gx in go land: it doesn't follow go idioms. In a world before go modules, it wasn't really an option for gx. Now that go modules exist I think someone should take a shot at making it work, even for a day, before calling this. If someone can get go get to work with gx, I'm convinced that would ease a lot of our frustrations. If we're dropping gx because we're all frustrated, that's bad. If we're dropping gx because it's creating slowdowns that are endangering other goals, then that's a very valid reason to do so.

Anyway, I'd fully support whatever is decided here. Go modules will make our life easier. I will be sad to see gx go, and would love to see it return. I want to say a big thank-you to @whyrusleeping. The fundamentals of gx are fantastic. I may lose my shit every time I see all that gak in import, but I agree with the ideas gx is getting at, and that's what matters more to me in the end.

[anacrolix]

There's no need to switch to or make go mod support available across all projects simultaneously. Because of minimum version selection, you can consume repos without a go.mod that later gain one without concern. When updating to to a new version of a dependency that has subsequently gained go.mod support, any new minimums will be applied appropriately in the dependent project. For this reason it should be possible to start adopting go.mod project by project as the need arises.

[Stebalien]

Please see: https://github.com/ipfs/go-ipfs/issues/5850#issuecomment-448467979.

[hannahhoward]

I continue to receive feedback that this is an important issue to discuss and potentially move forward with. I would recommend that we look at this either in an extended weekly core team sync or during our next in person meeting (hopefully in 2019!) and develop a clear plan for moving forward.

[anacrolix]

It's also making it difficult for me to test changes to libp2p with respect to IPFS without changing my workflow to accomodate gx. @Stebalien has written a tool that allows use of go.mod over IPFS to support those users that truly require the non-UI aspects of gx.

[lanzafame]

I am going to kick off migrating any go-ipfs dependencies that are listed as +incompatible by go mod as they require a little bit more work than a git tag and go.mod file.

[magik6k]

Should we close this as https://github.com/ipfs/go-ipfs/pull/6038 got merged?

[eingenito]

I think so.

[songjiayang]

We can use go module with ipfs GOPROXY.

[Stebalien]

@songjiayang https://github.com/Stebalien/ipgo (or for a bash version: https://gist.github.com/anacrolix/14e776bc5a7a5fb3f5c4035ab0c41a9b/revisions).