gx-workspace, a project management tool

[whyrusleeping]

@lgierth has been throwing the idea of a gx-workspace tool around for a while, and i've had some thoughts about it recently. So heres my (hopefully coherent) thoughts:

gx-workspace will be a gx subtool like gx-go and friends. you will be able to make a package of type "workspace" and add dependencies to it.

Packages which are dependencies of a workspace package are 'managed' by the workspace. Updating one package in the workspace will recursively update that package in all other deps of the workspace, bubbling changes up throughout as needed.

gx-workspace will also be able to generate a 'repo' from its package so that other projects may easily sync up with your set of packages.

gx-workspace might have some special hooks when we get around to the 'package index.html' pages that turn its welcome page into more of a 'registry' feeling site.

example mocked up commands:

$ gx init
what language will the project be in? workspace
initializing package foobar...
$ gx import github.com/ipfs/go-cid
$ gx import github.com/multiformats/go-multihash
$ gx import github.com/multiformats/go-multiaddr
...
some time later
...
$ gx update github.com/multiformats/go-multihash
updated go-cid to use go-multihash vX.Y.Z!
$ gx-workspace mkrepo
QmBlahBlahBlah

[whyrusleeping]

This package could then be easily added to source control, and hooks could be added for automated testing, and pushing and other such fun stuff.

[geoah]

Being able to update all dependencies or at least be warned about incompatible versions would indeed be insanely useful.

I'm not quite sure if this is in scope and if it is possible, but since gx already has semver it would be really nice to be able to define up until where you'd like a package to be updated. NPM's (and others') caret ranges work nicely without changing the current structure of the gx package.

{
  "author": "whyrusleeping",
  "hash": "Qmxxx",
  "name": "go-libp2p",
  "version": "^4.0.1",
}

You should be able to now do gx update github.com/libp2p/go-libp2p, it will find the latext 4.x.x version, update it with all its dependencies and get it over with. -- The problem with this is that you'd need the devs to be tagging their repos at each version.

---

Could the version management for gx be build as a plugin on top of glide, gvp, or whatever else is out there? Glide's wizard is a pretty cool feature to get for free.

[ghost]

@whyrusleeping I generally like the workspace package idea, nice.

I'm not yet sure how to fit this into gopath, i.e. how to connect this with a "working directory" for each package. Many subtools will be very flexible there, but e.g. go is very strict (it must be at an exact filesystem path). A simple option is gx-workspace getting an install path from each package's respective subtool, i.e. gopath+/+dvcsimport and putting each package there. Sound good?

Another concern is that a workspace-bootstrap command should clone each dvcsimport, and not just use the package without repo, so that we can actually work with the directories managed by gx-workspace.

---

since gx already has semver it would be really nice to be able to define up until where you'd like a package to be updated.

(Let's discuss this in another issue)

I'm hesitant to consider flexible versions -- we'd sacrifice the reproducibility of dependencies, and very likely would want to bring it back again later, but in a much more complicated way (by means of a lockfile). The biggest difficulty though is to defer versions from from the git repo.

[Kubuxu]

@lgierth if we exclude windows from the fun, you can symlink directories inside the vendor/github.com/....

[ghost]

Huh I didn't think about that! I wonder if that might lead to funny situations like go seeing the exact same file at two different paths, e.g. go-ipfs/vendor/.../go-multihash.Multihash and go-cid/vendor/.../go-multihash.Multihash.

My own local setup is having all of ipfs/, libp2p/, multiformats/, ipld/, gxed/ in a convenient location, then symlinking them into gopath/src/github.com/, so that I'm technically working within gopath as far as go tools and imports are concerned. So at least for me, both options would work.

But to be honest I'm totally fine excluding windows for now. We can patch in a --clone-into-gopath flag for windows later, which would clone to gopath/src/ instead of vendor/.

[Kubuxu]

go-ipfs/vendor/.../go-multihash.Multihash and go-cid/vendor/.../go-multihash.Multihash will be different packages

[ghost]

I know, just wondering if that situation will happen. I'll just test

[ghost]

Another concern is that a workspace-bootstrap command should clone each dvcsimport, and not just use the package without repo, so that we can actually work with the directories managed by gx-workspace.

It can just do go get -d -u $dvcsimport

[whyrusleeping]

@lgierth lets be careful not to specialize this tool for a certain language/environment

[ghost]

Yes yes it'd be the job of the language subtool to provide an install path

[ghost]

I've been sketching what the workflow would look like with a gx-workspace tool, and I think we can easily do it without any meta package. See https://github.com/ipfs/gx-workspace#usage:

Usage

We're working directly in the respective package, no administrative overhead

> cd $GOPATH/src/github.com/ipfs/go-ipfs/

Let's clone/pull all the respective dvcsimport repos into GOPATH. Most of them will already be there and it's our "workspace".

> gx-workspace pull

Now bubble up the hashes through the tree, this leaves dirty package.json and .gx/. This will abort if any repo is dirty, overridable by -f

> gx-workspace update go-cid go-libp2p-interface-pnet

Run tests in all packages.

> gx-workspace exec make test

Now do the git dance, but only for repos which got touched

> gx-workspace pr feat/gx-update-123456
# which is the equivalent of:
> gx-workspace exec --only-deps --changed 'git checkout -b $branch && git commit -am "gx release $VERSION" && git push origin $BRANCH && hub pull-request -m "Update go-cid go-libp2p-interface-pnet"'

There's going to be some packages that are outside of this dependency tree, but still want these updates. They can copy over the updates:

> cd $GOPATH/src/github.com/ipfs/ipns-pub
> gx-workspace pull
> gx-workspace update-from go-ipfs

[Kubuxu]

I am still for a tool that would setup separate workspace for task like that due to how easy it is to then reverse the process, just remove the director tree.

[ghost]

Okay, I think that can be done without too much of a change. Setting GOPATH to a different dir, making sure the "root" package is symlinked there.

[whyrusleeping]

I would love to have this line:

> gx-workspace exec --only-deps --changed 'git checkout -b $branch && git commit -am "gx release $VERSION" && git push origin $BRANCH && hub pull-request -m "Update go-cid go-libp2p-interface-pnet"'

A bit more scriptable. For example, having the hub pull-request command be able to embed links to the other PRs we've created so far would be amazing. And also have a gx test thrown in there for good measure would be :ok_hand:

Also, its definitely worth checking that gx deps dupes doesnt output anything, that can help prevent a lot of dep issues (though this is currently broken for go-ipfs itself as two different packages are named sys)

[ghost]

This has been continued in ipfs/gx-workspace btw