Is there a good way to mark a dependency as "in development"

[fasaxc]

I'm struggling to find a good workflow to work on our "product" repo and our "library" repo at the same time when I'm making changes that affect both. Is there a good way to tell glide to ignore a particular dependency and to allow go to find the one in my go path? I'm looking for a setting that'd be local to my development box. Right now I either check out my library code into the vendor dir or I delete the library code from the vendor dir before I build so that go picks up the gopath version.

[chris-garrett]

I'm also struggling with this workflow. It would be nice to have a flag to the package that would cause it to just rsync into the vendor dir from GOPATH or something like that.

It looks like this issue is related: (https://github.com/Masterminds/glide/issues/548)

[sdboyer]

very common problem. also, very not easy to solve, as it cuts to the heart of a lot of issues. the pm committee is wrestling with this quite a bit. i expect that'll be evident when we release our first round of docs...i believe that's scheduled for next week.

the big issue with wiring something into glide right now is that, because we lack a good solution that's safe, predictable, covers the bases, etc., anything added to glide itself would have nasty second-order effects, as well as creating a use pattern that we end up having to support, even as a legacy situation, in the future.

i'd love to see someone put together even a slapdash approach, and we could at least recommend that while we deal with the deeper issues. something like "run a daemon that notifies on fs changes and rsyncs the appropriate dirs over to the appropriate vendor dir."

[fasaxc]

Not sure there's any need for a daemon for MVP. How about:

• add a new glide.local.yaml, which overrides glide.yaml but doesn't get checked in (need somewhere to put the override config)
• allow me to set a local directory as a source (similar to how you can set an alternative repo to get a package from) or just allow a local flag, which make it default to using the gopath
• explicitly run glide up/install to do the rsync and vendor stripping as if it was a remote dependency.

[sdboyer]

the big issue with wiring something into glide right now is that, because we lack a good solution that's safe, predictable, covers the bases, etc., anything added to glide itself would have nasty second-order effects, as well as creating a use pattern that we end up having to support, even as a legacy situation, in the future.

[sdboyer]

actually...maybe that wouldn't be so harmful. i have to ponder.

sorry, i need to not answer design questions first thing after waking up.

[johnnadratowski]

@fasaxc @sdboyer @chris-garrett @mattfarina - I've been struggling with this recently as well, where we have 2 apps and a single shared library. I've attempted to solve this using local mirrors, and having the engineers run glide update after they check in to their local repo. The crappy part about this is, for a feature branch off trunk, it still wants to pull from trunk.

I want to avoid having them to symlink from the vendor folder as it's going to be cleared out on a glide update.

Here is the best workflow I can think of:

1. Have the engineers merge to their local trunk [in the lib]
2. Engineers run glide update [in the app] when they make a change to their local [lib] trunk
3. When they want to submit their PR, they create feature branch off [lib] trunk changes and submit the new branch as a PR
4. IMPORTANT: Put branch restrictions on the [lib] trunk for the project so no one accidentally pushes it. Also ensure that latest remote [lib] trunk is merged to feature branch before allowing a PR merge (this is specific to certain Git hosts, github recently supports this).

I would like to see this remedied by a simple update to the mirrors mechanism, allowing the user to specify always copying the current sources for a local file mirror. That way, the process is simplified, where a user would just need to run glide update after they change files on their local system, regardless of the branch they are on [in the lib].

Something like this:

glide mirror set https://github.com/example/foo file:///path/to/local/repo --vcs local

Where --vcs local tells glide "Hey, just copy whatever is in this file system w/all dependencies".

I was trying to find a way to do this w/o glide update, as it is easy to forget to do. The problem is, the way vendoring works, Golang doesn't know that libraries in the vendor folder are the same as libraries in the $GOPATH, so it's either put all libs in $GOPATH or put all libs in vendor.

[johnnadratowski]

There would also be some things that could help with this, like allowing the user to specify a single lib to glide update. So like glide update github.com/foo/bar

[johnnadratowski]

If you guys like this idea, I'll try to find some time to do a PR for it. I'm just really busy at work right now.

[bbuchanan-bcs]

I'm running into this as well. I'm heavily in favor of making a "local" vsc setting that just pulls directly from my file system. This makes it inline with the same workflow in python using pip. Please implement this.

[johnnadratowski]

@bbuchanan-bcs - I haven't gotten a chance to do it. I've never contributed to glide before, so I'd need to start looking at the sources. I've been hesitant to do it as the repo owners haven't responded to my comment, so I wasn't sure if I put in all the work it would have been accepted.

How to solve this problem seems to be a point of contention, based on the prior comments.

@mattfarina @chris-garrett @sdboyer @fasaxc - If you guys like this let me know, because I will find time to do this. I just don't want to work on this to no end.

[chris-garrett]

@johnnadratowski this would be awesome! I have stopped using glide because of this so I a big +1 from me.

[clarsonneur]

I'm in exactly the same concern described here.

@johnnadratowski Did you get time to work on this?

[reorx]

Another one who also struggled with this, here is my solution. Please note that I'm using dep recently, but the problem and solution are fairly the same.

The first thing is that I've written a script to accomplish this task: https://github.com/reorx/scripts/blob/master/localvendor.sh

The script localvendor does one thing: rm -rf a package under vendor/, then link the local repo to the same location.

Here is my situation, I have 3 repos, foo, bar, baz, all under namespace github.com/me. foo depends on bar and baz. bar and baz has no dependency relationships with each other. In foo's directory, I write a Makefile with command:

ensure-local:
    # the `glide up` for dep
    @dep ensure -v
    @bash localvendors.sh

this command will first run dep ensure to get all the dependencies to vendor/, then run localvendors.sh (ignored by git as a local only file) as follows:

#!/bin/bash
# file `localvendors.sh`
localvendor ../bar
localvendor ../baz

After ensure-local finishes, vendor/ will be like:

$ ls -l vendor/github.com/me
bar -> /Users/me/go/src/github.com/me/bar
baz -> /Users/me/go/src/github.com/me/baz 

So when you are developing this project locally, always run make ensure-local to update dependencies. When bar or baz changes its version, don't forget to update the Gopkg.toml file (or glide.yaml for using glide) to get the version specifications up to date. When doing packaging jobs in CI or other machines, just run dep ensure (glide up/install) and the build result should be the same as local.

This solution should works with most of the situations I can think of, and only costs very little effort and awareness. If you have any ideas about this, feel free to discuss with me :)

[sdboyer]

If you're using dep, https://github.com/GetStream/vg is also designed to help with this general workflow. (haven't used it myself, but we're in regular contact with the author, @JelteF)

[JelteF]

Virtualgo (or vg) is indeed built to solve this issue (as well as some others). It also works with glide, although the integration is not as good as with dep. This workflow should work fine however. If you want to try it out I recommend you use the branch from this pull request though: https://github.com/GetStream/vg/pull/8

That pull request makes the multi project flow much nicer to work with. It should be merged in and finalised somewhere next week. Also read the README from the pull request btw if you plan to use it, it's much better than the current one and includes the changes from the PR.

Any suggestions or remarks on the project are welcome btw.

[reorx]

Thanks @sdboyer and @JelteF , I've searched all around only to find issues other than solutions, vg looks really good, I'll give vg a try and use it for a while, to see if meets my situation precisely.

To @sdboyer , though it's a bit inappropriate to talk about another issue here, just want you to check my ideas about dep's version constraint and release management here, it's some complaints but with good heart and no offense, I hope you don't mind 🙂

[sdboyer]

@reorx of course not, no offense taken - constructive feedback is always welcome 🎉 i have a post-it up to actually respond to you over on the other issue 😄

[gggin]

@fasaxc @sdboyer @chris-garrett @mattfarina - I've been struggling with this recently as well, where we have 2 apps and a single shared library. I've attempted to solve this using local mirrors, and having the engineers run glide update after they check in to their local repo. The crappy part about this is, for a feature branch off trunk, it still wants to pull from trunk.

I want to avoid having them to symlink from the vendor folder as it's going to be cleared out on a glide update.

Here is the best workflow I can think of:

1. Have the engineers merge to their local trunk [in the lib]

2. Engineers run `glide update` [in the app] when they make a change to their local [lib] trunk

3. When they want to submit their PR, they create feature branch off [lib] trunk changes and submit the new branch as a PR

4. IMPORTANT: Put branch restrictions on the [lib] trunk for the project so no one accidentally pushes it.  Also ensure that latest remote [lib] trunk is merged to feature branch before allowing a PR merge (this is specific to certain Git hosts, github recently supports this).

I would like to see this remedied by a simple update to the mirrors mechanism, allowing the user to specify always copying the current sources for a local file mirror. That way, the process is simplified, where a user would just need to run glide update after they change files on their local system, regardless of the branch they are on [in the lib].

Something like this:

glide mirror set https://github.com/example/foo file:///path/to/local/repo --vcs local

Where --vcs local tells glide "Hey, just copy whatever is in this file system w/all dependencies".

I was trying to find a way to do this w/o glide update, as it is easy to forget to do. The problem is, the way vendoring works, Golang doesn't know that libraries in the vendor folder are the same as libraries in the $GOPATH, so it's either put all libs in $GOPATH or put all libs in vendor.

Now is 2018-10-22 in China, There is no type --vcs local, just remove --vcs local will solve this problem. But the repo url must be a github.com start not a go-project-name/xxx like name

[gggin]

@fasaxc @sdboyer @chris-garrett @mattfarina - I've been struggling with this recently as well, where we have 2 apps and a single shared library. I've attempted to solve this using local mirrors, and having the engineers run glide update after they check in to their local repo. The crappy part about this is, for a feature branch off trunk, it still wants to pull from trunk.

I want to avoid having them to symlink from the vendor folder as it's going to be cleared out on a glide update.

Here is the best workflow I can think of:

1. Have the engineers merge to their local trunk [in the lib]

2. Engineers run `glide update` [in the app] when they make a change to their local [lib] trunk

3. When they want to submit their PR, they create feature branch off [lib] trunk changes and submit the new branch as a PR

4. IMPORTANT: Put branch restrictions on the [lib] trunk for the project so no one accidentally pushes it.  Also ensure that latest remote [lib] trunk is merged to feature branch before allowing a PR merge (this is specific to certain Git hosts, github recently supports this).

I would like to see this remedied by a simple update to the mirrors mechanism, allowing the user to specify always copying the current sources for a local file mirror. That way, the process is simplified, where a user would just need to run glide update after they change files on their local system, regardless of the branch they are on [in the lib].

Something like this:

glide mirror set https://github.com/example/foo file:///path/to/local/repo --vcs local

Where --vcs local tells glide "Hey, just copy whatever is in this file system w/all dependencies".

I was trying to find a way to do this w/o glide update, as it is easy to forget to do. The problem is, the way vendoring works, Golang doesn't know that libraries in the vendor folder are the same as libraries in the $GOPATH, so it's either put all libs in $GOPATH or put all libs in vendor.

Now is 2018-10-22 in China, There is no type --vcs local, just remove --vcs local will solve this problem. But the repo url must be a github.com start not a go-project-name/xxx like name