vendor: difficult to add/update/remove individual deps

[dt]

After changing/adding/removing a single dep, Glide re-resolves the transitive closure of all dependencies -- which is "correct" since that new/changed dep could transitively depend on a new dep or different version of an existing one. Unfortunately, when no version preference for a dep is expressed, either from a direct pin or pin in a transitive dep, glide simply picks the latest, even if a previously resolved version would satisfy the requirements - thus, in effect, adding or changing any dependency updates all unpinned dependencies. While this is fine when we want a global update, it can interfere with feature work when we want to bump or add a single dep as part of a feature change, without potentially introducing unrelated changes in stability-affecting core libs (grpc, protobuf, rocksdb, etc).

Most users of glide appear to use specific version or release branch pins, and thus do no appear to have this issue, as they can unpin or change the pin for a specific dep or deps to update it in isolation. However, as this becomes the norm, it combines with our usage to exacerbate the issue of unintended version changes: in addition having unintended deps jump to latest, we also have deps jumping to the versions specified by our dependencies, since their transitive preferences are the only preferences glide has, as we don't express any ourselves. If we actually had no preference, this would be fine, but in reality we usually do in fact prefer not to use those older versions, it is just not documented in our glide configuration.

Adding version pins on the deps we actually care about would potentially help resolve both of these issues.

One drawback to adding more pins is that it increases the friction when we do want to do a global upgrade. This could potentially be partially mitigated by dividing our pins into two sections: those due to current blocking issues upstream and those simply to provide more granular control over resolutions, with the later being more easily commented out en masse.

[dt]

@tamird are there issues with pinning more of the deps we care about other than the added friction to doing across-the-board dep updates?

[tamird]

In my experience, "deps we care about" has a tendency to creep. Other than that, no.

On Thu, Jan 26, 2017 at 4:38 PM, David Taylor notifications@github.com wrote:

@tamird https://github.com/tamird are there issues with pinning more of the deps we care about other than the added friction to doing across-the-board dep updates?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cockroachdb/cockroach/issues/13138#issuecomment-275522116, or mute the thread https://github.com/notifications/unsubscribe-auth/ABdsPNx7dS1PmJxWRhlw3LTDHg3ofs38ks5rWRJtgaJpZM4LtsP6 .

[dt]

I'm guessing we will indeed need to cast a somewhat wide net there (as in a decent % of our direct imports). As an example: currently a resolve downgrades to the version of tablewriter that etcd wants, but in doing so discards knz's fix that he upstreamed -- he picked that up by manually editing the resolved version. Since we do now have a preference for a specific upstream version, it looks like we need to pin there to make that resolve stable/reproducible.

[petermattis]

FWIW, I have extreme anxiety about using glide up. For the recent RocksDB 5.0 upgrade, I edited the SHA in glide.lock and did glide install. That workflow generally works ok, but there are a lot of manual steps involved:

• Bumping the version in cockroachdb/c-rocksdb
• Picking up that SHA and adding it to glide.lock and running glide install
• cd vendor and create a new branch to push to cockroachdb/vendor
• Create a PR in cockroachdb/cockroach

I wish the above could be streamlined.

[dt]

I guess the question is if we'd want to streamline that workflow at the expense of the update-everything-to-latest workflow being slightly less streamlined -- I'd say yes, as optimizing for the former seems important to maintaining stability.

[petermattis]

Yes, I agree that optimizing for the single update workflow is more important. In fact, I think we should move to a much slower regular cadence for the update everything workflow. Strawman: update everything once at the start of the release cycle. Perform single dependency updates as necessary to pick up bug fixes or needed feature additions.

[dt]

"Fixed" in #13471