Proposal for lockfile versioning

[indirect]

This proposal tries to increase Bundler's expected backwards compatibility in future versions, while still allowing changes to the lockfile format if they are needed in the future.

Rendered text

[indirect]

/cc @colby-swandale @hsbt @deivid-rodriguez @bronzdoc @duckinator @eregon if you have feedback or comments on this, or even if you are just ok with this idea and would like to do it, I would love to hear from you.

[colby-swandale]

\cc @schneems

[eregon]

This looks great to me, and I think it will provide a much better migration path for newer versions of Bundler. The Reference-level explanation is pretty close to the suggestions in my blog post :smile: Thank you for creating this RFC.

[hmistry]

I like the idea of versioning lockfiles very much. I think it's a good way to decouple improving lockfiles and bundler versions while allowing backward compatibility. I would consider if Gemfile should also have versioning incase you'd like to make changes to it or have new features... food for thought.

With versioned lockfiles, I'm not sure I see a need for having bundler versions in the lockfile but maybe there is some use case or it doesn't hurt.

Normally I'd also think about how you'd migrate from one version to another but I don't see this being an issue here since bundler will parse the Gemfile and regenerate a new lockfile version. But let's say if this were Yaml files with user set fields, then migrating old Yaml files to a new version while transferring user data becomes an issue if it's not an easy 1 to 1 map (something I'm facing in another project)... again I don't see this being an issue here.

I think it would be good to specify an end of life plan for lockfile version support from the beginning to set expectations. You don't want to have the same issue 10 years from now having implementation complexities to support old versions like you went through with Bundler.

[schneems]

Thanks a ton for this. The proposal matches closely what we discussed at RubyConf. I like it.

Sorry for the delayed response, I'm giving this my 👍

[schneems]

One suggestion: I would like if there is a canonical location that documents what version of bundler introduced what version of the lockfile. For instance "Bundler 2.0.0 introduced lockfile format v2". While presumably getting a lockfile vers error would lead the developer to upgrade to the latest bundler, if for some reason there is a bug, feature, or some other concern doing that then it would be helpful to not have to guess and check which bundler versions provide support for which lockfile formats. Documenting the support somewhere will be helpful for that case and possibly for debugging.

[duckinator]

I'm definitely in favor of adding lockfile versioning. I'll go through your "unresolved questions" section and give my thoughts on some of them.

What implementation option do we want to choose to support installing v1 and v2 lockfiles, as well as allow one-way upgrades from v1 to v2? Separate lockfile parsers per version? Version flags and lots of conditionals?

I think it's worth having the parser and such for each lockfile version be separate implementations.

Here's my reasoning for that:

1. It allows us to rip out old cruft in new lockfile versions. This is not something we could do easily if we try to make one parser support every version.
2. It allows us to make more drastic changes, like completely changing the format.
3. It allows us move old parsers out to Bundler plugins — as usage of a lockfile version drops, we can remove them from the primary codebase while still providing a method for people to use them if necessary.
4. If anything winds up being truly 100% equivalent between two lockfile versions, we can add a mechanism to allow them to reference an old version's functionality. This feels a lot more reasonable and maintainable, to me, than having every lockfile version use the same code by default and just have a pile of conditionals everywhere.

How should we communicate with users about this? Bundler.io and RubyGems.org blog posts? Conference talks?

Something along these lines, perhaps:

• have a dedicated page discussing the overall topic on either the RubyGems or Bundler website (whichever is deemed more appropriate) and documenting bundler/rubygems versions support which lockfile versions, which links to separate pages for each lockfile version discussing any problems/limitations, new features, etc.
• on whichever website doesn't have that page, add a page about it that is nothing but a short explanation and a link to the other one. (that way folks can find it even if they go to the "wrong" website!)

[indirect]

Thanks for the discussion, everyone. Let’s do this!