Versioning Scheme

[jbenet]

In software, we've come to use things like semver to ensure programs and their dependencies interoperate well.

Problems with this:

• Forcing data publishers/researchers to follow a scheme does not seem fun or fruitful.
• semver focuses on API changes, and does not apply well to data.

Paths:

• Don't enforce anything. See what happens. This is flexible to "whatever you want to do." This is liable to yield a proliferating mess of "version" schemes. This seems like the worst thing to do.
• Find an existing standard that makes sense and use it. Are there well established (and sane) data versioning standards? I'm not too familiar with what's out there.
• data semver (or more researcher-friendly: Semantic Data Versioning), a semver fork tuned for data purposes. Perhaps something like this:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you REMOVE data.
MINOR version when you ADD data in a backwards-compatible manner, and
PATCH version when you CLEAN or REFORMAT data, without ADDING or REMOVING values.

Discussion welcome.

[davidad]

Reorganizing or reformatting a repository is a more disruptive change than adding new data, seems to me. I'd swap minor and patch.

Also, for data even more than code, it's important to adopt semver's "anything goes during 0.x" policy. On Jan 17, 2014 6:02 AM, "Juan Batiz-Benet" notifications@github.com wrote:

In software, we've come to use things like semver http://semver.org/ to ensure programs and their dependencies interoperate well.

Problems with this:

• Forcing data publishers/researchers to follow a scheme does not seem fun or fruitful.
• semver focuses on API changes, and does not apply well to data.

Paths:

-

Don't enforce anything. See what happens. This is flexible to "whatever you want to do." This is liable to yield a proliferating mess of "version" schemes. This seems like the worst thing to do.

Find an existing standard that makes sense and use it. Are there well established (and sane) data versioning standards? I'm not too familiar with what's out there.

data semver (or more researcher-friendly: Semantic Data Versioning), a semver fork tuned for data purposes. Perhaps something like this:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you REMOVE data. MINOR version when you ADD data in a backwards-compatible manner, and PATCH version when you CLEAN or REFORMAT data, without ADDING or REMOVING values.

Discussion welcome.

— Reply to this email directly or view it on GitHubhttps://github.com/jbenet/data/issues/13 .

[jbenet]

Reorganizing or reformatting a repository is a more disruptive change

Makes sense. I was considering data values the files attempt to describe, but in reality, files changing at all will cause a host of problems for any programs computing on them. Let's try this again:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you modify data or make incompatible file changes,
MINOR version when you add data in a backwards-compatible manner, and
PATCH version when you make backwards-compatible changes to metadata files.

metadata above is unfortunately broad and unclear. I refer to accompanying files that provide information on the data (docs, Datafile, etc). What would be a better word? Perhaps ancillary files? Perhaps the "backwards-compatible" description will get the point across.

"anything goes during 0.x" policy.

Indeed :)

[davidad]

I think "metadata" is good as it stands. It means "data about the data". As long as the changes are backwards-compatible and "the data itself" can reasonably be considered unchanged, that's a patch version.

It might do to an include an illustrative example or two, e.g. "adding, to a repository of astronomical images, new files describing the sky conditions during each capture would qualify as a patch update, if no new images are also included". On Jan 17, 2014 9:16 AM, "Juan Batiz-Benet" notifications@github.com wrote:

Reorganizing or reformatting a repository is a more disruptive change

Makes sense. I was considering data values the files attempt to describe, but in reality, files changing at all will cause a host of problems for any programs computing on them. Let's try this again:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you modify data or make incompatible file changes, MINOR version when you add data in a backwards-compatible manner, and PATCH version when you make backwards-compatible changes to metadata files.

metadata above is unfortunately broad and unclear. I refer to accompanying files that provide information on the data (docs, Datafile, etc). What would be a better word? Perhaps ancillary files? Perhaps the "backwards-compatible" description will get the point across.

"anything goes during 0.x" policy.

Indeed :)

— Reply to this email directly or view it on GitHubhttps://github.com/jbenet/data/issues/13#issuecomment-32608741 .

[feross]

I like the "data semver", neat idea!