version id scheme

[pixelzoom]

The version id for PhET sims has (imo) gotten a bit confused with the introduction of PhET-iO. The 'phetio' brand identifier has been tacked on in a non-general way, and I have yet to see anyone provide a coherent/comprehensive description of the version id's syntax and semantics (getVersionForBrand.js included).

Current version id scheme

Here is the current format for public, dev and RC versions respectively, when brand is 'phet':

1.1.0 1.1.0-dev.1 1.1.0-rc.1

So far, so good. Now here is the current format when brand is 'phetio':

1.1.0-phetio 1.1.0-phetiodev.1 1.1.0-phetiorc.1

In the public version case, the build process adds a '-phetio' prefix. In the dev and RC cases, the build process reads the version id from package.json and replaces 'dev' and 'rc' with 'phetiodev' and 'phetiorc' respectively.

A few problems with this scheme:

(1) It modifies what the developer puts in package.json. (2) It modifies test and public versions differently. (3) It combines the "test type" and "brand" info, making it more difficult to parse. (4) It is not general enough to support future test types or brands. (5) It's difficult to write a "general form" description.

I'm proposing that we revised the version id scheme, as described below.

Proposed version id scheme

The version id scheme will consists of 3 chunks:

number[-test][-brand]

where:

• number describes the version number • test describes the test (absence indicates a public version that has been QA tested) •brand describes the brand (absence indicates 'phet' brand) • square brackets indicate an optional chunk • hypen is the separator between chunks

Expanding the chunks, we have:

major.minor.maintenance[-testType.testNumber][-brandName]

where:

• major = major number, sequential integer • minor = minor number, sequential integer, resets to 0 when major is changed • maintenance = maintenance number, sequential integer, resets to 0 when minor is changed • typeType = the type of test, currently 'dev' or 'rc' • testNumber = sequential integer indicating the test number • brandName = the brand name, e.g. 'phetio'. Absence indicates 'phet' brand.

The developer specifies the number[-test] portion of the version identifier in package.json. The build process add the [-brandName] portion, if applicable, to each supported brand that is built.

Examples:

1.5.0 = public version of 'phet' brand 1.5.0-phetio = public version of 'phetio' brand 1.5.0-x = public version of brand 'x', where 'x' is a client or some future brand 1.5.0-dev.1 = dev test for 'phet' brand 1.5.0-dev.1-phetio = dev test for 'phetio' brand 1.5.0-dev.1-x = dev test for brand 'x' 1.5.0-rc.1 = RC test for 'phet' brand 1.5.0-rc.1-phetio = RC test for 'phetio' brand 1.5.0-rc.1-x = RC test for brand 'x'

Another advantage of this scheme will become apparent if/when we publish all supported brands for a sim simultaneously. We would want all related files to live in the same directory, with version ids that are clearly similar. For example, for a sim with supported brands 'phet', 'phetio' and 'x':

public versions, all located in 1.0.0/ directory: 1.0.0 1.0.0-phetio 1.0.0-x

dev versions, all located in 1.0.0-dev.15/ directory: 1.0.0-dev.15 1.0.0-dev.15-phetio 1.0.0-dev.15-x

RC version, all located in 1.0.0-rc.1/ directory: 1.0.0-rc.1 1.0.0-rc.1-phetio 1.0.0-rc.1-x

Labeling for discussion at developer meeting.

EDIT (@samreid): fixed a typo

[samreid]

Prior discussion about how to name the versions is in https://github.com/phetsims/phet-io/issues/623 and it is codified in chipper/js/getVersionForBrand.js

[pixelzoom]

The current scheme of inserting "phetio" in front of "dev" and "rc" in the version identifier breaks the detection scheme that we approved in https://github.com/phetsims/joist/issues/406#issuecomment-284893658. One of the hazards of not having a well-defined syntax for version id.

[pixelzoom]

For a summary of other problems with the current version id scheme, see https://github.com/phetsims/joist/issues/411 (clean up version identifier).

[pixelzoom]

Here's another possible general form that is closer to what we have now. The brand chunk comes before the test chunk:

number[-brand][-test]

E.g.:

1.0.0 1.0.0-phetio 1.0.0-rc.1 1.0.0-phetio-rc.1 1.0.0-dev.1 1.0.0-phetio-dev.1

This would probably require fewer changes to the build process. But imo it's preferable to put the brand chunk last, since it's absence indicates an implicit value ('phet').

[samreid]

It appears that the semantic versioning rules (which we looked at previously) does support hyphens in pre-release version identifiers. http://semver.org/#spec-item-9 . However, I'm not sure that we should be using "pre-release" style to identify production versions like "1.0.0-phetio".

[samreid]

Brainstorming: what if instead of changing the version name for phet-io sims, we change the project name.
Currently: faradays-law 1.0.0-phetio

Proposed: faradays-law-phetio 1.0.0

This idea was prompted by on the semantic versioning link above.

[samreid]

They are different applications, so doesn't it make sense that they would have different names?

[pixelzoom]

They are not different applications. They are the same application with different branding and different features enabled.

[samreid]

They are not different applications. They are the same application with different branding and different features enabled.

They also have different code (phet-io sim versions include phetio.js internally and wrappers externally).

UPDATE: build.json shows all of the code differences in the sim:

"phet-io": {
    "preload": [
      "../sherpa/lib/jsondiffpatch-0.1.31.js",
      "../phet-io/js/phetio-query-parameters.js"
    ],
    "phetLibs": [
      "phet-io",
      "phet-io-website"
    ]
  }

[pixelzoom]

So... phetio is phet with additional features (and code for those features) added, right?

[samreid]

Correct, there is added code and different brand metadata.

[samreid]

Would it be simpler from an engineering standpoint if we only had one file (which supports both phet and phet-io)? If so, we would need @kathy-phet to approve this before moving forward.

The PhET-iO version contains all of the dependencies.json for everything.

This is parallel to the conversation we had earlier about bundling all of the translations into a single file. Maybe bundling all of the "brands" into a single file would provide the same benefits.

CURRENT STATUS: Let's sleep on it and discuss at an upcoming developer meeting.

UPDATE: The PhET and PhET-iO simulations have very different licensing, so likely cannot be combined into a single file.

[samreid]

We are discussing making phet-io development part of the normal process and unifying the version numbers (and branch names) and having the build process build both artifacts.

[samreid]

Hence we would take the brand out of the version identifier and branch names.

@jonathanolson and @zepumph and @samreid will work together to investigate how to make this happen.

[samreid]

As part of https://github.com/phetsims/chipper/issues/567 we will investigate removing brand from version and branch.

[pixelzoom]

In 9/18/17 Slack discussion (developer channel), @jonathanolson needed to publish a version of pendulum lab from a branch named "ubc-study" (see https://github.com/phetsims/pendulum-lab/issues/193). What should the version id have been? There was much discussion and no consensus. The current version id scheme doesn't support something like "ucb-study-dev.1". How does the need to publish from branches that aren't named "major.minor" (e.g. "1.2") after the version id scheme?

[pixelzoom]

11/6/17 dev meeting notes (approved by @kathy-phet)

Current dev directory structure:

faradays-law/

  // brand=phet
  1.0.0-dev.1/
    faradays-law_en.html
  1.0.0-rc.1/
    faradays-law_en.html
  1.0.0/
    faradays-law_en.html

  // brand=phet-io
  1.0.0-phetiodev.1/
    faradays-law_en-phetio.html 

  // one offs
  1.0.1-sonification.1/
    faradays-law_en.html   
  1.3.7-phetiodevtest.1/
    faradays-law_en-phetio.html

Desired dev directory structure:

faradays-law/

  // brand=phet
  1.0.0-dev.1-phet/
    faradays-law_en-phet.html
  1.0.0-rc.1-phet/
    faradays-law_en-phet.html
  1.0.0-phet/
    faradays-law_en-phet.html

  // brand=phetio
  1.0.0-dev.1-phetio/
    faradays-law_en-phetio.html
  1.0.0-rc.1-phetio/
    faradays-law_en-phetio.html
  1.0.0-phetio/
    faradays-law_en-phetio.html

  // one off, brand=phetio
  1.1.0-dev.1-phetio-sonification/
    faradays-law_en-phetio-sonification.html
  1.1.0-dev.2-phetio-sonification/
    faradays-law_en-phetio-sonification.html
  1.1.0-phetio-sonification/
    faradays-law_en-phetio-sonification.html

  // one off, brand=phet
  1.1.0-dev.1-phet-foobar/
    faradays-law_en-phet-foobar.html
  1.1.0-dev.2-phet-foobar/
    faradays-law_en-phet-foobar.html
  1.1.0-phetio-sonification/
    faradays-law_en-phet-foobar.html

Decisions: • Version number (major.minor.maintence) remains in lock-step across all brands and one offs. • Put brand at tail of directory/file names, for all brands. • Brand names and one-off names are limited to alphanumeric chars - no punctuation or special char. • Change brand "phet-io" to "phetio". Typically use camelcase for other brands, e.g. "adaptedFromPhET" or "widgetsInc". • Put brand and (optional) one-off name in directory and file names on dev server. • "-phet" is stripped out of directory and file names when production versions are deployed to the production server. (AP, KP: We will not change the names on the production server, too problematic.)

General form of version id: major.minor.maintenance[-test]-brand[-oneOffName]

[mattpen]

Will the new versioning scheme be an automated step or still be a manual process?

If it is automated should it go in chipper (meaning old versioning scheme still used for some sims) or the build-server (would not be used for dev deploys)?

If we want to include this in perennial, would it be a good idea to migrate devDeploy to the build-server?

[samreid]

The heart of chipper 2.0 is automating things to prevent manual errors & labor, so I imagine we would want to codify and automate the versioning too.

[jonathanolson]

I'd also be interested in whether we would want to mark debug versions (with assertions included and potentially enabled, see https://github.com/phetsims/phet-io/issues/1149) with 'debug' information in its version.

Proposed version scheme doesn't have ways to add on any "in-the-future" additional tags, like -debug.

Thoughts?

[jonathanolson]

Or, since we're also not including the locale (affects the build) in the version, presumably we wouldn't include whether it's a debug build?

[samreid]

How does the locale get specified?

[jonathanolson]

I guess I'm trying to figure out conceptually what the "version" would include. Yotta and such cares a lot about what locale it is (and it may matter whether it's a debug build), and some things care about our build timestamp (which is how we can tell translation updates apart).

Additionally, it seems the version should also include a "one-off" identifier field, so that the directory/etc. can be fully determined from the "version".

[zepumph]

Currently is the only thing that specifies the locale in a built sim the html file? If so, and yotta doesn't mind managing locales this way, perhaps we could get away with yotta handling debug mode in the same way, with out touching the version?

Conversely maybe it would be better to maintain specificity on the version/directory level rather than just with different html files. It also seems like we may want consistency between built versions based on all of these factors (one-off/debug/locale/etc).

[samreid]

Conversely maybe it would be better to maintain specificity on the version/directory level rather than just with different html files.

It seems our choices are roughly:

faradays-law/1.0.0/faradays-law_en.html faradays-law/1.0.0-debug/faradays-law_en.html

vs

faradays-law/1.0.0/faradays-law_en.html faradays-law/1.0.0/faradays-law_en-debug.html

I was envisioning the second option, that the debug version would be supplied with each build.

[jonathanolson]

I was also thinking of the second version.

Would we have faradays-law_en-phetio-sonification-debug.html? We'd want to decide on an order to the flags.

[pixelzoom]

Note to self: review comments on this issue that occurred since 11/17/17.

[jbphet]

Personally, I'd prefer version ID instead of structured directories. The proposal for the change to version ID as I understand it is:

major.minor.maintenance[-test]-brand[-oneOffName][-debug]

We should decide if -debug is the only thing that can appear in this spot or if, like "-test" and "-oneOffName" it can take on different values. For now, my understanding is that -debug is the only proposed value. If it can take on more values, we will need to figure out how to distinguish it from other values, such as the oneOffName.

This doesn't seem too objectionable to me, especially since the -debug versions will rarely if ever be provided to users outside of PhET (I think). But it is starting to feel like we are trying to cram an awful lot of information into the version ID, and IMO if anything else comes up we should take a pause and investigate some alternatives, such as having some easily decipherable metadata embedded at the top of the built file with the additional information that we need.

[zepumph]

I think that the proposal to version ID is

major.minor.maintenance[-{{test}}]-brand[-{{oneOffName}}][-debug]

Where {{}} represents a template that changes, but -debug is just that string. I think this is what @jbphet was saying, but I want to be explicit.

There are many hyphens in this schema. Is it possible that an underscore would help differentiate components and decrease parsing complexity? Maybe something like

major.minor.maintenance[-{{test}}]-brand[-{{oneOffName}}][_debug]

the -debug versions will rarely if ever be provided to users outside of PhET (I think)

Developing these debug versions with assertions and such in them is for phet-io clients during their development as well.

[pixelzoom]

Personally, I'm losing the ability to comprehend PhET's versioning and publishing needs. They have become way too complicated, and should be someone's primary responsibility (as I suggested long ago in https://github.com/phetsims/chipper/issues/567#issuecomment-298974310). So I'm afraid that at this point I don't have any additional thoughts on this issue.

EDIT by @samreid: typo fix

[jonathanolson]

A few more notes:

• "-phet" is stripped out of directory and file names when production versions are deployed to the production server. (AP, KP: We will not change the names on the production server, too problematic.)

  This sounds like it may bite us later if we just search for that string (may match other parts). If we have a one-off name starting with phet, does it just get mangled horribly? Or do we need to fully parse filenames out?

• Versioning proposal means files under /build/ get changed, so phetmarks and all of our tooling that expects build/${sim}_en.html to exist needs to change.

• If you check out an older snapshot of a sim (maintenance release), that tooling (including phetmarks) wouldn't be compatible.

[samreid]

Things get a little simpler if we assume that a debug version always gets built, and all supported brands are always built. Then we don't need to include those attributes in the version (folder) name.

[ariel-phet]

Desired dev directory structure and naming schemes as of 11/30/2017:

faradays-law/

  // brand=phet
  1.0.0-dev.1-phet/
    faradays-law_en-phet.html
    faradays-law_all-phet-debug.html
    faradays-law_all-phet.html
  1.0.0-rc.1-phet/
    faradays-law_en-phet.html
    faradays-law_all-phet-debug.html
    faradays-law_all-phet.html
  1.0.0-phet/
    faradays-law_en-phet.html
    faradays-law_all-phet-debug.html
    faradays-law_all-phet.html

  // brand=phetio
  1.0.0-dev.1-phetio/
    faradays-law_all-phetio.html
    faradays-law_all-phetio-debug.html
  1.0.0-rc.1-phetio/
    faradays-law_all-phetio.html
    faradays-law_all-phetio-debug.html
  1.0.0-phetio/
    faradays-law_all-phetio.html
    faradays-law_all-phetio-debug.html

  // one off = sonification, brand=phetio
  1.1.0-dev.1-phetio-sonification/
    faradays-law_all-phetio-sonification.html
    faradays-law_all-phetio-sonification-debug.html
  1.1.0-dev.2-phetio-sonification/
    faradays-law_all-phetio-sonification.html
    faradays-law_all-phetio-sonification-debug.html
  1.1.0-phetio-sonification/
    faradays-law_all-phetio-sonification.html
    faradays-law_all-phetio-sonification-debug.html

  // one off, brand=phet
  1.1.0-dev.1-phet-foobar/
    faradays-law_en-phet-foobar.html
    faradays-law_all-phet-foobar-debug.html
  1.1.0-dev.2-phet-foobar/
    faradays-law_en-phet-foobar.html
    faradays-law_all-phet-foobar-debug.html
  1.1.0-phetio-foobar/
    faradays-law_all-phetio-foobar.html
    faradays-law_all-phetio-foobar-debug.html

[ariel-phet]

Consensus (for the moment) at 11/30/2017 dev meeting, plan on using _all.html for all debug versions.

using -debug was consensus and also not adding anything else to this naming scheme

[jonathanolson]

faradays-law_en-phetio.html

I thought from another issue that for phet-io we would be using a version with all locales? (Could omit the _en, or if we are enforcing consistency, faradays-law_all-phetio.html would be the main phet-io built sim)

[zepumph]

I agree with you. I am about 80% sure that we should be building *_all-phetio.html always, and never *_en-phetio.html. https://github.com/phetsims/chipper/issues/632#issuecomment-348325508 is meant to solidify that. I will update the above "Master Version Schema" above with those results.

[zepumph]

As per @ariel-phet's request. I updated the above master schema with https://github.com/phetsims/chipper/issues/632#issuecomment-348329508. If things change again I will comment before updating it.

[jonathanolson]

Sounds like we're really sticking to a template if we're including redundant information (e.g. "all" and "debug", "all" and "phet-io", etc.?)

[pixelzoom]

12/4/17 dev meeting notes.

In local working copy and build server, support existence of artifacts for multiple brand in build/ directory. Directory structure (e.g.):

faradays-law/
  build/
    phet/
      faradays-law_en-phet.html
      ...
    phetio/
      faradays-law_all-phetio.html
      ...

[pixelzoom]

12/4/17 dev meeting notes continued:

local

build/ phet/ faradays-law_en-phet.html faradays-law_all-phet-debug.html faradays-law_all-phet.html phetio/ faradays-law_all-phetio.html faradays-law_all-phetio-debug.html

spot (current chipper:2.0)

1.0.0-dev.1-phet/ faradays-law_en-phet.html faradays-law_all-phet-debug.html faradays-law_all-phet.html 1.0.0-dev.1-phetio/ faradays-law_all-phetio.html faradays-law_all-phetio-debug.html

spot (proposed chipper:2.0)

1.0.0-dev.1/ phet/ faradays-law_en-phet.html faradays-law_all-phet-debug.html faradays-law_all-phet.html phetio/ faradays-law_all-phetio.html faradays-law_all-phetio-debug.html

PROS: • structure matches build/ • easier to transfer to spot • SR: pulls brand out of top-level directory name

CONS: • more difficult to “find latest” of a specific brand

CONSENSUS: go with “proposed” above for spot

[pixelzoom]

12/4/17 revisions to https://github.com/phetsims/chipper/issues/560#issuecomment-348304508 based on decision in https://github.com/phetsims/chipper/issues/560#issuecomment-349051862:

dev directory structure (assuming brands=phet,phetio)

faradays-law/

  1.0.0-dev.1/
    phet/
      faradays-law_en-phet.html
      faradays-law_all-phet.html
      faradays-law_all-phet-debug.html
    phetio/
      faradays-law_all-phetio.html
      faradays-law_all-phetio-debug.html

  1.0.0-rc.1/
    phet/
      faradays-law_en-phet.html
      faradays-law_all-phet.html
      faradays-law_all-phet-debug.html
    phetio/
      faradays-law_all-phetio.html
      faradays-law_all-phetio-debug.html

  1.0.0/
    phet/
      faradays-law_en-phet.html
      faradays-law_all-phet.html
      faradays-law_all-phet-debug.html
    phetio/
      faradays-law_all-phetio.html
      faradays-law_all-phetio-debug.html

  // one-off is in the directory name
  1.0.1-dev.1-sonification/
    phet/
      faradays-law_en-phet-sonification.html
      faradays-law_all-phet-sonification.html
      faradays-law_all-phet-sonification-debug.html
    phetio/
      faradays-law_all-phetio-sonification.html
      faradays-law_all-phetio-sonification-debug.html

SR suggested that we should have unit test to verify that the above directory structure is indeed created by build tools.

One-offs will come off a named branch (including master), working copy will be clean, developer must provide one-off name (will not use the branch name). First person to do a one-off will help refine.

[jonathanolson]

Example of dev deployment (chipper 2.0 branch with perennial grunt dev --repo=chains --brands=phet,phet-io) with both brands (thanks @zepumph for instrumenting chains!!): https://www.colorado.edu/physics/phet/dev/html/chains/1.7.0-dev.9/

There's a lot of path-specific code in wrappers, and it I wasn't able to quickly (10mins) figure out how to get the wrappers to point to the correct sim (without breaking require.js), so I'll leave this for a phet-io person (assigning @zepumph initially).

Additionally, I'll need to coordinate with @mattpen for the build-server work to be compatible with these changes, assigning.

[jonathanolson]

It looks kind of funny with the directory names (below the version directory) being phet, phetio and adaptedFromPhet when it would be cleaner to use the raw brand name (phet, phet-io, adapted-from-phet).

Does anyone else feel the same way, or is there a reason why we chose to shorten to "phetio"?

[zepumph]

I think in general, phet-io has pushed for hyphens as much as the code will allow (sometimes you need to squash for var names). I think I'm in favor of seeing the directory "phet-io"

[pixelzoom]

@jonathanolson said:

... when it would be cleaner to use the raw brand name (phet, phet-io, adapted-from-phet).

What is a "raw brand name"? I thought we decided that the brand names were changing to camelcase, and the hyphens and punctuation were hereafter verboten in brand names.

[jonathanolson]

What is a "raw brand name"? I thought we decided that the brand names were changing to camelcase, and the hyphens and punctuation were hereafter verboten in brand names.

I was not aware we were changing the brand names, but instead mapping them differently for the filenames, like:

Change brand "phet-io" to "phetio". Typically use camelcase for other brands, e.g. "adaptedFromPhET" or "widgetsInc".

Instead, it sounds like a more major change, where we need to refactor a lot of common/sim code, change query parameters (brand=phetio), ensure backwards compatibility in build-server/yotta (but presumably don't treat phetio and phet-io as two different brands). I assume we still live with phet-io.colorado.edu (sounds like too much effort to change), but do we change repository names that start with phet-io?

I don't recall discussing any of those trade-offs, or scope of any renaming that was past "how the filenames and directory names are", so I'd like to get confirmation from other developers (or @ariel-phet ) if this is actually the intended goal (as it will deserve its own issues for handling a change that large).

[jonathanolson]

If it wasn't clear, my opinion is that usage with the hyphen (phet-io) is pervasive in code and names, and the costs of trying to change this is nowhere near the benefit. Instead, the remapping for filenames (but not directories, as there is no ambiguities there) would be best.

[samreid]

Instead, the remapping for filenames (but not directories, as there is no ambiguities there) would be best.

Can you please clarify by specifying the proposed filenames and directory names? Or specify how they differ from https://github.com/phetsims/chipper/issues/560#issuecomment-349054913

[pixelzoom]

My understanding what that the brand names were going to change as follows:

adapted-from-phet -> adaptedFromPhET phet -> phet (no change) phet-io -> phetio

... and that this involved changing the subdirectory names in the brand repository, as well as the changes @jonathanolson mentioned in https://github.com/phetsims/chipper/issues/560#issuecomment-349545979. I really think it's a bad idea to have (for example) "--brand=phet-io" that results in "phetio" appearing in directory and file names.

[samreid]

phet-io -> phet-io

Did you mean phetio?