Where should instrumented colors live in the Studio tree?

[pixelzoom]

We have an inconsistency with where colors (instrumented ProfileColorProperty elements) appear in the Studio tree.

cck-dc, cck-dc-virtual-lab, density, and calculus-grapher have them under global.view.colorProfile, because they have this duplicated in their *Colors.ts file:

const tandem = Tandem.GLOBAL_VIEW.createTandem( 'colorProfile' );

GAO and solar-system-common have them under global.view.colors because they are using Tandem.COLORS. I suspect that the solar-system-common occurrences are not acutally used in any PhET-iO sims. Tandem.COLORS was added by @samreid in https://github.com/phetsims/tandem/commit/f93fc8e1db9c3757dd34372c0b1ae156bf96c17b.

Questions:

• Which is correct?
• Do we want to standardize?
• Which sim APIs do we want to break?
• Who will make the changes, create the migration rules, etc?

This blocks Acid-Base Solutions 1.3, where we need to instrument pH colors. It should presumably block publication of any other PhET-iO sims that have instrumented colors.

[pixelzoom]

The elements being instrumented are instances of ProfileColorProperty, so "global.view.colorProfile" would be OK. But I have a slight preference for "global.view.colors", since they are (by convention) grouped in a file named *Colors.ts.

[zepumph]

Tandem.COLORS is the way of the future. Let's update the others and add migration rules eagerly so that there isn't a dangling wrong pattern in our code to potentially proliferate.

[pixelzoom]

In https://github.com/phetsims/acid-base-solutions/issues/215, I used Tandem.COLORS, so colors live under acidBaseSolutions.global.view.colors.

[arouinfar]

I prefer global.view.colors, glad to hear it's the way of the future. The colorProfile terminology is a little less clear to me.

[pixelzoom]

I'll convert sims to Tandem.COLORS.

[pixelzoom]

I started with calculus-grapher and ran into problem with joist changes in https://github.com/phetsims/joist/issues/934 that have broken migration. So I'm going to defer until that issue is addressed.

[samreid]

I put a patch in https://github.com/phetsims/joist/issues/934#issuecomment-1688146369 that seems to help calculus grapher past the joist issues.

[pixelzoom]

Thanks for the patch @samreid. But I suspect that I'll hit the same problem with cck-* and density. I don't want to spend the time to discover that, or (worse) leave this issue in a half-finished state that breaks migration for sims. So this issue will continue to be deferred until https://github.com/phetsims/joist/issues/934 is closed.

[pixelzoom]

I'm bailing on this for 2 reasons: (1) it does not block progress on Acid-Base Solutions, which is using Tandem.COLORS, and (2) there doesn't seem to be a consensus on when migration rules should be addressed, or the severity of breaking migration in master. So self unassigning, and we can revisit when other regressions in main have been addressed.

[zepumph]

there doesn't seem to be a consensus on when migration rules should be addressed

We have come together around eagerly solving migration problems as they arise. CT is now reporting these problems for many repos and we should be able to continue on this work now. @pixelzoom is this something you would like to pick up now that CT will back you up on migration rules, and we have a process of pulling out common migration rules over in https://github.com/phetsims/phet-io-wrappers/issues/467, like joistRules.uninstrumentAboutDialogDetails(). @samreid and I are happy to meet to discuss more if you'd like.

[pixelzoom]

From 9/7/23 phet-io meeting.... I'll resume work on this.

[zepumph]

• [x] From @arouinfar, double check that this doesn't show up in any examples.

[pixelzoom]

Changing calculus-grapher went smoothly, see the above commits.

[pixelzoom]

Changing density did not go smoothly. I discussed this with @samreid. He said this is a case that's not currently supported by migration, and I should assign back to him and @zepumph.

Here's the background... With this patch:

patch

```diff Subject: [PATCH] use simName in other migration rules, https://github.com/phetsims/tandem/issues/303 --- Index: phet-io-sim-specific/repos/density/density-phet-io-api.json IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-sim-specific/repos/density/density-phet-io-api.json b/phet-io-sim-specific/repos/density/density-phet-io-api.json --- a/phet-io-sim-specific/repos/density/density-phet-io-api.json (revision f8e4dbcafae5b99410442ba65f8a71bd3f17fe9c) +++ b/phet-io-sim-specific/repos/density/density-phet-io-api.json (date 1694118658451) @@ -3989,7 +3989,7 @@ "hidden": false, "identifier": "WATER", "liquidColor": { - "phetioID": "density.global.view.colorProfile.materialWaterColorProperty" + "phetioID": "density.global.view.colors.materialWaterColorProperty" }, "name": { "phetioID": "density.general.model.strings.densityBuoyancyCommon.material.waterStringProperty" @@ -10444,7 +10444,7 @@ } }, "view": { - "colorProfile": { + "colors": { "compareBlueColorProperty": { "_data": { "initialState": { @@ -11687,7 +11687,7 @@ "hidden": false, "identifier": "WATER", "liquidColor": { - "phetioID": "density.global.view.colorProfile.materialWaterColorProperty" + "phetioID": "density.global.view.colors.materialWaterColorProperty" }, "name": { "phetioID": "density.general.model.strings.densityBuoyancyCommon.material.waterStringProperty" @@ -15137,7 +15137,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryBrownColorProperty" + "phetioID": "density.global.view.colors.mysteryBrownColorProperty" }, "density": 8960, "hidden": false, @@ -15532,7 +15532,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryMustardColorProperty" + "phetioID": "density.global.view.colors.mysteryMustardColorProperty" }, "density": 919, "hidden": false, @@ -15927,7 +15927,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryPeachColorProperty" + "phetioID": "density.global.view.colors.mysteryPeachColorProperty" }, "density": 832, "hidden": false, @@ -16322,7 +16322,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.compareRedColorProperty" + "phetioID": "density.global.view.colors.compareRedColorProperty" }, "density": 3510, "hidden": false, @@ -16717,7 +16717,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryMaroonColorProperty" + "phetioID": "density.global.view.colors.mysteryMaroonColorProperty" }, "density": 950, "hidden": false, @@ -17114,7 +17114,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.comparePurpleColorProperty" + "phetioID": "density.global.view.colors.comparePurpleColorProperty" }, "density": 3510, "hidden": false, @@ -17509,7 +17509,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.compareBlueColorProperty" + "phetioID": "density.global.view.colors.compareBlueColorProperty" }, "density": 400, "hidden": false, @@ -17904,7 +17904,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.compareYellowColorProperty" + "phetioID": "density.global.view.colors.compareYellowColorProperty" }, "density": 19320, "hidden": false, @@ -18299,7 +18299,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.compareRedColorProperty" + "phetioID": "density.global.view.colors.compareRedColorProperty" }, "density": 1000, "hidden": false, @@ -18694,7 +18694,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.compareGreenColorProperty" + "phetioID": "density.global.view.colors.compareGreenColorProperty" }, "density": 400, "hidden": false, @@ -19091,7 +19091,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryOrangeColorProperty" + "phetioID": "density.global.view.colors.mysteryOrangeColorProperty" }, "density": 11340, "hidden": false, @@ -19486,7 +19486,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryBrownColorProperty" + "phetioID": "density.global.view.colors.mysteryBrownColorProperty" }, "density": 2700, "hidden": false, @@ -19881,7 +19881,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryLightGreenColorProperty" + "phetioID": "density.global.view.colors.mysteryLightGreenColorProperty" }, "density": 2700, "hidden": false, @@ -20276,7 +20276,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryPinkColorProperty" + "phetioID": "density.global.view.colors.mysteryPinkColorProperty" }, "density": 4500, "hidden": false, @@ -20671,7 +20671,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryLightPurpleColorProperty" + "phetioID": "density.global.view.colors.mysteryLightPurpleColorProperty" }, "density": 8960, "hidden": false, @@ -21068,7 +21068,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryMaroonColorProperty" + "phetioID": "density.global.view.colors.mysteryMaroonColorProperty" }, "density": 950, "hidden": false, @@ -21463,7 +21463,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryGrayColorProperty" + "phetioID": "density.global.view.colors.mysteryGrayColorProperty" }, "density": 1000, "hidden": false, @@ -21858,7 +21858,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryPeachColorProperty" + "phetioID": "density.global.view.colors.mysteryPeachColorProperty" }, "density": 7800, "hidden": false, @@ -22253,7 +22253,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryMustardColorProperty" + "phetioID": "density.global.view.colors.mysteryMustardColorProperty" }, "density": 400, "hidden": false, @@ -22648,7 +22648,7 @@ "value": { "custom": true, "customColor": { - "phetioID": "density.global.view.colorProfile.mysteryWhiteColorProperty" + "phetioID": "density.global.view.colors.mysteryWhiteColorProperty" }, "density": 950, "hidden": false, @@ -22841,7 +22841,7 @@ "hidden": false, "identifier": "WATER", "liquidColor": { - "phetioID": "density.global.view.colorProfile.materialWaterColorProperty" + "phetioID": "density.global.view.colors.materialWaterColorProperty" }, "name": { "phetioID": "density.general.model.strings.densityBuoyancyCommon.material.waterStringProperty" Index: phet-io-sim-specific/repos/density/js/density-migration-rules.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-sim-specific/repos/density/js/density-migration-rules.ts b/phet-io-sim-specific/repos/density/js/density-migration-rules.ts --- a/phet-io-sim-specific/repos/density/js/density-migration-rules.ts (revision f8e4dbcafae5b99410442ba65f8a71bd3f17fe9c) +++ b/phet-io-sim-specific/repos/density/js/density-migration-rules.ts (date 1694118639944) @@ -5,12 +5,19 @@ import { MigrationRules } from '../../../../phet-io-wrappers/common/js/migration/MigrationEngine.js'; import { titleAndScreenButtonTextUninstrumented } from '../../../js/migration-common/joistRules.js'; +import TandemFragmentRenamed from '../../../../phet-io-wrappers/common/js/migration/TandemFragmentRenamed.js'; window.phetio = window.phetio || {}; +// Use ${simName} in rules that are general and might be usable for other sims +const simName = 'density'; + const migrationRules: MigrationRules = { '1.2.0': [ - ...titleAndScreenButtonTextUninstrumented( 'density', [ 'introScreen', 'compareScreen', 'mysteryScreen' ] ) + ...titleAndScreenButtonTextUninstrumented( 'density', [ 'introScreen', 'compareScreen', 'mysteryScreen' ] ), + + // The parent phetioID for colors was standardized. See https://github.com/phetsims/tandem/issues/303 + new TandemFragmentRenamed( `${simName}.global.view.colorProfile`, `${simName}.global.view.colors` ) ] }; Index: density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts b/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts --- a/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts (revision 5aea561852da0ae62530ec9bd94800357fe1d1b1) +++ b/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts (date 1694119503954) @@ -11,7 +11,7 @@ import Tandem from '../../../../tandem/js/Tandem.js'; import densityBuoyancyCommon from '../../densityBuoyancyCommon.js'; -const tandem = Tandem.GLOBAL_VIEW.createTandem( 'colorProfile' ); +const tandem = Tandem.COLORS; // Initial colors for each profile, by string key. Only profile currently is default (still helpful for making color // tweaks with the top-level files) ```

The Migration wrapper reports several errors of this form, where customColor.phetioID, and possibly liquidColor.phetioID, are still using the old "colorProfile" name:

  "density.mysteryScreen.model.blockSets.set1.block1B.materialProperty": {
    "value": {
      "name": {
        "phetioID": "density.general.model.strings.densityBuoyancyCommon.material.customStringProperty"
      },
      "identifier": null,
      "tandemName": "custom",
      "density": 400,
      "viscosity": 0.001,
      "custom": true,
      "hidden": false,
      "staticCustomColor": null,
      "customColor": {
        "phetioID": "density.global.view.colorProfile.compareBlueColorProperty"
      },
      "staticLiquidColor": null,
      "liquidColor": null
    },
    "validValues": null,
    "units": null
  },

The relevant lines in Material.ts are:

const NullableColorPropertyReferenceType = NullableIO( ReferenceIO( Property.PropertyIO( Color.ColorIO ) ) );
...
  public static readonly MaterialIO = new IOType<Material, MaterialState>( 'MaterialIO', {
...
    stateSchema: {
...
      customColor: NullableColorPropertyReferenceType,
...
      liquidColor: NullableColorPropertyReferenceType
    },

[pixelzoom]

cck-dc and cck-dc-virtual-lab went smoothly, see above commits.

density will need to be completed by @samreid and @zepumph, so I'll unassign myself.

@kathy-phet Reporting back on how this work went... This should have been a straightforward rename ("global.view.colorProfile" => "global.view.colors"), and I anticipated that it would take ~30 minutes. Instead it required a solid 1 hour. I needed to involve another developer. I was only able to accomplish 3/4 of the work. And we identifed a case that is not supported by migration.

[kathy-phet]

@pixelzoom - Thanks for taking it for a test drive and reporting back on where further work is to streamline these kinds of things.

[zepumph]

@samreid and I worked on this this afternoon. We knew we'd get here eventually because we were never renaming any state values that were associated with a phetioID (like ReferenceIO). Here is a patch to get us started. Our favorite part about it is an assertion after migrating a rename that checks to see if the "old" phetioID is anywhere in the state. This double check will help us make sure we cover all cases. For now we will focus on the ReferenceIO state case and see if others present themselves.

```diff Subject: [PATCH] temp --- Index: phet-io-wrappers/common/js/migration/MigrationEngine.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-wrappers/common/js/migration/MigrationEngine.ts b/phet-io-wrappers/common/js/migration/MigrationEngine.ts --- a/phet-io-wrappers/common/js/migration/MigrationEngine.ts (revision c13f023e02d44d26b65a206f378189937998c3a9) +++ b/phet-io-wrappers/common/js/migration/MigrationEngine.ts (date 1694127055258) @@ -87,7 +87,7 @@ const messageBatches: string[][] = []; this.rules.forEach( migrationRule => { - const result = migrationRule.processCommand( command, this.oldAPIFlat, this.newAPIFlat ); + const result = migrationRule.processCommand( command, this.oldAPIFlat, this.newAPIFlat, this ); command = result.command; messageBatches.push( result.messages ); } ); Index: phet-io-sim-specific/repos/density/js/density-migration-rules.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-sim-specific/repos/density/js/density-migration-rules.ts b/phet-io-sim-specific/repos/density/js/density-migration-rules.ts --- a/phet-io-sim-specific/repos/density/js/density-migration-rules.ts (revision 9fd7f0b32f6d516ffaa6d0f912e8eb77da9a78d1) +++ b/phet-io-sim-specific/repos/density/js/density-migration-rules.ts (date 1694125421399) @@ -5,12 +5,17 @@ import { MigrationRules } from '../../../../phet-io-wrappers/common/js/migration/MigrationEngine.js'; import { titleAndScreenButtonTextUninstrumented } from '../../../js/migration-common/joistRules.js'; +import TandemFragmentRenamed from '../../../../phet-io-wrappers/common/js/migration/TandemFragmentRenamed.js'; window.phetio = window.phetio || {}; const migrationRules: MigrationRules = { '1.2.0': [ - ...titleAndScreenButtonTextUninstrumented( 'density', [ 'introScreen', 'compareScreen', 'mysteryScreen' ] ) + ...titleAndScreenButtonTextUninstrumented( 'density', [ 'introScreen', 'compareScreen', 'mysteryScreen' ] ), + + // The parent phetioID for colors was standardized. See https://github.com/phetsims/tandem/issues/303 + new TandemFragmentRenamed( 'density.global.view.colorProfile', + 'density.global.view.colors' ) ] }; Index: phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts b/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts --- a/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts (revision c13f023e02d44d26b65a206f378189937998c3a9) +++ b/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts (date 1694130050119) @@ -11,7 +11,7 @@ * @author Sam Reid (PhET Interactive Simulations) */ -import { APIFlat, PhetioID, PhetioState } from '../../../../tandem/js/TandemConstants.js'; +import { APIFlat, PhetioAPI, PhetioID, PhetioState } from '../../../../tandem/js/TandemConstants.js'; import Command from './Command.js'; import MigrationRule, { MigrationResult, MigrationRuleOptions } from './MigrationRule.js'; import MigrationEngine from './MigrationEngine.js'; @@ -58,7 +58,7 @@ } } - public processCommand( command: Command, oldFlatAPI: APIFlat ): MigrationResult { + public processCommand( command: Command, oldFlatAPI: APIFlat, newFlatAPI: APIFlat, migrationEngine: MigrationEngine ): MigrationResult { const messages: string[] = []; if ( this.matchesTandem( command.phetioID ) ) { @@ -101,6 +101,30 @@ // Migrate the old value into the new state, always do this because we fill in newState from a clean, empty object. newState[ newPhetioID ] = oldState[ phetioID ]; } + else if (/*referenceIO*/ ) { + // TODO: do we have to assert that each phetioID we encounter is in the oldAPI, otherwise we can't traverse to find referenceIOs. + // handle other cases + /* + initialColor: NUllable(Reference(phetioObject)) + initialColor: null + + ['initialColor', 'fkd'] + newState[phetioID]( values) = + + + + >phetioTypeName.includes( ReferenceIO ) + + 1. look up the type of current phetioID (old phetioID) + + 2. assert all state schema keys are in there. + 3. For each state schema key, see if the value is a string + question: "density.global.view.colorProfile", "hello my name is density.global.view.colorProfile" + 4. If so, and it matches, rename it + 5. If not, recurse into the IOType that is the value of that key to see if any of its stateSchema keys can handle it. + */ + this.doesStateValueHaveReferenceIOInsideSomewhereEvenDeep( oldPhetioID, newState, oldFlatAPI, migrationEngine.oldAPI ); + } else { // Don't let something that has been migrated by this rule get overwritten by the value it holds in the "old" state. @@ -112,11 +136,57 @@ } } + if ( assert && typeof this.oldFragment === 'string' ) { + const string = JSON.stringify( newState ); + + if ( string.includes( this.oldFragment ) ) { + console.log( newState ); + } + assert && assert( !string.includes( this.oldFragment ), `extra usage of ${this.oldFragment}` ) + } + command = command.withArgs( [ newState ] ); } return { command: command, messages: messages }; } + doesStateValueHaveReferenceIOInsideSomewhereEvenDeep( phetioID: PhetioID, state: PhetioState, oldAPIFlat: APIFlat, oldAPI: PhetioAPI ): boolean { + if ( !oldAPIFlat.hasOwnProperty( phetioID ) ) { + return false; + } + + const typeName = oldAPIFlat[ phetioID ]._metadata.phetioTypeName; + + const typeAPI = oldAPI.phetioTypes[ typeName ]; + if ( !typeAPI || !typeAPI.stateSchema ) { + + // Nothing to look at here + return false; + } + // If stateSchema is an object + if ( typeAPI.stateSchema && typeAPI.stateSchema.constructor === Object ) { + return _.some( Object.keys( typeAPI.stateSchema ), stateKey => { + + // TODO: ugggggghghgh + return this.doesStateValueHaveReferenceIOInsideSomewhereEvenDeep( phetioID, state, oldAPIFlat, oldAPI ); + } ) + } + + // Now it is a string + if ( typeof typeAPI.stateSchema === 'string' ) { + return false; // I dont' Know?!!?!?! + + } + + return true + + + return true; + + + } + + /** * For determining if a state value is the same as the initialState from the API. */ Index: phet-io-wrappers/common/js/migration/MigrationRule.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-wrappers/common/js/migration/MigrationRule.ts b/phet-io-wrappers/common/js/migration/MigrationRule.ts --- a/phet-io-wrappers/common/js/migration/MigrationRule.ts (revision c13f023e02d44d26b65a206f378189937998c3a9) +++ b/phet-io-wrappers/common/js/migration/MigrationRule.ts (date 1694127055304) @@ -12,6 +12,7 @@ import { APIFlat } from '../../../../tandem/js/TandemConstants.js'; import Command from './Command.js'; import optionize from '../../../../phet-core/js/optionize.js'; +import type MigrationEngine from './MigrationEngine.js'; export type MigrationResult = { command: Command; @@ -38,7 +39,7 @@ this.canDoNothing = options.canDoNothing; } - public abstract processCommand( command: Command, oldFlatAPI: APIFlat, newFlatAPI: APIFlat ): MigrationResult; + public abstract processCommand( command: Command, oldFlatAPI: APIFlat, newFlatAPI: APIFlat, migrationEngine: MigrationEngine ): MigrationResult; public abstract toString(): string; Index: density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts b/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts --- a/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts (revision 5aea561852da0ae62530ec9bd94800357fe1d1b1) +++ b/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts (date 1694125421354) @@ -11,7 +11,7 @@ import Tandem from '../../../../tandem/js/Tandem.js'; import densityBuoyancyCommon from '../../densityBuoyancyCommon.js'; -const tandem = Tandem.GLOBAL_VIEW.createTandem( 'colorProfile' ); +const tandem = Tandem.COLORS; // Initial colors for each profile, by string key. Only profile currently is default (still helpful for making color // tweaks with the top-level files)

[samreid]

I worked on the recursive API visiting part and got this far. Here is a patch which includes the above patch (parts commented out). The main idea is that when putting a value into the top-level state, we call processStateValue like so:

          // Migrate the old value into the new state, always do this because we fill in newState from a clean, empty object.
          newState[ newPhetioID ] = this.processStateValue( phetioID, oldState, migrationEngine );

Then there are a pair of functions getStateSchema and processStateValue to recurse over a state. This is untested and half-baked but I wanted to share my progress before I have to step away.

```diff Subject: [PATCH] Add unselectedChildrenSceneGraphStrategy option to ToggleNode, see https://github.com/phetsims/center-and-variability/issues/527 --- Index: phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts b/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts --- a/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts (revision 709d5c7826da7b7eb39ee0edd31c49f7485be59f) +++ b/phet-io-wrappers/common/js/migration/TandemFragmentRenamed.ts (date 1694437861533) @@ -15,6 +15,7 @@ import Command from './Command.js'; import MigrationRule, { MigrationResult, MigrationRuleOptions } from './MigrationRule.js'; import MigrationEngine from './MigrationEngine.js'; +import migrationRule from './MigrationRule.js'; export type TandemFragmentRenamedOptions = MigrationRuleOptions; @@ -58,7 +59,7 @@ } } - public processCommand( command: Command, oldFlatAPI: APIFlat ): MigrationResult { + public processCommand( command: Command, oldFlatAPI: APIFlat, newFlatAPI: APIFlat, migrationEngine: MigrationEngine ): MigrationResult { const messages: string[] = []; if ( this.matchesTandem( command.phetioID ) ) { @@ -99,24 +100,129 @@ const phetioID = overwritingCustomizedValue ? newPhetioID : oldPhetioID; // Migrate the old value into the new state, always do this because we fill in newState from a clean, empty object. - newState[ newPhetioID ] = oldState[ phetioID ]; + newState[ newPhetioID ] = this.processStateValue( phetioID, oldState, migrationEngine ); } + // else if (/*referenceIO*/ ) { + // // TODO: do we have to assert that each phetioID we encounter is in the oldAPI, otherwise we can't traverse to find referenceIOs. + // // handle other cases + // /* + // initialColor: NUllable(Reference(phetioObject)) + // initialColor: null + // + // ['initialColor', 'fkd'] + // newState[phetioID]( values) = + // + // + // + // >phetioTypeName.includes( ReferenceIO ) + // + // 1. look up the type of current phetioID (old phetioID) + // + // 2. assert all state schema keys are in there. + // 3. For each state schema key, see if the value is a string + // question: "density.global.view.colorProfile", "hello my name is density.global.view.colorProfile" + // 4. If so, and it matches, rename it + // 5. If not, recurse into the IOType that is the value of that key to see if any of its stateSchema keys can handle it. + // */ + // this.doesStateValueHaveReferenceIOInsideSomewhereEvenDeep( oldPhetioID, newState, oldFlatAPI, migrationEngine.oldAPI ); + // } else { // Don't let something that has been migrated by this rule get overwritten by the value it holds in the "old" state. // This would happen if the "promoted"/newPhetioID was listed later in Object.keys(), such that Object.keys() // ordered it after the promoted logic. if ( !promotedMigratedIDs.includes( oldPhetioID ) ) { - newState[ oldPhetioID ] = oldState[ oldPhetioID ]; + newState[ oldPhetioID ] = this.processStateValue( oldPhetioID, oldState, migrationEngine ); } } } + + // if ( assert && typeof this.oldFragment === 'string' ) { + // const string = JSON.stringify( newState ); + // + // if ( string.includes( this.oldFragment ) ) { + // console.log( newState ); + // } + // assert && assert( !string.includes( this.oldFragment ), `extra usage of ${this.oldFragment}` ) + // } command = command.withArgs( [ newState ] ); } return { command: command, messages: messages }; } + public getStateSchema( phetioID: PhetioID, migrationEngine: MigrationEngine ): string | Record | null { + // look up the state schema: + // migrationEngine.oldAPI. + + if ( !migrationEngine.oldAPIFlat.hasOwnProperty( phetioID ) ) { + return null; + } + + const typeName = migrationEngine.oldAPIFlat[ phetioID ]._metadata.phetioTypeName; + const typeAPI = migrationEngine.oldAPI.phetioTypes[ typeName ]; + + if ( !typeAPI || !typeAPI.stateSchema ) { + + // Nothing to look at here + return null; + } + + else { + return typeAPI.stateSchema; + } + } + + public processStateValue( phetioID: PhetioID, state: PhetioState, migrationEngine: MigrationEngine ): object { + const stateSchema = this.getStateSchema( phetioID, migrationEngine ); + if ( typeof stateSchema === 'string' ) { + const choices = stateSchema.split( '|' ); + } + else if ( stateSchema === null ) { + return state[ phetioID ]; + } + else { + const newState: PhetioState = {}; + const stateSchemaKeys = Object.keys( stateSchema ); + for ( let i = 0; i < stateSchemaKeys.length; i++ ) { + const stateSchemaKey = stateSchemaKeys[ i ]; + newState[ stateSchemaKey ] = this.processStateValue( stateSchemaKey, state, migrationEngine ); + } + return newState; + } + } + + // doesStateValueHaveReferenceIOInsideSomewhereEvenDeep( phetioID: PhetioID, state: PhetioState, oldAPIFlat: APIFlat, oldAPI: PhetioAPI ): boolean { + // if ( !oldAPIFlat.hasOwnProperty( phetioID ) ) { + // return false; + // } + // + // const typeName = oldAPIFlat[ phetioID ]._metadata.phetioTypeName; + // + // const typeAPI = oldAPI.phetioTypes[ typeName ]; + // if ( !typeAPI || !typeAPI.stateSchema ) { + // + // // Nothing to look at here + // return false; + // } + // // If stateSchema is an object + // if ( typeAPI.stateSchema && typeAPI.stateSchema.constructor === Object ) { + // return _.some( Object.keys( typeAPI.stateSchema ), stateKey => { + // + // // TODO: ugggggghghgh + // return this.doesStateValueHaveReferenceIOInsideSomewhereEvenDeep( phetioID, state, oldAPIFlat, oldAPI ); + // } ) + // } + // + // // Now it is a string + // if ( typeof typeAPI.stateSchema === 'string' ) { + // return false; // I dont' Know?!!?!?! + // + // } + // + // return true; + // } + /** * For determining if a state value is the same as the initialState from the API. */ Index: phet-io-wrappers/common/js/migration/MigrationRule.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-wrappers/common/js/migration/MigrationRule.ts b/phet-io-wrappers/common/js/migration/MigrationRule.ts --- a/phet-io-wrappers/common/js/migration/MigrationRule.ts (revision 709d5c7826da7b7eb39ee0edd31c49f7485be59f) +++ b/phet-io-wrappers/common/js/migration/MigrationRule.ts (date 1694346623055) @@ -12,6 +12,7 @@ import { APIFlat } from '../../../../tandem/js/TandemConstants.js'; import Command from './Command.js'; import optionize from '../../../../phet-core/js/optionize.js'; +import type MigrationEngine from './MigrationEngine.js'; export type MigrationResult = { command: Command; @@ -38,7 +39,7 @@ this.canDoNothing = options.canDoNothing; } - public abstract processCommand( command: Command, oldFlatAPI: APIFlat, newFlatAPI: APIFlat ): MigrationResult; + public abstract processCommand( command: Command, oldFlatAPI: APIFlat, newFlatAPI: APIFlat, migrationEngine: MigrationEngine ): MigrationResult; public abstract toString(): string; Index: density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts b/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts --- a/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts (revision a0d96a61dafec4c474ad3c3c1bab311b2b82fe57) +++ b/density-buoyancy-common/js/common/view/DensityBuoyancyCommonColors.ts (date 1694346586530) @@ -11,7 +11,7 @@ import Tandem from '../../../../tandem/js/Tandem.js'; import densityBuoyancyCommon from '../../densityBuoyancyCommon.js'; -const tandem = Tandem.GLOBAL_VIEW.createTandem( 'colorProfile' ); +const tandem = Tandem.COLORS; // Initial colors for each profile, by string key. Only profile currently is default (still helpful for making color // tweaks with the top-level files) Index: phet-io-wrappers/common/js/migration/MigrationEngine.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-wrappers/common/js/migration/MigrationEngine.ts b/phet-io-wrappers/common/js/migration/MigrationEngine.ts --- a/phet-io-wrappers/common/js/migration/MigrationEngine.ts (revision 709d5c7826da7b7eb39ee0edd31c49f7485be59f) +++ b/phet-io-wrappers/common/js/migration/MigrationEngine.ts (date 1694437528719) @@ -57,8 +57,8 @@ const DATA_KEY_NAME = TandemConstants.DATA_KEY_NAME; export default class MigrationEngine { - private readonly oldAPIFlat: APIFlat; - private readonly newAPIFlat: APIFlat; + public readonly oldAPIFlat: APIFlat; + public readonly newAPIFlat: APIFlat; // Once migration is run on a command, the migrated command is run through each of these checkers, who have the ability // to assert out and console log if the behavior is not as it expects. @@ -86,7 +86,7 @@ const messageBatches: string[][] = []; this.rules.forEach( migrationRule => { - const result = migrationRule.processCommand( command, this.oldAPIFlat, this.newAPIFlat ); + const result = migrationRule.processCommand( command, this.oldAPIFlat, this.newAPIFlat, this ); command = result.command; messageBatches.push( result.messages ); } ); Index: phet-io-sim-specific/repos/density/js/density-migration-rules.ts IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/phet-io-sim-specific/repos/density/js/density-migration-rules.ts b/phet-io-sim-specific/repos/density/js/density-migration-rules.ts --- a/phet-io-sim-specific/repos/density/js/density-migration-rules.ts (revision 17345f300fc61a701527863e4e9cb931e4cb3675) +++ b/phet-io-sim-specific/repos/density/js/density-migration-rules.ts (date 1694347189871) @@ -5,6 +5,7 @@ import { MigrationRules } from '../../../../phet-io-wrappers/common/js/migration/MigrationEngine.js'; import { titleAndScreenButtonTextUninstrumented } from '../../../js/migration-common/joistRules.js'; +import TandemFragmentRenamed from '../../../../phet-io-wrappers/common/js/migration/TandemFragmentRenamed.js'; window.phetio = window.phetio || {}; @@ -12,7 +13,10 @@ const migrationRules: MigrationRules = { '1.2.0': [ - ...titleAndScreenButtonTextUninstrumented( simName, [ 'introScreen', 'compareScreen', 'mysteryScreen' ] ) + ...titleAndScreenButtonTextUninstrumented( simName, [ 'introScreen', 'compareScreen', 'mysteryScreen' ] ), + + // The parent phetioID for colors was standardized. See https://github.com/phetsims/tandem/issues/303 + new TandemFragmentRenamed( 'density.global.view.colorProfile', 'density.global.view.colors' ) ] }; ```

[zepumph]

@samreid and I got to a commit point today. There are still a bunch of TODOs, but we got Density's color rename committed and passing in the migration engine.

The most important next step is to use a bit less hard coded logic about checking phetioID when its a string. Perhaps by using the API files' stateSchema.

[samreid]

I moved the StringUnion problem to a new issue https://github.com/phetsims/tandem/issues/306. As far as I can tell this issue is close to closing. @zepumph what do you think?

[samreid]

@zepumph and I finished this up today, nice work team!