search

phetsims / natural-selection

"Natural Selection" is an educational simulation in HTML5, by PhET Interactive Simulations
GNU General Public License v3.0
3 stars 7 forks source link

phetioCompareAPI is failing. #301

Closed pixelzoom closed 3 years ago

pixelzoom commented 3 years ago

For phetsims/qa#662

In https://github.com/phetsims/natural-selection/issues/294, I followed all of the steps in https://github.com/phetsims/phet-io/blob/master/doc/phet-io-instrumentation-technical-guide.md#post-publication-steps for setting up phetioDesigned: true. Everything is commited and pushed.

When I run NS (master) in Studio with phetioCompareAPI, it's failing with:

Assertion failed: Designed API changes detected, please roll them back or revise the reference API: .... lots of json .... at window.assertions.assertFunction (assert.js:25) at XMLHttpRequest. (phetioEngine.js:312)

The json part of the console output is way too long to be examined manually.

I've tried re-running grunt generate-phet-io-api, and no changes are generated, so there's nothing for me to diff.

@samreid @zepumph suggestions on how to proceed?

pixelzoom commented 3 years ago

I'm also wondering why this error is not showing up in CT. I added phetioDesigned: true on 7/6/21 @ 6:32pm, and CT has run 6 test cycles since then. Is this test actually being run by CT?

zepumph commented 3 years ago

I'll take a look!

zepumph commented 3 years ago

I pulled all, and then ran:

http://localhost:8080/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&ea&debugger

Then I parsed the horrible json-ish output by replacing Expected: and , actual: with newlines, and I got this:


naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.22165343022406,"y":48.54714899884378,"z":12.684829323916581},"hopStartPosition":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234},"hopTime":0.4357117786067223,"restTime":2.9807053366829397},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.7018245396764424,"hopTime":0.37145791805325357,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-16.780560589326512,"y":32.560992241519045,"z":-9.734973426342497},"hopStartPosition":{"x":-59.77427630950419,"y":-62.26770743671871,"z":206.59843884492193}}}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234}}
{"value":{"x":-59.77427630950419,"y":-62.26770743671871,"z":206.59843884492193},"validValues":null,"units":null}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype.xDirectionProperty._data.initialState differs:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"}
{"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-16.17943842560941,"y":49.17372834079743,"z":-6.723558180699183},"hopStartPosition":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044},"hopTime":0.4105757432869891,"restTime":1.733928075687837},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.3926442195784587,"hopTime":0.3664566424200951,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-13.345672919838014,"y":32.62236026723778,"z":10.377831239320455},"hopStartPosition":{"x":-25.143955421198655,"y":-77.12669201353901,"z":184.30996197969148}}}

naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044}}
{"value":{"x":-25.143955421198655,"y":-77.12669201353901,"z":184.30996197969148},"validValues":null,"units":null}

naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs:
{"_private":{"speed":158.03332201218853}}
{"_private":{"speed":170.958872349204}}

naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-125.23828095416042,"y":-58.25886076740691,"z":212.61170884888963}}
{"value":{"x":-71.00571623876567,"y":-54.996686657376905,"z":217.50497001393464},"validValues":null,"units":null}

naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.xDirectionProperty._data.initialState differs:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"}
{"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.026184316645049,"y":41.21830940413096,"z":-10.247822414681844},"hopStartPosition":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693},"hopTime":0.40510814444208754,"restTime":1.926434494549918},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.806020437221202,"hopTime":0.43276774069209145,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":17.63867517890751,"y":49.090402481554236,"z":-8.40494483084945},"hopStartPosition":{"x":103.34636259832627,"y":-50.670095616209345,"z":223.994856575686}}}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693}}
{"value":{"x":103.34636259832627,"y":-50.670095616209345,"z":223.994856575686},"validValues":null,"units":null}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs:
{"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-14.756223648971094,"y":47.44263166490198,"z":7.335895397398229},"hopStartPosition":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763},"hopTime":0.4596950485642879,"restTime":1.9745883803442004},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null}
{"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.2176901511580194,"hopTime":0.32057176331792103,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":15.733891888178658,"y":39.70201648770933,"z":-2.475136734138804},"hopStartPosition":{"x":71.37289231164004,"y":-78.85318525008829,"z":181.72022212486758}}}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763}}
{"value":{"x":71.37289231164004,"y":-78.85318525008829,"z":181.72022212486758},"validValues":null,"units":null}

naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.xDirectionProperty._data.initialState differs:
{"units":null,"validValues":["LEFT","RIGHT"],"value":"LEFT"}
{"value":"RIGHT","validValues":["LEFT","RIGHT"],"units":null}

naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs:
{"_private":{"speed":161.4491569500465}}
{"_private":{"speed":189.15268002265276}}

naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs:
{"units":null,"validValues":null,"value":{"x":-260.3667461546996,"y":-13.013580371909939,"z":280.4796294421351}}
{"value":{"x":178.88171902020912,"y":-9.68208837704151,"z":285.47686743443774},"validValues":null,"units":null}

All of the above look to be values of PhET-iO element's initial state that are based on a seeded random, and so are different between runtimes. This seems like something to try to manage, but I'm not sure off the top of my head the best solution. I'll create a general issue.

pixelzoom commented 3 years ago

Any idea why this is not showing up as an error in CT?

zepumph commented 3 years ago

Hmm, not totally sure. Looks like it isn't run very often, but I guess it should have failed when it ran that one time:

image

zepumph commented 3 years ago

After the above formatting commits, this is what the assertion message now looks like off the bat:

``` Assertion failed: Designed API changes detected, please roll them back or revise the reference API: naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs. Expected: {"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.22165343022406,"y":48.54714899884378,"z":12.684829323916581},"hopStartPosition":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234},"hopTime":0.4357117786067223,"restTime":2.9807053366829397},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null} actual: {"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.7929579531085924,"hopTime":0.4044680057386113,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":19.846291550894136,"y":43.770404575613156,"z":-1.5172937110237914},"hopStartPosition":{"x":139.36410382280886,"y":-82.16378669842732,"z":176.75431995235903}}} naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs. Expected: {"units":null,"validValues":null,"value":{"x":60.532144260044134,"y":-78.9058606222451,"z":181.64120906663234}} actual: {"value":{"x":139.36410382280886,"y":-82.16378669842732,"z":176.75431995235903},"validValues":null,"units":null} naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs. Expected: {"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-16.17943842560941,"y":49.17372834079743,"z":-6.723558180699183},"hopStartPosition":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044},"hopTime":0.4105757432869891,"restTime":1.733928075687837},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null} actual: {"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.216618591386859,"hopTime":0.3102949295904416,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-16.28015246843647,"y":38.86503891787034,"z":1.768838080914106},"hopStartPosition":{"x":-52.238887323577785,"y":-9.744994988234037,"z":285.38250751764895}}} naturalSelection.introScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs. Expected: {"units":null,"validValues":null,"value":{"x":242.34395155461738,"y":-22.852816208966374,"z":265.72077568655044}} actual: {"value":{"x":-52.238887323577785,"y":-9.744994988234037,"z":285.38250751764895},"validValues":null,"units":null} naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs. Expected: {"_private":{"speed":158.03332201218853}} actual: {"_private":{"speed":197.3664738102289}} naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs. Expected: {"units":null,"validValues":null,"value":{"x":-125.23828095416042,"y":-58.25886076740691,"z":212.61170884888963}} actual: {"value":{"x":119.24290492209968,"y":-58.093556786655654,"z":212.85966482001652},"validValues":null,"units":null} naturalSelection.introScreen.model.wolfCollection.wolfGroup.archetype.xDirectionProperty._data.initialState differs. Expected: {"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"} actual: {"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null} naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype._data.initialState differs. Expected: {"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":15.026184316645049,"y":41.21830940413096,"z":-10.247822414681844},"hopStartPosition":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693},"hopTime":0.40510814444208754,"restTime":1.926434494549918},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null} actual: {"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":1.4965507520181018,"hopTime":0.4575100400118636,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":-12.829774489733234,"y":42.61902453026255,"z":-10.258552183046708},"hopStartPosition":{"x":255.24723725659146,"y":-26.555662155718473,"z":260.1665067664223}}} naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype.positionProperty._data.initialState differs. Expected: {"units":null,"validValues":null,"value":{"x":-204.4362527095605,"y":-5.374966999287115,"z":291.9375495010693}} actual: {"value":{"x":255.24723725659146,"y":-26.555662155718473,"z":260.1665067664223},"validValues":null,"units":null} naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.archetype.xDirectionProperty._data.initialState differs. Expected: {"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"} actual: {"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null} naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0._data.initialState differs. Expected: {"_private":{"cumulativeHopTime":0,"cumulativeRestTime":0,"hopDelta":{"x":-14.756223648971094,"y":47.44263166490198,"z":7.335895397398229},"hopStartPosition":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763},"hopTime":0.4596950485642879,"restTime":1.9745883803442004},"age":0,"father":null,"generation":0,"isAlive":true,"mother":null} actual: {"father":null,"mother":null,"generation":0,"isAlive":true,"age":0,"_private":{"restTime":2.021831147444063,"hopTime":0.42458830499204303,"cumulativeRestTime":0,"cumulativeHopTime":0,"hopDelta":{"x":17.842566644128816,"y":35.56028711778421,"z":-8.173613873384644},"hopStartPosition":{"x":-299.1660441608122,"y":-3.3060569794218586,"z":295.0409145308672}}} naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.positionProperty._data.initialState differs. Expected: {"units":null,"validValues":null,"value":{"x":-225.75124546357583,"y":-45.09699228876825,"z":232.35451156684763}} actual: {"value":{"x":-299.1660441608122,"y":-3.3060569794218586,"z":295.0409145308672},"validValues":null,"units":null} naturalSelection.labScreen.model.bunnyCollection.bunnyGroup.bunny_0.xDirectionProperty._data.initialState differs. Expected: {"units":null,"validValues":["LEFT","RIGHT"],"value":"LEFT"} actual: {"value":"RIGHT","validValues":["LEFT","RIGHT"],"units":null} naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype._data.initialState differs. Expected: {"_private":{"speed":161.4491569500465}} actual: {"_private":{"speed":134.9222695517728}} naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype.positionProperty._data.initialState differs. Expected: {"units":null,"validValues":null,"value":{"x":-260.3667461546996,"y":-13.013580371909939,"z":280.4796294421351}} actual: {"value":{"x":168.88009675579448,"y":-80.42750768111614,"z":179.3587384783258},"validValues":null,"units":null} naturalSelection.labScreen.model.wolfCollection.wolfGroup.archetype.xDirectionProperty._data.initialState differs. Expected: {"units":null,"validValues":["LEFT","RIGHT"],"value":"RIGHT"} actual: {"value":"LEFT","validValues":["LEFT","RIGHT"],"units":null} ```
zepumph commented 3 years ago

I found https://github.com/phetsims/phet-io/issues/1765, noting that the generated API file is consistent as it pertains to randomness. From this I don't feel like this needs to block publication. The issue is in our ability to maintain master and prevent changes from occurring that change the API on master.

pixelzoom commented 3 years ago

OK, I think I understand why this doesn't need to be blocking -- it's relevant to master, not a release branch.

But if these types of errors persist, how will we identify real errors? And will these types of errors occur in CT?

zepumph commented 3 years ago

This is certainly something to fix, and to not let linger.

Any idea why this is not showing up as an error in CT?

I poked around and tried to see if CT was seeding anything, purposefully or accidentally. I couldn't find anything. More investigation to be done in https://github.com/phetsims/phet-io/issues/1799

zepumph commented 3 years ago

My guess is that something is broken with this test for NS, and it is breaking something in CT, because this feels unlikely to be probabilistically possible:

image

zepumph commented 3 years ago

I recommend closing this issue after @pixelzoom notes that locally this link will not fail:

http://localhost:8080/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&randomSeed=12345&ea&debugger

And changing the randomSeed to anything else will cause an assertion error.

We will continue the CT trouble over in https://github.com/phetsims/phet-io/issues/1799.

pixelzoom commented 3 years ago

And changing the randomSeed to anything else will cause an assertion error.

I can use any number for randomSeed and it does not cause an error. For example http://localhost/~cmalley/GitHub/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&randomSeed=789&ea&debugger

You've also duplicated the API-generation seed "12345" in two places: generatePhetioMacroAPI.js, listContinuousTests.js. This should be factored out, not duplicated in string literals.

Back to @zepumph.

zepumph commented 3 years ago

I can use any number for randomSeed and it does not cause an error. For example http://localhost/~cmalley/GitHub/natural-selection/natural-selection_en.html?brand=phet-io&phetioStandalone&phetioCompareAPI&randomSeed=789&ea&debugger

With that link I get image

I see that I am pushed, pulled, and entirely on master.

You've also duplicated the API-generation seed "12345" in two places: generatePhetioMacroAPI.js, listContinuousTests.js. This should be factored out, not duplicated in string literals.

Yes, it is duplicated in more than just those two spots. I cannot think of a place to factor it out to, see https://github.com/phetsims/phet-io/issues/1798#issuecomment-877845936 for discussion. Conclusion still TBD.

pixelzoom commented 3 years ago

Hmm.... It seems to be working as expected now. Maybe I didn't have something pulled, or had a cacheing problem. Sorry for the reassignment.

Closing - because this pertains to master, it does not need to be verified by QA for phetsims/qa#662.