search

phetsims / aqua

Automatic QUality Assurance
MIT License
2 stars 4 forks source link

How do I tell CT not to run PhET-iO tests? #113

Open pixelzoom opened 3 years ago

pixelzoom commented 3 years ago

From https://github.com/phetsims/fourier-making-waves/issues/25#issuecomment-743917935 ...

Fourier is a new sim that is in development. I am incrementally adding PhET-iO support, and I don't expect the sim to pass tests related to PhET-iO. At this stage of implementation, I'm mainly interested in structure of the Studio tree. I don't care whether there are State wrapper problems, and I fully expect that there will be problems. 1.0 release is not going to include brand=phet-io or any PhET-iO testing.

I've added this to package.json:

    "phet-io": {
      "validation": false
    }

But this only turns off validation. I still see PhET-iO tests being run by CT (e.g. phet-io-state-fuzz : unbuilt in https://github.com/phetsims/fourier-making-waves/issues/25) and there can be other non-validation types of failures that I'm not interested in now (e.g. bad intermediate states when state is restored.)

The problem seems to be that the same setting ("phet-io" is included in "supportedBrands", in package.json) is used to enable PhET-iO in phetmarks and enable PhET-iO tests in CT.

How do I tell CT not to run PhET-iO tests?

samreid commented 3 years ago

Please correct me if I'm wrong, but I don't see a way to have a simulation listed in phet-io brand but not run any phet-io tests. We have a list to opt-out of phet-io state tests in https://github.com/phetsims/perennial/blob/master/data/phet-io-state-unsupported. Should we add another list or other lists for opting out of more tests?

pixelzoom commented 3 years ago

Thanks, I'm aware of phet-io-state-unsupported. My question is how do I opt out of all PhET-iO tests. Fourier is not going to support PhET-iO until > 1.0, but I still need to add a minimal amount of PhET-iO instrumentation in order to confirm that I'm going to be able to meet design requirements.

If it's really impossible to add the PhET-iO modes to phetmarks without enabling CT PhET-iO tests, then I'll just remove my sim from phetmarks and create my own bookmarks. But that seems really silly, not to mention inconvenient.

pixelzoom commented 3 years ago

But that seems really silly, not to mention inconvenient.

It also makes it more difficult for my sim team to (for example) run Fourier in Studio.

pixelzoom commented 3 years ago

I've added fourier-making-waves to phet-io-state-unsupported, see https://github.com/phetsims/fourier-making-waves/issues/12#issuecomment-743939264.

zepumph commented 2 years ago

What if we add phet-io links in phetmarks to all sims? And use the list to set a default of the phetioValidation query parameter. I agree it is an annoying hurdle to have to wait for data lists to update to locally test these links.

pixelzoom commented 2 years ago

I don't know what https://github.com/phetsims/aqua/issues/113#issuecomment-1151271063 means. What are "phet-io links"?

zepumph commented 2 years ago

The links added to a sim in phetmarks that are specific to PhET-iO (studio/state/index). The only way we can add them ad this time is by having the sim in perennial/data/phet-io

pixelzoom commented 2 years ago

I don't understand how adding phet-io links for all sims addresses the problem. And it creates a new problem -- there will be bad links for sims that do not support PhET-iO. Feels like a really bad bandaid, not a solution.

The problem here is that you've coupled phetmarks links with CT phet-io tests, all controlled by the same flag. And that's not a good fit for development -- I need phetmarks links in order to test my work way before I want CT to be testing my work. So why not decouple them?

zepumph commented 2 years ago

I think I'm trying to say that all sims currently support phet-io with validation=false. So why not just let them be added to phetmarks for them? They wouldn't be bad links, they just may not do much, since sim-specific code wouldn't be supported. Do you still not like that idea?

I need phetmarks links in order to test my work way before I want CT to be testing my work. So why not decouple them?

And I find myself needing links on sims that aren't in the phet-io list quite often, so why not decouple those too?

pixelzoom commented 2 years ago

I think I'm trying to say that all sims currently support phet-io with validation=false. ...

That's not the case. http://localhost/~cmalley/GitHub/example-sim/example-sim_en.html?brand=phet-io&phetioStandalone&ea&debugger fails. There's no overrides.js.

cd example-sim; grunt --brands=phet-io also fails. phet-io is an unsupported brand, because it's not specified in package.json.

If you want to move to "all sims support PhET-iO", that's fine with me. But I'm not sure that everyone on the team will agree. And there's significant work to be done.