phetsims / molecule-polarity

"Molecule Polarity" is an educational simulation in HTML5, by PhET Interactive Simulations.
GNU General Public License v3.0
2 stars 6 forks source link

Publish PhET-iO 1.1 RC to support research study (4/4/2021) #87

Closed pixelzoom closed 3 years ago

pixelzoom commented 3 years ago

This issue was the motivation for #85 (Add basic PhET-iO instrumentation).

A research collaborator would like to use this sim for a study. @kathy-phet provided some info on Slack, here's my summary:

Unless I hear otherwise, I'll assume "end of March" means 3/31/21.

pixelzoom commented 3 years ago

Let's discuss the plan to provide a dev version. Two concerns:

(1) The dev team has previously recommended against providing dev versions. A dev version cannot be patched via PhET processes and tools. An RC version has no additional cost, and provides all of the maintenance benefits.

(2) The dev team has previously expressed concerns about the impact of study traffic on phet-dev.colorado.edu, and has recommended against giving links to phet-dev.colorado.edu. This will be an issue for a dev or RC version. Do we know how many students are involved in this study? (I recall that @kathy-phet was going to ask.) Has @mattpen been consulted?

pixelzoom commented 3 years ago

Re the QA plan:

QA will do some testing, as will @kathy-phet, @arouinfar, and the researcher.

PhET-iO instrumentation (#85) required big changes throughout the sim. There is a very good chance that something is broken. So I recommend a thorough test, definitely not a spot check. If we don't have time for a thorough test, then let's make sure that we are all aware of the risk to the study.

I'm unclear on what "QA will do some testing" means. I know what the process and expectations are for "Dev Test", "RC Test", and "RC-Lite Test". It sounds like PhET will be doing similar QA for more Ido studies, so I think its important that we all know how to communicate about this type of test. Can we define a name and process for this type of test, which I interpret to be less rigorous than "RC-Lite"?

Proposal, assuming that PhET will provide an RC version:

"RC-Research":

  • The sim will be reviewed by QA, preferrably on the platform(s) that will be used by the researcher.
  • A test martix will be created.
  • A QA issue will be created and labeled "RC-Research".
  • The QA issue will be specific about what to focus on. Focus will be tailored to the needs of the research study. (Specific features, specific screens,...)
  • Test coverage will be somewhere below "RC-Lite" testing, as time permits.
  • Additionally, the simulation team and the researcher may be asked to test.
pixelzoom commented 3 years ago

3/4/21 phet-io meeting

We discussed the deliverable and the QA plan.

Consensus:

pixelzoom commented 3 years ago

3/11/21 phet-io meeting:

The research collaborator was provided with 1.1.0-dev.2 for review. Feedback was that it looked good. There were no change requests.

pixelzoom commented 3 years ago

Update: The researcher needs the link to the final version by 4/4/21.

pixelzoom commented 3 years ago

NOTE: I can publish the RC version using grunt rc. Work with @zepumph to create the links for QA and the researcher, so that Metacog recording will work.

zepumph commented 3 years ago

Note that the qa issues will likely look much like https://github.com/phetsims/QA/issues/624.

pixelzoom commented 3 years ago

1.1.0-rc.1 has been published and is ready for testing. @zepumph and I created the QA issue, with appropriate links: https://github.com/phetsims/QA/issues/627.

pixelzoom commented 3 years ago

On 3/23/21, @arouinfar sent the research collaborator a link to https://phet-dev.colorado.edu/html/molecule-polarity/1.1.0-rc.1/phet-io/, and requested feedback by Monday 3/29/21 @ 9:00 AM MDT:

Molecule Polarity is currently undergoing QA testing for your upcoming study. Here is the latest version: https://phet-dev.colorado.edu/html/molecule-polarity/1.1.0-rc.1/phet-io/

Please review this version to ensure that it will meet your needs. If you have any feedback or find any bugs, please let us know by Monday 9:00 AM MDT so that we will have time to make changes before your study begins.

Best, Amy

pixelzoom commented 3 years ago

QA complete testing of 1.1.0-rc.1 on 3/24/21. There were 2 issues sim-specific issues (#106, #107), see phetsims/QA#627.

Feedback from the research collaborator on 3/25/21:

Hi Amy and all, This looks terrific - no comments from us. Thanks.

I will proceed with the next RC (spot-check).

pixelzoom commented 3 years ago

Progress is blocked. I can't create the next RC due to a problem with the build server, see https://github.com/phetsims/perennial/issues/221.

pixelzoom commented 3 years ago

phetsims/perennial#221 is no longer blocking.

1.1.0-rc.2 was published and is being tested in https://github.com/phetsims/QA/issues/631.

pixelzoom commented 3 years ago

1.1.0-rc.2 testing has been completed, see https://github.com/phetsims/QA/issues/631. All previous issues were successfully verified. There was one new issue #113 which does not need to be addressed for this research version.

@kathy-phet @arouinfar 1.1.0-rc.2 is ready to be delivered to the research collaborator. The link to the index wrapper is: https://phet-dev.colorado.edu/html/molecule-polarity/1.1.0-rc.2/phet-io

Please note in this issue when 1.1.0-rc.2 has been delivered, then close this issue.

kathy-phet commented 3 years ago

Thanks - will work with MK when he gets back from vacation, to make sure I send correct links for data collection and testing. On hold until March 31.

arouinfar commented 3 years ago

Unassigning myself since @kathy-phet will be sending the final links to the collaborator.

zepumph commented 3 years ago

I don't want phet-io links in public issues, so I made https://github.com/phetsims/special-ops/issues/195 for the delivery, please see there. Closing.