PhET-iO Input Event Instrumentation

[samreid]

On Feb 12, 2017, at 11:06 AM, Whitacre, Ian asked if we can instrument the simulation for input events. This will likely be not too much work, mainly making sure it can run without tandems. I'll look into this part.

@jbphet can you please comment on whether the simulation is stable enough to use with one of our collaborators? I'm not sure what level of QA @kathy-phet has in mind.

[kathy-phet]

FYI - He's likely to use it, even if it has some bugs! He is part of the grant group for this project so even if it has bugs, we will learn something. Teachers know these are "underconstruction". The game will not be used, obviously.

[samreid]

The preceding commit makes it possible to run the simulation with phetioValidateTandems=false and to capture the input events. After we get the go-ahead from @jbphet we can create a new Metacog application for this use case.

@kathy-phet any idea what to use for the Learner ID login pattern? Would you like to check with Ian about it?

[jbphet]

Wow, I had no idea that instrumentation for this simulation was on the horizon. I am currently in the process of developing the game screen, and as I do that I'm generalizing the code from the other screens to maximize the amount of reuse. This is causing a lot of code to change, little of which has been retested, so I think that starting to instrument the sim now would result in a fair amount of instability. Also, there is one major refactor that I was considering doing once the bulk of the functionality is in place, which is essentially eliminating the model-view divide. The MV architecture feels a bit forced in this sim, and makes things more complex than perhaps they need to be. @pixelzoom ran into a similar situation in the Function Builder simulation and ended up not having a separate model.

I just talked with @samreid via Skype, and he said we would also want to establish a time when we could create a stable version of the simulation and run it through a QA cycle. This is possible, but would block continued development of the game until it was completed. @samreid and I discussed the possibility of using a previous stable release instead. I just went through my records, and version 1.0.0-dev.18 was published in November of 2016 for the purpose of interviewing, so it got a fair amount of QA testing (see https://github.com/phetsims/tasks/issues/735). Since the game is apparently not needed by Ian yet, perhaps using 1.0.0-dev.18 would be a better approach.

The dependencies file for the 1.0.0-dev.18 release can be found here: http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.0-dev.18/dependencies.json

@samreid - can you take a look and see if this release could be used as the basis for adding input event instrumentation?

[pixelzoom]

@jbphet said:

@pixelzoom ran into a similar situation in the Function Builder simulation and ended up not having a separate model.

Function Builder has no separate model-view transform (the coordinate frames are identical), so perhaps that's what you're referring to. But in all other respects, it has very separate model and view.

[kathy-phet]

John - Just to clarify ... input instrumentation is on the horizon, not full PhET-iO instrumentation! (no events, no states). That is not even on the radar at all.

The input instrumentation just allows Ian to essentially collect video of the students interacting with the sim, since we don't have the capability of recording from the screen in the classrooms.

[kathy-phet]

Dev.18 would be good for Ian if Sam could build with those shas. If it's not straight forward Sam let me know.

Kathy

Sent from my iPhone

On Feb 13, 2017, at 3:09 PM, John B notifications@github.com<mailto:notifications@github.com> wrote:

Wow, I had no idea that instrumentation for this simulation was on the horizon. I am currently in the process of developing the game screen, and as I do that I'm generalizing the code from the other screens to maximize the amount of reuse. This is causing a lot of code to change, little of which has been retested, so I think that starting to instrument the sim now would result in a fair amount of instability. Also, there is one major refactor that I was considering doing once the bulk of the functionality is in place, which is essentially eliminating the model-view divide. The MV architecture feels a bit forced in this sim, and makes things more complex than perhaps they need to be. @pixelzoomhttps://github.com/pixelzoom ran into a similar situation in the Function Builder simulation and ended up not having a separate model.

I just talked with @samreidhttps://github.com/samreid via Skype, and he said we would also want to establish a time when we could create a stable version of the simulation and run it through a QA cycle. This is possible, but would block continued development of the game until it was completed. @samreidhttps://github.com/samreid and I discussed the possibility of using a previous stable release instead. I just went through my records, and version 1.0.0-dev.18 was published in November of 2016 for the purpose of interviewing, so it got a fair amount of QA testing (see phetsims/tasks#735https://github.com/phetsims/tasks/issues/735). Since the game is apparently not needed by Ian yet, perhaps using 1.0.0-dev.18 would be a better approach.

The dependencies file for the 1.0.0-dev.18 release can be found here: http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.0-dev.18/dependencies.json

@samreidhttps://github.com/samreid - can you take a look and see if this release could be used as the basis for adding input event instrumentation?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/phetsims/expression-exchange/issues/57#issuecomment-279538843, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AE3FZPZl4R-39QFBSQyrV4M510iEqOtdks5rcNRXgaJpZM4L_rvv.

[samreid]

Dev.18 is incompatible with master phet-io. PhET-iO from the same day seems to work, @zepumph do you think using the input events from Nov 17 will be OK? It seems OK in the mirror wrapper from nov 17.

[jbphet]

@kathy-phet said,

John - Just to clarify ... input instrumentation is on the horizon, not full PhET-iO instrumentation! (no events, no states). That is not even on the radar at all.

Yes, I understand that. My concern is more that the code has undergone a lot of changes recently and isn't really in an ideal state to do a quick release. It can be done, but it will likely take time away from the main development effort, so it would be less disruptive to go with a previous dev release if possible. If not, we'll do it from master.

[samreid]

Lots of hacks to get this to work. EE isn't working with ?ea, builds are failing because package.json is missing uglify. Lint isn't passing. I tried to build an RC but it isn't appearing. I'll check the build logs.

[samreid]

Oh perhaps build failure due to lint error.

[samreid]

I patched phet-io with a branch and published an RC version here: http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.0-phetiorc.1/wrappers/mirror-inputs/mirror-inputs.html?sim=expression-exchange&launchLocalVersion

Next steps:

1. Testing the mirror wrapper
2. Building a metacog application
3. Testing sending data to metacog
4. Test retrieving data from metacog (may be buggy currently)
5. Test playing back data (may be difficult if metacog buggy and we don't have saveToLocalFile feature)

[kathy-phet]

Thanks.

@John- priority is move forward with your current implementation of what you are doing, not pause to make a stable version now. But since Sam was able to make dev.18 work we will try to deliver that with inputs to Ian.

K

Sent from my iPhone

On Feb 14, 2017, at 10:19 AM, Sam Reid notifications@github.com<mailto:notifications@github.com> wrote:

I patched phet-io with a branch and published here: http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.0-phetiorc.1/wrappers/mirror-inputs/mirror-inputs.html?sim=expression-exchange&launchLocalVersion

Next steps:

1. Testing the mirror wrapper
2. Building a metacog application
3. Testing sending data to metacog
4. Test retrieving data from metacog (may be buggy currently){
5. Test playing back data

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/phetsims/expression-exchange/issues/57#issuecomment-279773350, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AE3FZPcJejU7s4pDDl9YeAfBAbsn6C2Jks5rceH_gaJpZM4L_rvv.

[samreid]

Ian said:

I believe we already have 100 IDs that begin with C. Those are for Julie's school.

@kathy-phet said:

For expression exchange, just use Ian's same set of logins that he has had.

[zepumph]

Dev.18 is incompatible with master phet-io. PhET-iO from the same day seems to work, @zepumph do you think using the input events from Nov 17 will be OK? It seems OK in the mirror wrapper from nov 17.

Yeah I think that would be fine. I'm a little worried about us knowing the api for these shas. I guess we can always look in the client manually to figure things out. Sorry for a late response, it looks like you moved on in the aforementioned direction.

Let me know if you need anything else from me.

[zepumph]

For expression exchange, just use Ian's same set of logins that he has had.

This seems reasonable because we will still be able to filter data based on a different widget_id, and date_start

[samreid]

The only “C” I remember is CMS16001a which was for Casey Middle School for Susan’s project.

[samreid]

Here's the URL I have constructed so far. Needs testing and perhaps a new validation rule:

http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.0-phetiorc.1/wrappers/login/login.html?sim=proportion-playground&screens=1.2&metacog&publisher_id=c290c6ab&application_id=047d5548d7dd2de2584c58d34a9bc40d&widget_id=record-expression-exchange&phet-io.emitInputEvents&wrapper=record&validationRule=validateFSU

[zepumph]

From validateFSU.js:

  var startLetter = [ 'U', 'F', 'C', 'R' ];

Looks like C will work just like R and U did for Ian.

[samreid]

Here is the URL we ended up with for the RC: http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.0-phetiorc.1/wrappers/login/login.html?sim=expression-exchange&screens=1,2,3&metacog&publisher_id=c290c6ab&application_id=047d5548d7dd2de2584c58d34a9bc40d&widget_id=record-expression-exchange&phet-io.emitInputEvents=true&wrapper=record&validationRule=validateFSU&launchLocalVersion

We also used R16001a to send a small sample session. Hopefully one day we will be able to retrieve it from metacog.

For the production one, we should strip out the "launchLocalVersion" (though it may still work with that).

[phet-steele]

@samreid have you tried following that link? The sim's background is white for me and it becomes very hard to use.

[samreid]

@phet-steele we tried it and it seemed legit on my Mac. Not opposed to further testing or more detail about the error you saw, though. (Please note that I changed the link a few minutes ago though).

@zepumph can you please try requesting data for that session above, and let me know what happens?

[zepumph]

@phet-steele It looked good for me as well. Windows 10 chrome.

[phet-steele]

I just ran the sim is all, on both macOS 10.12.3 and Win 10 Chrome. Simply went to the first screen and it has a white background. It's supposed to be a light green like the screen icons:

This screenshot is not from the above link, but nevertheless I saw it there, too

[zepumph]

I requested data to go to @samreid's email. Here is the curl conversation:

curl -X POST --header "Content-Type: application/json" --header "Accept: application/json" --header "publisher_id: c290c6ab" --header "application_id: 047d5548d7
dd2de2584c58d34a9bc40d" -d "{
>   \"widget_ids\": [
>     \"record-expression-exchange\"
>   ],
>   \"date_start\": \"2017-01-01\",
>   \"email\": \"reids@colorado.edu\"
> }
>
> " "https://api.metacog.com/DataRequest"

{"id":"b66ae38d-8af5-48fe-a9d4-dd9a418b5a94","publisher_id":"c290c6ab","application_id":"047d5548d7dd2de2584c58d34a9bc40d","request_type":"data","widget_ids":["record-expression-exchange"],"learner_ids":[],"date_start":"2017-01-01","date_end":"None","email":"reids@colorado.edu","status":"QUEUED","created_at":1487103782406000}

I also made a request for my email address. I will keep track of this today, but will not probably be able to follow through completely.

[zepumph]

In regards to @phet-steele, we were seeing that too, just didn't notice. @samreid perhaps this has to do with the problem we were finding with the 'homeScreenColor', perhaps ea would catch the root cause of the problem.

[zepumph]

I just got the data request back. It was only 4 MB, and it successfully played back on the playback wrapper on master. I think we are all clear.

[samreid]

I got the data request, but am seeing this error on playback when using the phet-io website. Should I be using master on localhost?

Mixed Content: The page at 'https://phet-io.colorado.edu/metacog/playback/' was loaded over HTTPS, but requested an insecure resource 'http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.0-phe…63@1487103117436&phet-io.emitStates=false&screens=1,2,3&launchLocalVersion'. This request has been blocked; the content must be served over HTTPS.

[zepumph]

Ahhh I'm using the local playback wrapper, which is on http. But no matter I don't think this would be a problem because we will publish this sim onto phet-io which is https no?

[samreid]

Thanks @zepumph, I used localhost to check the playback and it matched what I recorded. @kathy-phet do you want any more testing on this RC before we publish it to the phet-io website?

[samreid]

I'll start the publication process, but we can always update it if @kathy-phet recommends further testing.

[samreid]

I published [redacted]

I realized it has the white background--I'll check if dev.18 had that too.

[samreid]

Here are the changes I observed between dev.18 and 1.0.0-phetio (none of which are expected)

Feature	dev.18	1.0.0-phetio
background color	green	white
variables in carousel page 1	x^2, x, y, 1	x, -x, 1, -1
toolbox size	small	large
explore screen collection box	wide	thin

These changes are all quite disconcerting because I was supposedly using the same shas from dev.18's dependencies.json.

[samreid]

Perhaps I branched from master instead of 082f9cea5165fa08cd6b11337b5a71a0bdb49abc ?

[samreid]

I rebranched from the sha mentioned above and things are looking much better.

[samreid]

Here's the new testing URL: [redacted]

[kathy-phet]

Sam, I was going to try this out, what is the test password for the wrapper?

[samreid]

I've been using R16001a

[kathy-phet]

I tried it out, and in terms of sim behavior it is looking good. I used the above password too. K

[samreid]

@kathy-phet would you like to share the link with Ian?

[kathy-phet]

Before I share this, can someone un-password protect the link? It will still have the login page, but it needs to not have the phet-io user account protection. Thanks!

[kathy-phet]

Also, @samreid, what would be the link for looking at the playback wrapper? I cobbled together this one: http://www.colorado.edu/physics/phet/dev/html/expression-exchange/1.0.1-phetio/wrappers/mirror-inputs/mirror-inputs.html?sim=expression-exchange&launchLocalVersion It seems to work -- is this right?

[samreid]

The mirror-inputs wrapper allows immediate testing of transient input events (not stored to metacog), it is primarily for testing. For playback, clients will get their data from metacog then use the wrapper at: https://phet-io.colorado.edu/metacog/playback/ . Note that playback wrapper is incompatible with sims published on HTTP (such as spot), and can only play back sims recorded from HTTPS, such as the phet-io website.

To request the data, clients will use this page (though I don't think @zepumph told clients about it yet because we are still working out issues with Metacog): https://phet-io.colorado.edu/metacog/request-data.html

[kathy-phet]

Hi Sam, Yes - I understood that (except for the https need), but is the link above correct for checking the mirroring? It seemed to work for that.

[samreid]

Yes, that link is sufficient for testing whether the input events could be used for reliable playback.

[samreid]

@kathy-phet I removed password protection from the link in https://github.com/phetsims/expression-exchange/issues/57#issuecomment-279862419

Thanks for the reminder!

[kathy-phet]

I shared the link.

[jbphet]

An instrumented version was delivered, and apparently met the needs of the users. Subsequent phet-io instrumentation should be done under a new GitHub issue. Closing.