Simulation mode idea: data playback

[jodeleeuw]

Add an option to jsPsych.simulate() to accept a complete data object (e.g., jsPsych.data.get()) from a run of the same experiment and replay the behavior in visual mode.

For experiments with randomization, this would require some method of recreating the same randomized sequence. Generating a more accurate playback would also require collecting more data in some plugins, like the timing of particular form entries. Getting to the point of being able to generate very accurate playbacks could be a way to identify useful directions to move in development.

[nikbpetrov]

my $0.02: If this would genuinely be used to search for directions for development (which I can definitely use in some of my experiments -- in some designs, participants seem to be using sliders very weirdly so I want to find out why), I see it almost as a slight separate feature. I would want ALL participant behaviour reproduced including every px of a mouse move per ms, every keystroke etc. I want a perfect playback of what the PP did. The amount of data collect would therefore be large so I see this as a useful feature in a pilot, where I can set playback_data_save: true and then use the saved data to replay the PP's behaviour.

I believe collecting the above data, for most cases, albeit arduous to set up, is possible.

It's also possible to have the same replay function accepting default parameters or some 'typical' jsPsych data (i.e. not one produced from the extensive data collection from playback_data_save: true) that uses some random times/generators (i.e. wait 500 ms, then in form field 1 the word 'alohomora' is written at once, rather than keystroke by keystroke; mouse does not move; button clicks are done automatically etc)

[nikbpetrov]

Okay, happy to report some exciting development on this front as I spent some time researching this as part of a project for which I would find this feature very useful.

As far as I see it, there are two major components to this: 1) recreating the DOM and 2) simulating the user behaviour.

Re-creating the DOM

Luckily, it appears that recreating the DOM is super easy, given jsPsych's existing setup. Basically, all we need to recreate the entire timeline is... well.. the timeline! Again, luckily, the timeline is a simple JS object. Once a PP submits a response, we can save the timeline they saw as a JSON object. To do this, we can use the Cryo package, which extends the JSON methods with much-improved functionality. It's as simple as:

// saving the timeline
let timeline_stringified = Cryo.stringify(timeline)

// retrieving the timeline for replay
let timeline = Cryo.parse(timeline_stringified)

A few considerations:

• To make sure this is working, the latest build of the Cryo package (0.0.6) needs to be updated to work with jsPsych - see here
• Crucially, the Cryo package has been reported to contain critical security vulnerabilities. See here and here. The latter link seems to contain a solution. This vulnerability is quite dangerous as I think a participant might insert some malicious code into the timeline object; then, when the object is deserialized by the experimenter, the malicious code might be executed. An in-house jsPsych-native handling of the (de-)serialization process is also possible - and desirable, albeit arduous.

Simulating the user behaviour

Once again, we are lucky in this regard, though there is some effort in bringing it all together. The existing simulation mode, mouse tracking extension, as well as my proposed keyboard tracking (#2868), all provide the blueprint for simulating the user behaviour.

Additionally, I've actually ended up using the jsReplay package, to achieve the replay functionality and it works. This functionality definitely needs to be brought in-house, as further customization (e.g. the mouse moving visualisation for the mouse tracking extensions is superb) and maintenance (e.g. in case event properties change in future JS releases and/or browsers) would definitely be required.

Timing would also need to be handled nicely, in case the experiment requires sub-16ms precision (my below demos is off by 50ms or so).

Demo

Here's a quick demo but note that this is customized to work on my setup so I don't know if it's replicable (only works in Chrome due to jsReplay compatibility - see repo). I've customized the jsReplay code so that it saves the playback data in a localStorage variable and retrieves it from there. I've done the same for saving the stringified timeline - also stores in the local storage.

(note that jsReplay outputs in the console "scroll event" although it's more like a mouse move - it's not perfect!)

<!DOCTYPE html>
<html>
<head>
    <title>My experiment</title>
    <script src="dist/jspsych.js"></script>
    <script src="dist/plugin-html-keyboard-response.js"></script>
    <script src="dist/plugin-html-button-response.js"></script>
    <link href="dist/jspsych.css" rel="stylesheet" type="text/css"></link>

    <script src="cryo-0.0.6.js"></script>
    <script src="replay.js"></script>
    <!-- jquery necessary for replay -->
    <script src="jquery-3.6.0.min.js"></script>
</head>

<body>
</body>

<script>

    // REPLAY = false if you want to record
    // REPLAY = true if you want to ... replay
    const REPLAY = true

    let timeline = []
    if (REPLAY) {
        // the stringified timeline is saved in a local storage browser for the demo and is then retrived from there
        timeline = Cryo.parse(localStorage.getItem('timeline'))
    } else {
        var trial1 = {
            type: jsPsychHtmlKeyboardResponse,
            stimulus: function() {return `timeline var var1: ${jsPsych.timelineVariable('var1', true)}`},
            choices: ' '
        };

        var trial2 = {
            type: jsPsychHtmlButtonResponse,
            stimulus: 'trial2',
            choices: ['button1', 'button2'],
            on_start: function(trial) {
                // this and the above stimulus function are both evidence that retriving the stringified timeline works like a charm!
                trial.stimulus = `timeline variable var2: ${jsPsych.timelineVariable('var2', true)}`
            }
        };
        timeline.push({timeline: [trial1, trial2],
                        timeline_variables: [{var1: 'a', var2: 'b'}]})
    }

    let jsPsych = initJsPsych({
        on_finish: function() {
            jsPsych.data.displayData('json');

            if (!REPLAY) {
                // save the stringified timeline
                localStorage.setItem('timeline', Cryo.stringify(timeline))
                // stop the recording - replay.js code modified to save the stringified JSON in a localstorage variable, called 'playbackScript'
                jsReplay.record.stop();
            }
        }
    })

    jsPsych.run(timeline)

    if (REPLAY) {
        // start the replay - the argument refers to the local storage variable that holds the stringified JSON
        var widgetTest = new jsReplay.playback("playbackScript");
        widgetTest.start();
    } else {
        jsReplay.record.start();
    }

  </script>
</html>

[nikbpetrov]

Draft in #2894