NVlabs / FPSci

Aim Training Experiments
Other
70 stars 23 forks source link

Task flow integration #396

Closed bboudaoud-nv closed 1 year ago

bboudaoud-nv commented 2 years ago

This branch adds support for "tasks" in FPSci. The new heirarchy of experiments is as shown below:

Experiment (general configuration)
|- Session (more specific configuration, may include questions)
   |- Block (repeat of conditions below, doesn't support questions)
      |- Task (affiliated set of trials in an order, may include questions)
         |- Trial (most specific configuration, may include questions)

Old configuration files still provide the same experience by using the count from the trials array to generate tasks when no tasks array is specified. In this model each trial from prior configs becomes a task of its own.

The new model allows for a tasks array to be specified with the following structure:

tasks = [
    {
        id = "example task";  // A name to refer to this task with (when logging)
        // A task specified to run 2 trials in their 2 possible orders
        trialOrders = [
            {order= ["trial1", "trial2"];},
            {order = ["trial2", "trial1"];}
        ];
        // This question(s) will only be asked once both trials in a full "trialOrder" are complete
        questions = [
            {
                 // Specify question here (same as session/trial-level questions)
            } 
        ];
];
jspjutNV commented 1 year ago

In reviewing this I noticed that the correctAnswer was something I naively assumed would show up in the results db next to the answers in the Questions table, but it isn't printed there. Since the correct answer was only included to provide user feedback, it makes sense that we wouldn't record it.

@joohwankimNV , would you prefer that the correct answer showed up in the Questions table, or is it okay for us to just use the traditional style to figure that out? It's probably worth having you try to run a pilot to see if you can get the analysis you want out of this.

jspjutNV commented 1 year ago

I confirmed with Joohwan that the correct answer does not need to appear in the Questions table.