Currently covfee make is creating the database and pushing instances based on the spec found in the main .py file. While this is ok to conduct a well scoped experiment, in practice, the research questions keep evolving and/or there are problems with annotators, or more data was to be annotated, etc. etc.
Therefore, we propose to add the capability to covfee (likely through the covfee make command) to shutdown the server, run covfee make again, and to retain the database with the data obtained so far whereas new journeys can be pushed to the database. In practice, this will allow to address the following use cases:
Mid-study, decide to do inter-annotator agreement, or increase the number of inter-annotators. More journeys could be added to the same task.
Mid-study for action annotations, deciding to add other actions to annotate (for example speaking, laughing, etc.). More journeys could be added but specifying the new action.
Conduct the same experiments but on different data. More journeys would be added specifying said data.
Assuming the annotations from an annotator are useless. Thus, the task could be "cloned", but the study manager would manually keep track on which annotator data to ignore.
Currently covfee make is creating the database and pushing instances based on the spec found in the main .py file. While this is ok to conduct a well scoped experiment, in practice, the research questions keep evolving and/or there are problems with annotators, or more data was to be annotated, etc. etc.
Therefore, we propose to add the capability to covfee (likely through the covfee make command) to shutdown the server, run
covfee makeagain, and to retain the database with the data obtained so far whereas new journeys can be pushed to the database. In practice, this will allow to address the following use cases: