Watts-Lab / commonsense-platform

Commonsense platform
https://commonsense.seas.upenn.edu
1 stars 0 forks source link

Integrative experiment of statement types #86

Open markwhiting opened 9 months ago

markwhiting commented 9 months ago

Based on the idea of https://github.com/Watts-Lab/Commonsense-Platform/issues/61, we want to conduct an integrative experiment across statement design points. Here we outline the experiment and a registration for it.

Preparation procedures

Sampling procedure

  1. Sample 10 starting design points at random.
  2. Sample 15 statements for each design point.
  3. Assign 100 participants to each design point using inverse weighting
  4. When a design point is finished, evaluate PQ common sense and train a model to predict other design points with all completed data.
  5. Compute model certainty on unsampled design points and select the next design point based on the one about which we are least certain.
  6. Register predictions for the new design point before starting to sample it.

Required features

We should finish these other things before launching

amirrr commented 8 months ago

Dimensions of statement and their definition:

behavior

everyday

figure_of_speech

judgment

opinion

reasoning

markwhiting commented 1 month ago

@amirrr How should I specify the experiment for this?

amirrr commented 1 month ago

@amirrr How should I specify the experiment for this?

You can create a new file that ends with *.expaeriment.js under server/survey/experiments/ or modify an already existing one and it will added to the records. For what to put in the file you can look at this example here: integrative.experiment.js