Data preprocessing -- Correspond with Kevin

[lrperlmu]

Might offer to take over on this if I can get to it before he can. But, we probably won't need it in a hurry. And I have other stuff to do right now. Hold off for now and just correspond on slack.

[lrperlmu]

also ask his advice on what our sample size should be to reasonably find significance given the number of groups

[lrperlmu]

12/8

• Leah to update the scripts to they produce the right CSV
• see also: the script I wrote to read pilot responses

[lrperlmu]

Journal

12/10

• gather requirements for data mining script
• list the pids that we think represent participants who completed the study (have completion code, and [assign, start, end] events)

12/14

• get the responses for each valid pid
• look into assigning slugs to each question

12/15

• reproduce the knowledgebase.js file
• add id slugs to each question in knowledgebase.js
• ruminate about the necessity of assigning slugs to each question

12/17

• code to validate that the expected questions are present in the incoming data (COMPLETENESS)

2/25

• revisiting these scripts after a while away working on TA and cs ed research project
• load stuff back into brain
• move some utility functions into a separate file for use in the completion checking script

6/16

• filter for participant data objects that have data in all the categories of the present app
• 1st draft of writing data to csv

To do:

• produce CSV for Kevin
• validation
  • update app so the data goes into the latest categories
  • walk through the app and enter some fake data
  • run data validation on it

Validation thoughts:

• this involves creating data that we can validate. what might we want to validate?
  • whether a given row of data is "complete" -- all the required questions answered
  • whether a given row is "valid" -- all the entries are a question that was supposed to be answered
  • we might want to do this to validate the integrity of the data -- that the app is outputting correct data -- but it's not like users have any control over it.

Clean up firebase for switchover to latest data model

1. download and save json (sensitive, has pilot data)
2. wipe the current firebase instance

Write data mining scripts

• write data mining script (old format)
• mine data from pilots (old format)
• make data mining script configurable to new format
• make some new fake data
• test the script on it

[lrperlmu]

Reproducible process for getting knowledgebase.js from the google sheet

Procedure

As of commit 0dc3fbf, output of convert_to_json matches appside/data/knowledgebase.js that dbt_worksheet_model consumes

• sort by Source, Emotion, Category (equivalent to sorting by statement ID)
• select columns: Statement, Emotion, Valence, Category, Body Part and rows with Source: DBT worksheet
• copy into a scratch spreadsheet and download as CSV
• run convert to json

to do

• try the procedure above and diff with existing knowledgebase.csv and knowledgebase.json
• add colmun to knowledgebase
  • identifier for each question
    • make sure existing code doesn't barf on identifiers
• add code to convert_to_json to
  • add line breaks
  • add assignment statement