IMPORT kremin_2021

[alvinwmtan]

Kremin et al. (2021) has two data subsets: Eng–Fra bilingual and Eng–Spa bilingual. The former is included in montat_2022, but the latter is not; we will import this whole dataset separately.

[alvinwmtan]

IDless import complete; ready for review

[alvinwmtan]

Checklist for code review v2024

To start:

• [x] Git pull this repo to get the latest version
• [x] Update your peekds and peekbankr to the latest version
  • [x] Be sure to restart your R Session to apply these updates
• [x] Get the latest version of the dataset from osf (delete your raw_data, so that the script automatically downloads the data)
• [x] Run the import script
• [x] Does it run into issues due to missing libraries? during restructuring, import statements for libraries like janitor might have been lost in some datasets - re-add them if necessary
• [ ] Does the validator complain about the processed data? Complain to Adrian (or fix the listed issues if you feel like it)

Common issues to check:

Trials

• [x] Are trials now unique between administrations?
• [x] Is exclusion info handled correctly? (look closely at how exclusion information is marked)

Trial Types

• [x] Check if the trial type IDs are created independently of administrations/subjects
• [x] Is vanilla_trial coded appropriately?

Stimuli

• [ ] If the images are on osf, make sure the image path contains the image path on osf
• [x] Make sure each row represents a label-image association
  • [x] the labels should be the words that the participants hear. For example, "apple" is okay, "red_apple_little" is wrong and was probably erroneously extracted from the file name
• [x] Are there items in the imported dataset not mentioned in the paper?
• [x] Are distractors represented correctly?
  • Special explanation for distractors: If an item only ever appeared in distractor position, it still gets its own row. The label is typically the label given to the image in the experiment description (e.g., "the distractor was an image of a chair"). If there is no obvious label provided in the experiment design, leave label blank.

Subjects

• [ ] Does CDI data follow the new aux_data format?
• [x] Age rounded correctly? (decision: we do no rounding)

General

• [x] Double-check the citation and update it in the dataset table and make sure it’s consistent with the peekbank datasets google sheet: peekbank datasets
• [x] Are there any TODOs left in the code - resolve/double check
• [x] Review (or add) a ReadME (example)
  • [x] Make sure any TO-DOs or other decision points in the comments of the code are documented in the ReadMe AND removed from the code to prevent ambiguity
• [ ] General data sanity-checking (summary output helps here)
  • [ ] is there are the general numbers (e.g. # of participants, # of stimuli, average trials per administratoin) in the summary consistent with the paper? aoi_timepoints are hard to gague, but a super small number is probably bad
  • [ ] is the subject summary (age, sex distribution) approximately consistent with the paper? (note that it is not surprising if it is not identical - often we have a slightly different dataset and are not trying reproduce the exact numbers)
  • [ ] is the target side distribution skewed towards one side?
  • [ ] any weird trial durations?
  • [ ] do the cdi rawscore numbers match the instrument and measure?
  • [ ] is the exclusion % and the exclusion reasons sensible? (bearing in mind that we only have exclusion info for some datasets)
  • [ ] Inspect the timecourse and accuracy plots/output at the end of the import:
    • [ ] Compare timecourse patterns with paper (as best as possible)
    • [ ] Does the timing seem right? (accuracy spike later than PoD might be sensible, earlier is suspicious)
    • [ ] (if multiple conditions) Does the number conditions make sense in the context of the paper?
    • [ ] (if multiple conditions) Are the overall accuracies for conditions vastly different in a way not explained by the paper?
    • [ ] Any odd item-level patterns?
    • [ ] Any odd subject-level patterns?
  • [ ] Any large (unexpected) discrepancies between data reported in paper vs. data in the imported dataset?
• [ ] After checking everything and rerunning the script: Upload the output to osf

[vboyce]

@alvinwmtan Is the raw data for kremlin on the peekbank osf? (because I'm not able to download it and not seeing it there?)

[alvinwmtan]

sorry, just uploaded. should be there now

[vboyce]

Thanks! Might I also have "demo_comp.Rda"?

[alvinwmtan]

ah yes sorry, forgot i had to copy it over from sander-montant_2022

[vboyce]

sorry to keep asking for files (@alvinwmtan), but I can't find target-distractor-pairs.csv trial_info_fr.csv trial_info_sp.csv and also the images if we have them

[alvinwmtan]

my bad; added them. there are wmv files that could be screenshotted to grab the images but i haven't done so—feel free to!

note also that trial_info_fr.csv and trial_info_sp.csv were constructed by me rather than from the original raw_data; it is possible but just somewhat annoying to programmatically pull together all the different pieces of info needed.

[vboyce]

Images could be pulled via screenshot readme mentions that CDI data for the montreal subset exists (but has not been imported yet) (and DVAP -- another vocab measure)

[vboyce]

@alvinwmtan I'm getting a validation error because of an aoi region that is all NA's -- it looks like this is coming from the fact that one of the datasets has aoi coordinates and the other doesn't (hand coded) and NAs get added to that one in a bind_rows? Does that sound right? / Do you have ideas for fixing?

[alvinwmtan]

this is correct (montreal has AOIs and princeton doesn't). somehow i didn't get a validation error when i ran it though? not sure why this is popping up

[vboyce]

so the validation issue seems to not be about the NAs and more be about that there are multiple regions in the aoi_region_set, but there's only 1 in the trial_types? which I think I've traced back to

 if(!is.na(data$l_x_max[[1]])){

    trial_types$aoi_region_set_id <- 0

in the digest_data function at https://github.com/peekbank/peekbank-data-import/blob/33e750103cb1249b35daa01c6a7e9de1da5a6749/helper_functions/idless_draft.R#L284C1-L284C39.

Commenting out that line does seem to "fix" things, but I don't know what it's purpose is -- @adriansteffan what is this line supposed to be doing?

[adriansteffan]

so the validation issue seems to not be about the NAs and more be about that there are multiple regions in the aoi_region_set, but there's only 1 in the trial_types? which I think I've traced back to

 if(!is.na(data$l_x_max[[1]])){

    trial_types$aoi_region_set_id <- 0

in the digest_data function at https://github.com/peekbank/peekbank-data-import/blob/33e750103cb1249b35daa01c6a7e9de1da5a6749/helper_functions/idless_draft.R#L284C1-L284C39.

Commenting out that line does seem to "fix" things, but I don't know what it's purpose is -- @adriansteffan what is this line supposed to be doing?

The line is supposed to remind me to read my code more carefully before committing. I removed it, thanks for the catch!