Pre-Screening Pilot Test

[doomlab]

Pre-Screening Sampling Ideas:

• [x] @troettge add the https://doi.org/ to the doi column so it's clickable
• [x] Get a list of students together to work
• [x] We will pilot 50 or so papers with the students (will be double checked by Erin)
• [x] When we are satisfied we have good coding instructions, we will get 1000 papers and go through them.
• [x] As we get through them, we will stop when we have 600 (300 from each) coding to move on to real coding.
• [x] We will prescreen more papers if necessary to ensure a sample of 250 articles with linguistics data for the final sample.

[timo-b-roettger]

So there will be 10% of entries that have no DOI, just for expectation management. The list can be found here. Have a look @jvcasillas and @doomlab and lemme know if you think thats okay as is or whether you would like it to look different.

[jvcasillas]

Looks good to me. I'm still waiting to hear back from 3 students. I fear that I may not be able to squander volunteers during the summer, but we'll see.

[timo-b-roettger]

Sounds good. Don't worry, if you can't find anyone. If we need (wo)man power, I can see whether one of my students would be interested in a paid summer internship. Maybe you two could also think about a workflow for them when it comes to that. Like step-by step guide and also how they note down whether the paper is about language or not (another gform maybe?)

[doomlab]

@jvcasillas @troettge I am following up with a few folks since they could join and help code until we are done. The list you sent looks great. I am unsure I'll be able to make the next meeting because that's the day we are trying to look at houses but I will code the first 50 and take notes to pass along so we can make progress on both sides.

[doomlab]

Just finished downloading articles to review (a random 50 found in your prescreen list, I pushed a CSV with notes) and here's an interesting summary @troettge:

• I used unpaywall and found 14/50 that were open access directly from that button
• Then I looked on google scholar and found 20/36 were linked through scholar
• The rest I found through magic just for pilot testing (3 completely unavailable due to no magic or broken links)
• I was not connected to my university network while doing this, so that tells us the real accessibility without uni paywalls.

Other things:

• Two are automatically excluded due to not being in English
• ...more updates on the number of "useable" as I go, but thought this information might be useful.

[timo-b-roettger]

@doomlab hey thats fantastic. Maybe we should discuss the open access issue tomorrow during the meeting. Did you also check whether the papers are on language or not? (can't find that information in the .csv. Keen to hear about your experiences tomorrow.

[doomlab]

@troettge - not yet, that’s today’s goal while on a plane. We are looking at houses tomorrow so that schedule will determine if I could make it but I thought this information would be useful for the meeting either way. I’ll ping you when the language-or-not part is done too.

[timo-b-roettger]

@doomlab please don't stress about it and focus on the house hunt (exciting!!!). Thanks for these updates, that give us something to chew on for tomorrow!

[doomlab]

@troettge didn't make a lot progress on screening and definitely will be out seeing houses tomorrow. I will try to tag everyone in an update though when I have it soon.

In some good news, some students have contacted me! I have one that wants to start now, and I’ll schedule a meeting with them ASAP 👍

[doomlab]

Final summary for tomorrow's meeting:

• From older articles: (8/17) 32% nos
• From newer articles: (5/10/10) 20% unavailable, 40% no
• Overall: 10% unavailable, 36% nos, 54% yes
• Big question: literature studies