Taking pilot survey :-)

[chartgerink]

Hi,

Just took the pilot survey. For your context, I am a methodologist/statistician so I am also trained (somewhat) in survey methodology.

Page 1

• [ ] Is this an informed consent page? If so, there is no clear consent included.
• [ ] Do you want to share the anonymized data? If so, include that on the page so that you can freely do so.
• [x] I wouldn't include the link to the github repo in the final survey, considering most scientists will not be familiar (keep the survey as simple as possible; don't know if that was the plan all along anyway :-))
• [ ] Maybe you could provide some details on how the anonymization will take place? I noticed on page 3 that institution was requested; is that also anonymized?

Page 2

• [x] I guess this is just for testing purposes and not for final survey? :-)

Page 3

• [x] PhD candidate and PhD student mean two different things in the US I think, so maybe just include that?
• [x] How is this defined? "How many years have you been working in research?"
  • I can imagine you mean since getting PhD? That's what we usually ask
• [x] Is "methodologies" a word? Maybe "methods"?

Page 4

• [ ] Maybe good to also have this definition at the top at the later pages about Open Science?
• [ ] Maybe include the duplicate text from the answers in the question itself? (Q below)

"Based on what you know about Open Science, what are some of the positive aspects that come from practicing Open Science? (check all that apply) *

• [x] For same question, maybe include an option "I don't know about OS", given that it's forced response?
• [x] Is it an idea to flip around the positive-negative questions? I got the idea from it that the survey wanted to skew answers. You might even consider randomizing the order these are presented in.

Page 5

• [x] Maybe rephrase the options in the question to be more dense?

Please rate the importance of the following factors in choosing to publish your manuscripts as preprints *

• for example, "Having preprints read by more researchers in my field and receiving comments" (note the double-barrelled answer here) to "Preprints are read more"
• [x] Two final questions are forced response but seem optional

Page 6

• [x] The Likert scale is asymmetric, given that the left side has "not at all" and "slightly", while on the right side it says "very" and "extremely"
• [x] The option "It allows pre- and post-prints" is double-barreled and is not a good response option because of it
• [x] The response option "Having the Open Access fees paid by grant money or by my institution" is also double-barreled; might be an idea to just change it into "paid for" or something like that
• [x] Negatively phrased questions often cause problems (i.e., with NOT in them). The answers also contain it, causing the double negative problems.
• [x] Two final questions are forced response but seem optional
• [x] I was able to fill in text for the question asking a forced numeric response

Page 7

• [x] The response option "Open Access" in

  What access level did you set for these data? (select all that apply) is potentially confusing. Maybe rephrase to "freely accessible"

• [ ] Again the Likert scale
• [x] Two final questions are forced response but seem optional

Page 8

• [x] The definition of open source is quite short, which Richard Stallman will probably have problems with ;-)

  Open source refers to a computer program in which the source code is available to anyone for use and reuse. Do you mean open source or free and open source? Important to delineate because they mean different things (and you will have participants who know the difference, I guess).

• [x] This question is not forced but would need to be, no?

  Do you use open source software for data acquisition and/or data analysis?

• [ ] Again the Likert scale
• [x] Two final questions are forced response but seem optional

Page 9

• [x] Maybe remind the definition of Open Science at the top?
• [x] This question is a bit unclear; does it mean the respondent would have to pay for it or that the institution dissuades the respondent from taking the course?

  Would you take a program/course/workshop to learn more about open science, if you were to take it yourself and NOT have it supported by your institution?

• [x] Maybe add an option that the respondent doesn't need a course?

Page 10

• [x] no comments

Page 11

• [x] no comments

Page 12

• [x] no comments

[amiefairs]

Thank you so so much for this feedback! Regarding all your comments for the first page - for this pilot data we are not going to share anything, but you are completely right that we should explain that demographic data will be separated from other responses when the data are shared. They will be kept together for our analysis purposes though, so that we can have some more fine-grained analyses.

The comments you have given about adding more information to the information sections at the top of each survey section is great, we definitely need to improve on that. And we need to standardise which questions are optional and which aren't, and fix the bugs with being able to add responses we don't want (e.g. typing letters into a numerical field).

Thank you also for your advice about the double-barrelling of the questions. I have understood from it that there are two ways to go about this: 1) splitting the response into two with one 'barrel' each, or 2) collapse across 'barrels' when appropriate. Do you think that is a good way to go about editing?

[dasaderi]

@chartgerink Thank you so much for this amazing feedback!

[chartgerink]

You are welcome :-)

Double-barrelled questions are easiest to resolve by either splitting the question yes or to take a higher level as assessor (if available). Which is better depends on context, so I think you're the best judge of that in the end!

[dasaderi]

@chartgerink I'm finally getting to answering your comments one at the time. Thank you again for the awesome feedback and for making your comments a checklist!

• Do you want to share the anonymized data? If so, include that on the page so that you can freely do so. We do want to share the anonymized data and we will need ethical approval and to get a consent, but we don't have it yet. We won't share the data we are getting with this pilot survey. I agree, we will need to explain that well at the beginning.

• I wouldn't include the link to the github repo... We won't. This and the issue request is only for the sprint. :)

• Two final questions are forced response but seem optional We kind of want people to answer these. Do you think it's bad to force people to answer? ;P

• The Likert scale is asymmetric, given that the left side has "not at all" and "slightly", while on the right side it says "very" and "extremely" Good point! So what do you recommend? Would "of little importance", "moderately important", "important", "very important" work better?

• I was able to fill in text for the question asking a forced numeric response Yeah, that's google form for you. We are probably going to use another survey platform/software. Do you have any recommendation? Better if free.

• The definition of open source is quite short, which Richard Stallman will probably have problems with ;-) This is so true. I was hoping to get feedback on this. Thank you! Do you think we should be asking about both (after giving definitions at the beginning) or should we pick one (in which case I would go for open source)?