Campaign: Pre-registration (encourage authors to complete)

[dlholf]

Rationale Pre-registration requires researchers to specify their planned research protocol, objectives and hypotheses, and plans for data collection and the confirmatory analysis to be conducted in order to test the stated hypotheses prior to beginning a study. This means that researchers are accountable to producing the research as intended and will help to protect against practices such as “hypothesising after results are known” or “p-hacking”, which can inflate positive results in scientific research.

Pledge By joining this campaign, you pledge to pre-register all one of your conducted studies via one of the pre-registration formats specified by the Center for Open Science (https://help.osf.io/hc/en-us/articles/360019738834-Create-a-Preregistration#Go-to-the-OSF-Prereg-Challenge-landing-page). This does not need to be hosted on the Open Science Framework (e.g., it could be on a site like AsPredicted.org). There simply needs to be a time-stamped, publicly available document that can be verified as existing before the study was conducted.

Issues to work out (all sorted, see comments below): 1. I wrote this as all studies conducted since the pledge activates, but perhaps some buffer time is needed so that studies just about to launch at the time won't be suddenly help back by a pledge activation. Maybe a grace period for pledgers to adjust?

2. Should this focus instead on hitting a target number of pre-registered studies (instead of creating a pre-registration 'habit')? It is a smaller step, but maybe we need the smaller step to allow people to committing to pre-registering X studies in a time frame before making a larger pledge.

3. Threshold — Your pledge will activate when 100 researchers have joined the campaign. --> what is a reasonable threshold to set?

4. Duration—Your pledge will expire after 2 years, but you are encouraged to renew it! --> is expiration a good idea for this? This is also related to point (2) about whether the commitment should be all studies in the specified duration or X number of studies pre-registered in the specified duration.

5. Anonymity?—Your pledge will be public once the threshold is activated. --> does this seem reasonable?

Note: We'll be developing the campaign text using HackMD here

[CooperSmout]

Thanks for posting @dlholf!

Should this focus instead on hitting a target number of pre-registered studies (instead of creating a pre-registration 'habit')? It is a smaller step, but maybe we need the smaller step to allow people to committing to pre-registering X studies in a time frame before making a larger pledge.

Have been thinking about this lately. Making it a habit would certainly be a bigger statement, but then could also turn some people off, e.g. if unsure supervisor might resist etc. The other end of the extreme could be to ask for just a single preregistration in a 2 year period? I think I lean toward this end, just because its easy and tangible for people to commit to, and doesn't stop people from doing more than that. I talked to a friend last night who said they would commit to such a pledge -- so that's a start ;)

Maybe we need to ask around a bit and see what people would be willing to commit to? Would be great if we can get a 'mini critical mass' ready to sign when it goes live, so that we can build momentum quickly -- people seem more willing to commit to a sure thing after all. I started a google doc to collect expressions of interest for the RR campaign (#8), could do something similar here once the details are clearer.

[AnthMHarris]

I think it may be helpful to specify the situations in which pre-registration is expected, or alternatively to specify the preregistration options for different kinds of studies. From the conversations I've been having, people are finding the push to preregister everything unrealistic because they dont realise they can preregister exploratory work.

Is the pledge aiming to have people preregister all of their work, or only their confirmatory studies?

[dlholf]

I wondered about having different levels of this—at the basic level, go for one pre-registered study in the next two years. At the most committed, pre-register everything. And in between that, pre-register all confirmatory. It's a bit of a slow habit-building exercise I guess. I'm not sure what is the best way to present that in practice. Only put out the most basic at the start, then put in a new pledge for people to re-commit to? Or put all three and let people pick what they can commit to?

I think a good motivator would be realising after doing it that it's easy and has its rewards (for example, if reviewers are favourable about seeing a study has been pre-registered). Though this is a culture change that needs to happen, and I'd like to think would happen it is enough of a norm that we start to question why a study is not pre-registered.

[CooperSmout]

Is the pledge aiming to have people preregister all of their work, or only their confirmatory studies?

Good point, we'll definitely need to be clear about this in the campaign description. But also, I think this becomes a moot point if we're just asking people to complete a single preregistration (see below), because then people can just work it out for themselves which study they want to preregister.

I wondered about having different levels of this—at the basic level, go for one pre-registered study in the next two years. At the most committed, pre-register everything. And in between that, pre-register all confirmatory. It's a bit of a slow habit-building exercise I guess. I'm not sure what is the best way to present that in practice. Only put out the most basic at the start, then put in a new pledge for people to re-commit to? Or put all three and let people pick what they can commit to?

I'm also not sure about this, but suspect we'll have the most success by defining a single thing for people to commit to, rather than giving them multiple options. Every time we give people a decision to make, we increase the chance they bounce without making a pledge. Point in case: one person who I know is keen on the project told me they hadn't signed campaign #5 because they really wanted to think about it and get the settings right, but then hadn't managed to return to the site and made a pledge. Admittedly the OA campaigns are much more complex than what we have here, but I think the same principle applies -- keep it simple as possible. So to me it's a choice between: (a) preregister one study in the next X years (which means we don't need to specify whether confirmatory/exploratory), or (b) preregister all confirmatory studies in the next X years (as per @anthmharris' point)

(cc @angier2310 if you have any thoughts/preferences)

[MalikaIhle]

Threshold — Your pledge will activate when 100 researchers have joined the campaign. --> what is a reasonable threshold to set?

Here I think it could be 100 people from your field (thinking of the Tier 2 from such list: https://docs.google.com/spreadsheets/d/1APu3Hi5UtexvISTGkk1D1uZK-7Efjpj9_sQBq_dH38I/edit#gid=222728450) which means you would need to ask people to pick their field from a list.

Duration—Your pledge will expire after 2 years, but you are encouraged to renew it! --> is expiration a good idea for this? Anonymity?—Your pledge will be public once the threshold is activated. --> does this seem reasonable?

both yes

By joining this campaign, you pledge to pre-register all your conducted studies maybe we need the smaller step to allow people to committing to pre-registering X studies in a time frame Is the pledge aiming to have people preregister all of their work, or only their confirmatory studies? (a) preregister one study in the next X years (which means we don't need to specify whether confirmatory/exploratory), or (b) preregister all confirmatory studies in the next X years

Even if I can see the value of preregistering qualitative research just for the sake of transparency and reflection of peoples positionality, for someone involved with both type of research it would primarily be important for them to preregister quantitative research and especially confirmatory one, so I would also not suggest to preregister all. And not to start polarizing people from different disciplines, I like the idea of not mentioning it like with (a) preregister one study in the next X years . However, I do wish people who do mostly experimental work would preregister all their experiments... so I do like (b) preregister all confirmatory studies in the next X years.... I think both could be offered, but probably the first one matters most. The first one will be most appealing to people who have never written a publication, the second will be for people who are already convinced by it and have already written one, and who will probably do this anyway??

[dlholf]

100 people from your field is a good idea!

I think both could be offered, but probably the first one matters most. The first one will be most appealing to people who have never written a publication, the second will be for people who are already convinced by it and have already written one, and who will probably do this anyway??

I wonder if those who would probably do it anyway would still be willing to join the 'pre-register one study in the next X years' pledge as a way of showing people who are new to pre-registration that this is popular and accepted?

In which case we probably do want (a) as a simple starting point to spread the practice beyond the converted.

[CooperSmout]

Duration—Your pledge will expire after 2 years, but you are encouraged to renew it! --> is expiration a good idea for this? Anonymity?—Your pledge will be public once the threshold is activated. --> does this seem reasonable?

both yes

agreed :)

I wonder if those who would probably do it anyway would still be willing to join the 'pre-register one study in the next X years' pledge as a way of showing people who are new to pre-registration that this is popular and accepted?

In which case we probably do want (a) as a simple starting point to spread the practice beyond the converted.

This is how I feel about it. There's nothing stopping people from going the next step, and preregistering all of their studies, but if we start out by asking too much we might lose some people. I guess the alternative could be true too though -- people might look at it and ask "What's the point in pledging to preregister one study? I already do that (or I'm planning to do that anyway)". So if we go with this option, we'd have to make it clear that this campaign is just the beginning, and aims to build a supportive community with bigger goals in the future.

Alternatively, we could include an additional checkbox in the pledge, where people pledge to prereg all studies. But this would mean extra development of the platform, and also at the end of the day wouldn't mean much because we'd still just be counting the number of pledges.

100 people from your field is a good idea!

I like it too. We should be able to categorise people using their previous publications in ORCID, or alternatively could get people to fill in a dropdown box. We actually built the platform to have a dropdown box in the first place (currently hidden), so might not be too hard to re-implement. But the tricky part would be showing people how many pledges there are for each field, because the platform is only set up for a single number to be shown on each campaign page (the total number of pledges). I'm trying to minimise development work at this stage until we learn more about what features are necessary, and so was thinking it best to leave these early campaigns as either (a) field-general, or (b) pick a single field to target (and put this in the campaign description). But if you think it's an important feature, I can raise it with the platform developers to see how much work would be involved?

[MalikaIhle]

I think it would be nice to have the total number displayed for the pledge (across disciplines) but actually activate the pledge per discipline (with a drop down menu if this can't be done automatically with ORCID - and perhaps it is best to let people pick the field for which they want to activate their pledge for people spanning several disciplines?)

[CooperSmout]

I think it would be nice to have the total number displayed for the pledge (across disciplines) but actually activate the pledge per discipline

Good to know. I was thinking that people would prefer to see how many pledges there have been in their field, but perhaps that's just me. In any case what you suggest is the easiest given the current state of the platform, so it makes sense to go with that and build in field-specific metrics down the track if/when required.

perhaps it is best to let people pick the field for which they want to activate their pledge for people spanning several disciplines

Great point. At present, the field variable is built in at the user level, rather than the campaign level, so unfortunately must be the same across all campaigns (i.e. not possible to pick psychology for one campaign and neuroscience for another). But I don't think this is a big problem for now, because we'll just be uploading one campaign at a time, and we can re-assess down the track when we get a clearer picture of who our users are and what features they want to see.

[CooperSmout]

I like the idea of not mentioning it like with (a) preregister one study in the next X years . However, I do wish people who do mostly experimental work would preregister all their experiments... so I do like (b) preregister all confirmatory studies in the next X years.... I think both could be offered, but probably the first one matters most. The first one will be most appealing to people who have never written a publication, the second will be for people who are already convinced by it and have already written one, and who will probably do this anyway??

I might be coming back around to the idea of having different pledge 'levels', providing we do it in a way that doesn't confuse people. One thing I'm conscious of is offering people a reason to join the campaign, rather than just carry out the action by themselves (@bnosek suggested this a while back). I think a real strength of FOK is the chance to connect with a community of like-minded individuals, many of whom have experience with the behaviour in question. So this got me thinking -- what if we set the default pledge to 'preregister one study in the next X years', but give people an opportunity to 'level up' and become 'mentors' in the community? Mentors should have experience carrying out the behaviour in question (preregistration) and be available to advise newcomers if they run into troubles (or be paired with newcomers in their field?). This could be a powerful drawcard by showing people that there's a supportive community ready to help them carry out their pledged action. And since mentors should already be familiar with preregistration, we could get them to pledge to preregister all of their (confirmatory) studies rather than just one, as a sign of their commitment?

I know this goes against what I said about keeping things simple, but wondering if it might also help motivate those people who are already familiar with prereg to join our campaign, as it gives them recognition for their skills and place in the community. If it helps, we could offer them a special 'mentor badge', which recognises their contribution to the community, and extra points on the community 'leaderboard' if and when this becomes a thing :)

We could set this option up when people pledge (e.g. via a pop-up box), but I'm also thinking that we could keep things simpler in these early days by just mentioning this as an option in the campaign description and then email people with more information after they pledge.

[dlholf]

Following up from a chat with @CooperSmout today, here is a summary and actions we'll take to move this campaign forward:

• Keep the text of the pledge simple, focusing on what and why to sign up.

• A global why pledge link (or possibly later, if functionality allows) could be added so people can see why pledging is important, but this would apply to all campaigns, so the pledge text should focus on why this campaign in particular.

• Start with pledging to pre-register one study in the next year (we forgot to speak about what X should be, but I think one year should be enough time to make just one pre-registration?)

• Something to test could be whether positioning a pledge with gain-framed reasons (e.g., providing support for the movement) vs. loss-framed reasons (e.g., avoiding being on the 'wrong' side of research history) is better at converting interest to pledges. We'd be able to learn better from this on how to craft an effective pledge. If the site developers can manage split testing easily, we could try presenting different versions randomly to site visitors (would this be by IP address?); an alternative is to amend the text daily to the different versions, though there are many potential confounds such as people who visit the site more than once, and the inability to effectively randomise, limiting conclusions.

• To keep things simple for signing up to the pledge, we should focus just on the basic pledge, and then email pledgers offering for them to take on a mentor role and/or 'level up' their pledge to a higher commitment voluntarily as part of their pledge activation. (And we should track uptake of this!)

Actions:

• I'll start drafting the pledge text to include on the website (with gain and loss versions so we can decide how we want to proceed with that)
• We'll include a link to an editable pledge document at the top of this thread (thanks @CooperSmout !)

[CooperSmout]

The other thing to mention is that we've decided to make this campaign the next to go live on the website. I'm still interested in developing a peer-review protocol to assess other campaign proposals, but in lieu of that being developed (and advice I've been given) think it makes sense to just go ahead with this campaign and then look to develop a peer review protocol later.

• A global why pledge link

I started this document to collect ideas for reasons why people should pledge (click the 'Edit on HackMD' button for easy editing)

• Start with pledging to pre-register one study in the next year (we forgot to speak about what X should be, but I think one year should be enough time to make just one pre-registration?)

I think the general consensus above was 2 years (Malika and I agreed with your original post)? I agree that one year should be enough time, but as you said above we should give people some leeway after their pledges go live (in case they've just moved jobs/country etc) and so I think 2 years seems like a less confronting option. My bet is that everyone will do it in the first year anyway, but will be interesting to see.

• We'll include a link to an editable pledge document at the top of this thread (thanks @CooperSmout !)

Have added that to the first comment, but can also find it here for easy reference :)

[CooperSmout]

Here I think it could be 100 people from your field (thinking of the Tier 2 from such list: https://docs.google.com/spreadsheets/d/1APu3Hi5UtexvISTGkk1D1uZK-7Efjpj9_sQBq_dH38I/edit#gid=222728450) which means you would need to ask people to pick their field from a list.

100 people from your field is a good idea!

I like it too. We should be able to categorise people using their previous publications in ORCID, or alternatively could get people to fill in a dropdown box. We actually built the platform to have a dropdown box in the first place (currently hidden), so might not be too hard to re-implement.

So on further thought, I think it will be easiest for this campaign if we ask people to self-identify their field using a drop-down box, rather than categorise using ORCID profiles (which requires extra coding/testing). I'd still prefer to use the Fields of Research taxonomy (as used in Dimensions) rather than the BePress taxonomy, unless there's a good reason not to @MalikaIhle ? The main reason is that this leaves open the option to categorise people/papers automatically down the track, as per the original intention for the OA campaigns (and we can also validate any code we write to do this using people's self-chosen fields). But please let me know if you have any thoughts on this!

[MalikaIhle]

I think it is best to leave open the option to categorise people/papers automatically down the track, as you suggest!

[air2310]

Had a skim read of the above conversation, but there was a lot of prose. What are the key things you wanted help making decisions on/ where are you at with launching the campaign? :)

Re: the level of commitment you're asking people to make to pre-registering. One avenue you could take, instead of having to think of every possible exception to why someone might not be able to pre-register a specific study, is to ask people to commit to pre-registering all upcoming studies unless there are extenuating circumstances. This puts the onus on the individual researcher to know the exceptions for their own field. I imagine if people are making the effort to sign this in the first place, their hearts are in the right place, and they won't be looking for ways to wiggle out of it.

[dlholf]

What are the key things you wanted help making decisions on/ where are you at with launching the campaign?

We've moved into developing the campaign text in HackMD, and were last looking into whether it was possible to test if different framing of the reasons to pledge is more effective at getting support. @CooperSmout may have other updates on the technological side of implementation?

I imagine if people are making the effort to sign this in the first place, their hearts are in the right place, and they won't be looking for ways to wiggle out of it.

I agree that people will have the right motivations. The complex issue may be when collaborative work crosses disciplines though—would that end up being an exemption?

Our current draft goes with pre-register one study within the next two years so avoids the issue of when something is exempt or not. We wanted in addition to give the option as well for those more advanced in pre-registration to 'mentor' those just starting out with it.

[CooperSmout]

Yep, the main parameters of the campaign are worked out now, so the main obstacle to moving forward is the platform development. I've just emailed @ZoranPandovski to ask if he will have time over the next few weeks to take a look at it, as it would be great if we could use the upcoming AIMOS conference to launch this new campaign (3-4 Dec). Either way we might have to scale back our expectations on what will be possible (e.g. the different framings might be too hard to implement). Will keep you posted.

In the meantime, if anyone here is interested we could start working on marketing materials for the campaign launch, e.g. social media/blog posts etc. There's a list of places we will target here and a few prior posts here and here

[CooperSmout]

Hi all, just an update on the development side of things. The custom python platform that we've been running until now is proving difficult to work on or find support for, and so I'm considering moving to a much simpler platform that lets us get the campaign out there and start collecting pledges/data/feedback sooner rather than later. I'll be presenting FOK at AIMOS in a couple weeks (3-4 December) and think it's too good an opportunity to miss. At the end of the day, we could build the perfect website but if no one is interested in the campaigns, we'd have wasted all that effort.

So to get things rolling, I've created a Github pages website today using Beautiful Jekkyl. It's still a work in progress but thought I'd post it here to see if anyone has any preliminary thoughts. If you click on the 'Pre-registration challenge' post, you'll see some of the text that @dlholf drafted (just the positive framing for now, as we won't be able to test multiple options) plus a Google form that I've embedded in the page. Unfortunately there's no simple way for users to login with ORCID using this setup, but I think that's a worthwhile sacrifice for the sake of moving things ahead. My plan is to enter people's details to the database manually when they pledge, at which point I'll be able to check if the people who registered are legitimate. Any thoughts welcome.

And it's my first time using Github pages so if anyone has experience let me know (currently trying to work out if I can host the website within this 'community' repository, rather than a [separate repository]... without breaking things!).

[dlholf]

Website looks cool! A few thoughts:

• Should we call it a pledge instead of challenge? (Because I know CoS had a preregistration challenge at some point, and we don't want to confuse the two.

• The pledge page has quite a large header. Maybe worth shrinking that so people can see the pledge details straight off.

Join X researchers who have taken the pledge!

• Would there be a way to update X to reflect the number of sign-ups currently available?

• We probably need to amend this since there is no ORCID login currently.

Your pledge will activate when X people from your field (as determined by researcher field in ORCID) have joined the pledge.

(Also sorry I omitted to update the X in the draft after our discussions!)

• Alternatively, we could ask the ORCID field in Forms to be a required answer. Thoughts on this?

• Would tick boxes for Research Field work better, so multi-disciplinary researchers can select more than one?

[CooperSmout]

Should we call it a pledge instead of challenge?

Ahhh so that's why 'Prereg Challenge' was sounding so nice in my head! :) I'll change it to pledge instead.

• The pledge page has quite a large header

True. Not sure how to adjust but can take a look.

Would there be a way to update X to reflect the number of sign-ups currently available?

I'm planning to update the count manually as pledges come in, since this website is just a static page and doesn't have access to the database (feel free to test the form out and become number 1 ;))

• Alternatively, we could ask the ORCID field in Forms to be a required answer. Thoughts on this?

Good question, I'm unsure about this one. On the one hand, requiring an ORCID means that we can check people are actual researchers and also that their chosen field makes sense, but on the other, we might scare some people off who don't have an ORCID (or more likely, don't have it ready to hand). I'm leaning toward making it optional at this stage.

• Would tick boxes for Research Field work better, so multi-disciplinary researchers can select more than one?

Tick boxes are listed vertically in Google Forms, so take up heaps of space... Maybe people with multiple fields could add their second field in the 'Other' section...?

[dlholf]

Tick boxes are listed vertically in Google Forms, so take up heaps of space...

Ugh, yes, that definitely is problematic. I wonder if we should just let people state their own fields, and we map it to classifications on ORCID? (Though I can't seem to find the relevant ORCID metadata at the moment.) But I'm ambivalent here, and I guess it's worth seeing how many self-input fields we get to start with so we have a better idea for future campaigns.

or more likely, don't have it ready to hand

Yes, that's probably the bigger issue! I certainly have to have mine on a sticky note for copy/pasting purposes because it's impossible to remember.

[CooperSmout]

I wonder if we should just let people state their own fields

Would be interesting to get a feel for how people see themselves/their fields. But the trouble is people would pick their own level for categorisation, and then we'd have to re-classify people according to some taxonomy anyway (which requires communication to check they're ok with the new category). I think for this first campaign, probably makes sense to just pick the categories a priori so that people know what they're getting themselves into, and we can adjust later based on feedback if people aren't happy

[CooperSmout]

Hi all, just closing this thread because this campaign is now live on the new website 🎉

Thanks for all your support in drafting this campaign. I ended up using the BePress taxonomy as per your suggestion @MalikaIhle, because the new website doesn't have ORCID integration anyway and so we won't be automatically classifying people anytime soon, and the BePress taxonomy seems like a cleaner taxonomy.

The comments on the website are now linked to a new Issue (#22), so I'll close this thread and reference it in the new thread in case anyone wants to see the discussion history. Thanks so much for all your input in getting this campaign out into the world!

[CooperSmout]

Ok so we've had our first pledge with a self-entry for the field of Education (Tier 1 in Bepress). I've since emailed the pledger and asked what they thought of the classification level, and they responded that they (a) guessed a field because the Google Sheet wasn't shared (I've fixed this now) and (b) think "there is a chance to get quite a few education researchers involved, but not that many under any specific subclassification".

This is an interesting point -- the chance of reaching threshold really depends on how advanced the field is AND how big it is. Something like Psychology (tier 2) is quite likely to get 100 pledges because it's relatively progressive, but something like Education might need to include all sub-disciplines to have any chance of reaching the target. I also just did a quick search on Dimensions, and it looks like Psychology (Bepress tier 2) is larger than the entire field of Education (Bepress tier 1), with 2.67 million and 1.76 million papers, respectively, so probably makes sense that both are options despite them being at different tiers in the taxonomy. For now, I've entered Education into the dropdown for now and will see if other Education researchers select that or enter something new.

I'm also wondering if I should remove the dropdown options that haven't been selected yet (I pre-populated the dropdown with ~35 Tier 2 fields -- mostly life sciences), so that people can enter whatever level they think makes sense and we can connect people up after the fact? Looks like I'll be emailing the first wave of pledgers a fair bit anyway, so not much more effort to check in with people who enter something new. I guess what I'm trying to say is: @dlholf you were right all along:

I wonder if we should just let people state their own fields, and we map it to classifications on ORCID? (Though I can't seem to find the relevant ORCID metadata at the moment.) But I'm ambivalent here, and I guess it's worth seeing how many self-input fields we get to start with so we have a better idea for future campaigns

;)

(although I missed your comment about ORCID metadata first time around, and not entirely sure what that means :))

[dlholf]

(although I missed your comment about ORCID metadata first time around, and not entirely sure what that means :))

It was just that I couldn't find a place on ORCID that identified what fields researchers were in. I could have sworn they had a list of fields, but I feel like I must have been hallucinating that!