Build Next Generation Reprohack Infrastructure

[annakrystalli]

One of the goals is to make ReproHacks reproducible, ie develop customisable templates and tools so that hosts can run ReproHacks easily.

Problems with current setup

The current approach to event setup & delivery uses shiny and googledocs to compile a list of papers and collect feedback for the authors. This approach comes with limitations:

• Requirement to authenticate with googledocs
• Requirement to have a shinyapps.io account or access to a shiny server/institutional shinyapps.io account. Although it is easy to set up a shinyapps.io account, there is a cap on the amount of traffic to the app allowed in the free account and upgrading the subscription to run an event might be required (I had to do this for the first two OpenCon events I ran from my personal account).
• Currently, distributing feedback to the authors is not automated, although it could be set up as a script attached to the feedback googlesheet
• Overall, the materials feel like they are too spread out.

Solutions

Manage everything through GitHub:

Taking inspiration from Carpentries workshop website deployments for event sites, and rOpenSci management of package reviews to track reproductions and feedback to authors, we will be developing a system that is fully open and relies solely on GitHub for compiling and serving paper lists as well as tracking reproductions and participant feedback. Managing everything through GitHub also means we have a way of contacting authors through their with needing to publicise email addresses

Use pull requests for paper submissions

• pull request to add a YAML file that describes a paper
• Provide YAML file template
• Travis CI validates Pull Request YAML file.
• Explore easier ways to submit file (as any significant increase in complexity of procedure will reduce author interaction and submission levels. eg:
  • Use something like prose.io to provide a GUI for submission? (eg have a look at hackers.nyc file submission portal)
  • PRs could be reviewed and accepted by the organizers of the event or automatically merged once checks have passed: https://github.com/probot/ideas/issues/22
  • GitHub trick: how to prefill "create new file" form?: https://www.reddit.com/r/github/comments/3ty3g9/github_trick_how_to_prefill_create_new_file_form/

Make pages static and build after checks pass

• when merged Travis CI builds static site and serve as GH pages (or equivalent)

  This way we minimize the toolset and would not require Google Forms etc. Having everything in one place (one repository) would also simplify reuse for other events. Generate from a GH template and that's it: no other accounts and places to set up.

Use participant registration issues for author feedback

This helps maintain everything in the same place and can be handled by expanding the current GitHub issue template. Although the majority of authors submitting to the CarpentryConnect the fact that feedback will now also be open might put some people off initially. This makes a really clear and well enforced code of conduct, particularly with respect to treatment of authors even more important!

Use bots/tests to automate some processes?

• Check YAML for paper submissions: https://probot.github.io/apps/yamburger/
• Automatically merge PR after checks pass: https://github.com/dependabot or https://github.com/bobvanderlinden/probot-auto-merge
• More bots to consider on probot: https://probot.github.io/apps/

Key considerations

• Submission workflow needs to be as simple and as automated as possible so as to be able to compete with the simplicity of a google form.

[annakrystalli]

Pinging @bast. We can use this issue as a starting point for planning development. Feel free to edit summary or add more thought in this issue.

[bast]

This looks very good! Thank you!

I really like these points (summarizing them just to make sure I understood):

• Page is static and only needs to be forked or imported.
• Paper submissions/nominations are pull requests.
• Participants reproduce papers using issues.

Comments

Automatically merge PR after checks pass

For me this point is not so important. The ReproHack organizer looking through submissions and possibly giving feedback/help and clicking the green button is not a big hurdle and I think I like that a human is looking of a submission and not only a bot.

Also the page may get both "normal" pull requests to fix typos or some details and paper submissions and I think we don't need too much automation here.

Questions

• If reviews are issues, when are the issues closed? What "event" or who closes them? Or do they stay open forever?
• How can we keep track of ReproHacks which will import from our template? It would be nice to have a map of ReproHacks linking to their data.
• We could try that participants can nominate/submit papers, not only the authors (it is easier for us to reach participants than authors) but it would be good if we ask authors for an agreement to use their paper. If this is a good idea, how would this work in practice? On GitHub? Or outside of GitHub first?

[bast]

IIUC future ReproHack organizers would have to do these steps:

• import a web page from a template we provide
• activate Travis CI
• announce and communicate with future participants (by using links and good docs on the web page)
• review and merge paper submissions/nominations
• maybe do something with issues

[annakrystalli]

Plan updates: We have decided to address this issue as well as #6 #8 #10 & #11 by building a custom dynamic website for the project using Django.

Requirements

Forms for collecting data from/on:

• Organisers/hosts on Events
• Authors on Papers
• Participants on Reproduction reviews.

Website will serve:

• Event/Paper/Review lists
• Event/Paper/Review details
  • Event page will have custom paper list associated with event
• Review form to be designed around ReScience Reproducibility report with ability to auto-generate pdf

---

Benefits:

• django handles user authentication
• Data stored in sqlite3 db (good storage, easy querying)
• Easy definition of tables through models and generating forms
• Centralised storage of paper details that will hopefully grow continuoulsy.

I am planning on working on this on our RSE Sheffield Hacktoberfest and should have a prototype online by 2019-10-22.

[annakrystalli]

Help with Django skills welcome!!

[bast]

I can help out with that and brush up my Django ... But how will Reprohack organizers deploy the Django services in practice?

[annakrystalli]

So....my django knowledge is so limited that I don't even know what Django services are 😂.

But the idea is that once users have authenticated, they will be able to submit events or edit submitted ones. We will provide the standard text for an event as a default but they will be able to edit and customise it.

Here's what the draft events form looks like atm:

[annakrystalli]

I started by following this tutorial and slowly building it up to suit our needs: https://tutorial.djangogirls.org/en/django_admin/

[bast]

OK sorry my question was vague - what I meant is this: to run a Django site, the organizers will have to run a Python web server, either self-hosted or hosted somewhere. This can take some setting up. I only raised this because I would like to make sure that this is not a barrier for those who do not know how to set this up or do not have the resources.

[annakrystalli]

Ahhh ok got you!

So they won't need to recreate the site at all, all they would need to do is submit the details of their event and then they will have their own page on the main Reprohack site!

And for starters we'll be hosting the site on a University of Sheffield VM.

Does that make sense?

Sent from my iPhone

On 18 Oct 2019, at 11:11, Radovan Bast notifications@github.com wrote:

OK sorry my question was vague - what I meant is this: to run a Django site, the organizers will have to run a Python web server, either self-hosted or hosted somewhere. This can take some setting up. I only raised this because I would like to make sure that this is not a barrier for those who do not know how to set this up or do not have the resources.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[bast]

Got it - one site hosted in a central place that can be used for spin-off Reprohacks. That's great!