Reproducibility Reviews AGILE 2020

nuest commented 4 years ago

There are six papers that underwent a successful reproducibility review as part of the AGILE conference 2020. See the proceedings at https://agile-giss.copernicus.org/articles/1/index.html (including information about the review process). See also the report for the 2020 review at https://docs.google.com/document/d/1JHCQV7GP3YkKwp0Nii3dt3p3Y45hU56Xz2cr-xJVz34/edit#heading=h.bn7c2s1mem9m

The reports are published as components of a parent OSF project: https://osf.io/6k5fh/

Certificate: 2020-018 to 2020-023

Here are the report titles, DOIs to the report repos, direct link to the report, and respective certificate IDs, and the GH repo:

Reproducibility review of: Integrating cellular automata and discrete global grid systems: a case study into wildfire modelling
- https://doi.org/10.17605/OSF.IO/ZTC7M
- https://osf.io/bmhwe/
- 2020-018
- https://github.com/reproducible-agile/AGILECA (fork), copy in report repo
Reproducibility review of: What to do in the Meantime: A Service Coverage Analysis for Parked Autonomous Vehicles
- https://doi.org/10.17605/OSF.IO/5SVMT
- https://osf.io/qsp72/
- 2020-019
- Code/data on OSF at https://doi.org/10.17605/OSF.IO/PWYJN, copy in report repo, can also fork on OSF (https://osf.io/52vzu/?view_only=)
Reproducibility review of: Window Operators for Processing Spatio-Temporal Data Streams on Unmanned Vehicles
- https://osf.io/7hcxv/
- 2020-020
- Anonymous repo: https://figshare.com/s/cc758d056c8c6f193e52, original code/data not in the report repo
Reproducibility Review of: Comparing supervised learning algorithms for Spatial Nominal Entity recognition
- https://doi.org/10.17605/OSF.IO/SUWPJ
- https://osf.io/m6yhg/
- 2020-021
- https://github.com/reproducible-agile/Spatial-nominal-entity-recognition (fork)
Reproducibility review of: Extracting interrogative intents and concepts from geo-analytic questions
- https://doi.org/10.17605/OSF.IO/7XRQG
- https://osf.io/98pag/
- 2020-012
- https://figshare.com/s/b3f8b0834ca63b6c5d60 and copy in the report repo
Reproducibility review of: Tracking Hurricane Dorian in GDELT and Twitter

Decisions

[x] Decide how to handle the original repository link
- the original code and data are in 3 GH repos, 2 Figshare projects (no DOI, one anonymous), and one OSF project; a few report repositories on OSF have copies of the code/outputs
- Decision: we can just use these links in the repository: field in the codecheck.yml, we will use the reproducible-agile fork if available on GitHub, the Figshare links and the OSF link otherwise
[x] Decide where to store the codecheck.yml file
- straightforward for GitHub, where we could put them in the forks
- At the same time, we should start to distinguish between repos in the codechecker's org and other repos, e.g. by prepending the org, reproducible-agile/AGILECA, and assuming codecheckers/ if none is mentioned. Might be worth updating the repo info now and prepend gtihub:: and gitlab:: anyway > see #26
- in other cases, where we cannot "fork" that easily (Figshare), and even if we can (OSF) making a fork now is too complicated) > we need another solution
- idea: the codecheck.yml file alwas is in the repository that is listed in the register, which is not the code/data repository of the paper, but by definition the place where the codecheck.yml can be found. Implement support for other repositories, see #26, so that a codecheck.yml file may also be stored on OSF, or even Zenodo (#27)

Tasks

[x] Create codecheck.yml files for each check
- The OSF project DOI is used for the reference for the report
- May store copies of the yml files in https://github.com/reproducible-agile/reviews-2020/tree/master/reports (note that folder has subfolders for each reproducibility report which do NOT match the certificate IDs!)
- There is no requirement to have a manifest, and the reproduced files are not always included in the deposit/repository
[x] extend register management scripts to
- [x] identify a Repository as not being a GitHub repository within the codecheckers organisation - see #26
- [x] implement retrieval of codecheck.yml using the osfr package
[x] put the codecheck.yml in the right places

nuest commented 4 years ago

@sje30 are you fine with the following shortcoming(s)?

The codecheck.yml will have a placeholder manifest, because the manifest was not a requirement in the AGILE reproducibility review - see example below. Alternatively, we could relax the requirement in the specification (i.e. make manifest not mandatory), but I feel like the below is more expressive.

---
version: https://codecheck.org.uk/spec/config/1.0/

manifest:
  - file: NA
    comment: "The AGILE 2020 Reproducibility Review did not include manifest documentation, see https://github.com/codecheckers/register/issues/25"

paper:
  title: "Integrating cellular automata and discrete global grid systems: a case study into wildfire modelling"
  authors:
    - name:  Majid Hojati
      ORCID: 0000-0001-7350-0055
    - name:  Colin Robertson
  reference: https://doi.org/10.5194/agile-giss-1-6-2020

codechecker:
  - name: Daniel N.
    ORCID: 0000-0002-0024-5046

report: https://doi.org/10.17605/OSF.IO/ZTC7M
summary: |
    The check was straightforward as all material was provided and documented well, but computations took about 3 hours to run.
repository: https://github.com/reproducible-agile/AGILECA
check_time: "2029-07-13 11:32:00"
certificate: 2020-018

The certificate IDs are not included in the actual report document.

sje30 commented 4 years ago

hi Daniel, I'm happy with the workaround about NA for manifest. I'd prefer keeping the manifest compulsory for what we do normally. This is a nice edge case that we can refer to in the paper as "adapting" to different reviews.

sje30 commented 4 years ago

so, I say go for it, and fold it in!

codecheckers / register

Reproducibility Reviews AGILE 2020 #25

Decisions

Tasks