Challenge submission: XXL-logistics_repo-challenge

[MertenNefs]

Rbanism Reproducibility Challenge submission

The submission consists of the following steps:

1. Fill in the project information in the form below;
2. Submit this issue by clicking on Submit new issue without deleting the text below;
3. Fill in the (self-)assessment checklist generated by the issue;
4. Submit a separate comment in the same issue with a statement on the level of reproducibility of your project. Make sure you elaborate especially on the challenges you encountered and the boxes that are not ticked in the checklist -- word limit: 300 words;
5. Add another comment letting @cforgaci know that the submission is ready for assessment.

Project information

• Project title: XXL-logistics_repo-challenge
• Author(s): Merten Nefs, Jeroen van Haaren & Frank van Oort
• Type of work: PhD thesis paper
• Status: finished but not published
• Link to the GitHub repository of the project: https://github.com/MertenNefs/XXL-logistics_repo-challenge/

(Self-)Assessment Checklist

The following checklist is used for the assessment of the level of reproducibility of your project. Note that the checklist serves as a tool to help you think about the reproducibility of the data analysis. Although many of the questions can be thought of as having a yes/no answer, we encourage you to see the questions as being open ended with the real question being, "What can I do to improve the status of my project on this bullet point?" With that in mind, you will never get 100% of the bullets right for your project, but you will always be improving.

Organization

• [x] Are all data, code, results, and documentation housed within a monophyletic (i.e., within one root) folder structure?
• [x] Is the project folder structured to separate your data, code, documentation, and results?
• [ ] Can the project be run from the project's root folder?
• [ ] Is the project under version control?
• [x] Do files use a consistent naming scheme that indicates what they contain?

Workflow and automation

• [x] Is the workflow clearly documented (with or without code)?
• [x] Is the workflow scripted, i.e., using code?
• [x] Is the code well documented?
• [x] Is the code modular, i.e., does it use functions?
• [ ] Are unit tests available for the code?
• [ ] Does the repository make use of continuous integration tools to insure internal reproducibility?

Documentation

• [x] Is there a README file that indicates
  • [x] the purpose of the project,
  • [x] who to contact with questions,
  • [x] a map of the directory structure, and
  • [x] a description of what software and hardware are needed to reproduce your workflow?
• [x] Are there README files in each folder describing the contents of the folder, and how they were acquired/generated?
• [x] Is there a CITATION file that tells users how to site the project, data, and code?
• [x] Does the project have a LICENSE file?
• [x] Is the project's repository publicly available?

Publication

• [ ] Are papers and reports from the project generated using literate programming tools so that results are not hard-coded?
• [x] Does the project have a persistent identifier?

Data

• [x] Is the data included or linked, with instructions on how to obtain the data?
• [ ] If data is not included, is this because it is not necessary or generated as part of the project?
• [x] Are your raw data (if any) and processed data files separated?
• [x] If data is not open:
  • [x] Is it accessible via a protocol?
  • [ ] Is there synthetic data provided so the project can be run?
• [ ] Is citation information available for the data?
• [x] Does the data use open formats, such as CSV and TXT?

Software

• [ ] Is there a list of dependencies?
• [ ] Is a container available to run the project?
• [ ] Are the version numbers of every external application used in the process?
• [ ] Does the project use only open software?

This list was inspired by the more extensive Checklist Questions to stimulate thought about a Project's Reproducibility.

[MertenNefs]

Although reproducibility has been an aim of the submitted project, the repo-challenge checklist is useful to improve several parts of the research repository structure, and helps reflect on future projects. In general terms, the project attempts to be as transparent as possible, providing all the code files, describing the analysis workflow and using a fixed DOI. Some concepts might have been used more extensively, such as the use of functions in the code - they are there specially to avoid the repetition of large code chunks and the easy changing of data years for instance, but could have been used more. The nature of the project, being a collaboration with a partner who uses proprietary software, and the necessity to use proprietary microdata, makes it impossible to construct a repository that can be run instantly by others. The code would report missing data in the data folder. What seems viable, is to show as much as possible what all analysis steps were. Providing sample data is an interesting idea, which we might use in a next project - although I'm not sure if even parts of the Lisa data could be distributed legally. Maybe in a next project, trying to avoid proprietary software altogether would be worthwhile. The software generated does not seem so complex as to justify the construction of a container, with list of dependencies, version numbers etc., but maybe building and using it from the beginning would make more sense. The use of reports from literate code such as R-markdown would be interesting for intermediate results next time. In my view it does not have the potential to easily generate journal-ready documents with referencing, notes etc. like word-processors do.

[MertenNefs]

@cforgaci , the submission is ready for assessment! Best regards,

[cforgaci]

Excellent, thank you @MertenNefs! Please leave an update in this issue if you make any changes until the extended deadline.

[MertenNefs]

small adaptation, box ticked on readme files

[cforgaci]

@MertenNefs, could you please invite @hugoledoux, one of our reviewers, to your repository?

[cforgaci]

@MertenNefs, please also invite @rusne, the second reviewer, to your repository.

[MertenNefs]

Hi @cforgaci , done!

[rusne]

Hi @MertenNefs could you add me once again? I was late with accepting the invitation and it has expired in the meantime. Sorry about that!

[cforgaci]

Hi, @MertenNefs. Can you please email me at C.Forgaci@tudelft.nl, so that I can communicate with you about the next step via email?

[cforgaci]

Challenge ended. Congrats on the prize! 🎉 Closing this issue.