Code under 'Examples' in the help files

[wviechtb]

Hi all,

I would like to start an open/transparent discussion about the following issue:

At the end of the help files (in the Examples section), there is often at times quite extensive code to illustrate some ways of analyzing the data or reproducing the results from the paper from which the data were taken. Essentially, this was a result of me moving the datasets that were originally part of the metafor package to metadat and hence copying the help files from one package to the other. As a result, the majority of help files illustrate the analysis of the data using the metafor package (given that about 3/4 of the datasets that are currently in metadat were contributed by me). The question has come up whether it should be possible to contribute code to illustrate the analysis of datasets using other packages (of course this would apply to any dataset, not just those that were contributed by me).

My approach so far has been this: The person who spends the time to extract, document, and contribute a dataset to the package (which often can take considerable amounts of time) is also the person who gets to write the Examples section (can think of this as a reward). This is why I have also never touched the Examples section on any of the datasets that were contributed to the package by other people. But given that many datasets were contributed by me, this might come across as me 'gate keeping' what packages are used to illustrate the analyses. Hence, I would like to solicit feedback on this issue in an open/transparent manner.

A few additional thoughts:

• Additional example code is certainly useful for those who would like to see different packages/approaches for analyzing the same data.
• If additional code can be contributed, who gets to decide what code is actually added?
• What if somebody just wants to add 'boiler plate' analysis code to every single dataset?
• Also, how would the code section be structured? For example, whose code is shown first? What if there is disagreement as to the appropriate way(s) of analyzing the data?
• Is there a limit to the extent of the code? Is there a danger that the Examples section might become confusing/messy if there is too much code/output there?
• Analysis code is wrapped in \dontrun{} because the package would already be rejected by CRAN due to excessively long run times on some of the examples. However, for creating the pkgdown docs (https://wviechtb.github.io/metadat/), I build the docs with pkgdown::build_site(run_dont_run = TRUE), which is why the actual results are shown there. What if code breaks? Who is then responsible for fixing things? As maintainer (and the person who currently builds these docs), it would then at least be my responsibility to email people about their code.
• Right now, it takes about 15-20 minutes to build these docs. Can this start to get out of hand if a lot more code is added? Should there be a limit on the run times?
• If we decide to stick to the 'the contributor gets to write the Examples section' approach: I see lots of datasets in other packages that are not part of metadat. Why not move those datasets over in which case one could balance out which packages are emphasized in the Examples section?
• Another possibility: No code at all in the Examples section (but everybody is of course free to put code to illustrate the analysis of datasets on their own website or some other repository).

Hope to hear other people's thoughts on this!

[thomased]

Hi Wolfgang,

It's a good question of course — I hadn't considered it. My very scattered thoughts:

• You're the maintainer, and the work on fixing these things when they break & making sure pages build etc. falls on you (& us of course), so I think it's perfectly fine to require a standard or template for the examples, if so desired. I wouldn't see that as 'gatekeeping' in any way.
• Since it's already quite a popular package (near ~200k downloads? wow...), it does seem like there's a (small) risk of people dropping in boilerplate examples that don't add a great deal, perhaps to 'advertise' other packages. In which case it would be handy to have some transparent standards/rules to point to when considering how to handle those contributions.
• With all that said, I wonder if the work is worth it. If I had to choose, I might be inclined to just not have examples at all, like you say. People are welcome to build on the package and use the datasets to illustrate anything they like in their own packages, or websites etc. And it feels like a very neutral stance. Here's a growing collection of interesting & very well-documented datasets — do with it what you will!

That's just my 2c anyway — very open to other arguments!

[daniel1noble]

Hi Wolfgang and Tom,

Wolfgang, good points. I largely agree with what Tom says here. Some more random thoughts.

1. The fact that most example code is demonstrated using the metafor package is not at all a problem to me. Nor do I see it as you 'gatekeeping'. metadat, while its own package, is tightly intertwined with metafor. Having example code that primarily relies on metafor therefore makes sense. I guess I would ask the question, "How often is metadat used outside on its own versus it being downloaded as a dependency with metafor?". I'm not sure if download counts can disentangle this distinction, but it's probably an important one to consider.

2. I think sticking to basic examples that demonstrate the use of the dataset in relation to a single example of how it was analysed in the paper is all that needs to be done, and no more. To me, these should be short examples to demonstrate the types of questions being addressed with the data not a tutorial on how to fit meta-analytic models or model selection. You already have a fantastic web page that does that! Having said that, I think some code is a good idea. I wouldn't get rid of it completely as it gives users interested in the dataset some context.

Anyway, just my thoughts, but you're the main driver of the package Wolfgang. So, I agree with Tom that I think we should be accomodating you and ensuring that the pipelines for you maintaining the package are made easy and streamlined.

Cheers, Dan

[guido-s]

Hi all,

I finally found the time to reply to this thread (which has been started by Wolfgang after a brief email exchange with me).

My main idea to contribute R code to existing help pages would be to make additional statistical methods known to the user. For example, R package metasens provides some methods to adjust for selection bias which are not available in metafor (and vice versa). In my view, it would be a great help to users to find these methods in a single help file.

Having no analysis code in Examples would in my view severely diminish the usefulness of the help pages in metadat. This would be OK if the main purpose of metadat is to act only as an online database for meta-analysis data sets.

If code contributions to examples are allowed, these contributions

• could be restricted to authors and data contributors of metadat,

• could be only added as commented out code which would neither increase the run time nor produce any errors,

• could be provided below / at the end of the dontrun environment,

• should start with an informative header providing information on the author of the R code (and that s/he is responsible for the code), his/her email address (for questions on the code etc.) and the purpose of the additional code.

Best, Guido