Datasets version for reports

[juliacarbajal]

I noticed that the reports are run always on the current version of the available datasets in Metalab, while the conclusions were written based on a previous version. This might be problematic as with new data, the conclusions may change. For instance, I have the impression that the discussion in the developmental curve report doesn't quite match the results of the models anymore (in the first model comparison it states that the quadratic model is better than the linear or the log model, yet Pr(>Chisq) = 1 in both comparisons).

To avoid incongruities between the text and the models it would be good to fix the dataset version used in the reports, and perhaps have someone re-run the analysis once or twice a year to incorporate new data, updating the text if necessary.

[mcfrank]

Julia, I agree about this - maybe we need to tone down the text so that it is more general?

In practice it can be very hard to ensure that someone checks all aspects of the site, even once every six months. Much better to make the reports more descriptive!

On Wed, Nov 14, 2018 at 7:01 AM Julia notifications@github.com wrote:

I noticed that the reports are run always on the current version of the available datasets in Metalab, while the conclusions were written based on a previous version. This might be problematic as with new data, the conclusions may change. For instance, I have the impression that the discussion in the developmental curve report doesn't quite match the results of the models anymore (in the first model comparison it states that the quadratic model is better than the linear or the log model, yet Pr(>Chisq) = 1 in both comparisons).

To avoid incongruities between the text and the models it would be good to fix the dataset version used in the reports, and perhaps have someone re-run the analysis once or twice a year to incorporate new data, updating the text if necessary.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/langcog/metalab2/issues/102, or mute the thread https://github.com/notifications/unsubscribe-auth/ABGul72b2StoxPCMysEfO8FA4fNjiQ9Hks5uvDBQgaJpZM4Yd6-L .

[christinabergmann]

Excellent observation, Julia! And I agree about making it more descriptive. We might need to reframe reports as demonstrations what you can do with the data (and maybe add some practical examples, too).

Since some reports were written before there were domains we also might need to add subsetting to the correct domain for some reports.

(Reports with concrete conclusions could state "This report was written based on the data from XXX.")