eerolinna commented 4 years ago

Fixes #65

This test sort of works right now, but not completely yet.

The generation settings on travis seem to be slightly different from what was used to generate the current README (see later comments in this thread)

The code outputs contain absolute paths like

dfp <- data_file_path(po)
dfp
## [1] "/var/folders/9h/yf354vb917z6gr6mz7bfb1d40000gn/T//RtmptE97ZB/posteriordb_cache/data/data/eight_schools.json"

These paths will be different when running on CI, causing the test to fail

The test might not work on Windows as it uses unix-style paths

eerolinna commented 4 years ago

Edit: fixed Looks like readLines("../../../README.md") doesn't work on travis (it did work locally)

codecov-io commented 4 years ago

Codecov Report

Merging #89 into master will increase coverage by 20.96%. The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #89       +/-   ##
==========================================
+ Coverage   78.34%   99.3%   +20.96%     
==========================================
  Files          20       6       -14     
  Lines         651     144      -507     
==========================================
- Hits          510     143      -367     
+ Misses        141       1      -140

Impacted Files	Coverage Δ
rpackage/R/gold_standard.R
rpackage/R/filter.R
rpackage/R/utils.R
rpackage/R/data_info.R
rpackage/R/utils_gold_standard.R
rpackage/R/write_pdb.R
rpackage/R/utils_tests.R
rpackage/R/tibble.R
rpackage/R/posterior.R
rpackage/R/model_code.R
... and 4 more

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 8c7d0d5...f200542. Read the comment docs.

eerolinna commented 4 years ago

Now the logic of the test works and the remaining test failures are from actual differences between current README.md and what travis generates from README.Rmd

── 3. Failure: Check that the generated README is up-to-date (@test-readme.R#8) current_readme not equal to generated_readme. 28/367 mismatches x[148]: " ## name model_name gold_standard_n… data_name added_by added_date x[148]: keywords"
y[148]: " ## name model_name gold_standard_na… data_name added_by added y[148]: _date keywords"

etc, see https://travis-ci.org/MansMeg/posteriordb/jobs/614618733 for more

I guess there is some truncation length variable that is different between travis and what was used to generate the current README that should not be too hard to fix.

The issue with machine-specific paths like

dfp <- data_file_path(po)
dfp
## [1] "/var/folders/9h/yf354vb917z6gr6mz7bfb1d40000gn/T//RtmptE97ZB/posteriordb_cache/data/data/eight_schools.json"

still remains though. @MansMeg do you have any ideas on how to tackle it?

MansMeg commented 4 years ago

It still seem to fail? Seem like the generation of the readme fails due to difference in no of whitespaces?

In general, these kinds of tests are tricky since they can change due to new versions of for example rmarkdown. Im not sure how to solve this as efficiently as possible. We dont want it to fail for whitespaces or newlines.

eerolinna commented 4 years ago

The current failure is due to for example gold_standard_n… and gold_standard_na… not matching, which is more about having different truncation thresholds than whitespace. I don't think there should be any whitespace differences*? (let me know if you saw something)

A problematic situation is

dfp <- data_file_path(po)
dfp

Where on one computer the result might be

## [1] "/var/folders/9h/yf354vb917z6gr6mz7bfb1d40000gn/T//RtmptE97ZB/posteriordb_cache/data/data/eight_schools.json"

and on other computer

## [1] /home/eero/...

The truncation thresholds should be pretty easy to solve but this one I don't know what to do about.

[*] The newline in

x[148]: " ## name model_name gold_standard_n… data_name added_by added_date
x[148]: keywords"
y[148]: " ## name model_name gold_standard_na… data_name added_by added
y[148]: _date keywords"

is in different place which could be considered a whitespace difference, but ultimately it was caused by different truncation length.

MansMeg commented 4 years ago

So: 1) I suggest you remove all lines with search path. That would solve the first issue. 2) I guess we can try to avoid the truncation somehow? Can that be specified as a parameter?

eerolinna commented 4 years ago

I don't think I'm going to work on this in a while. Feel free to pick this up if you want to.

MansMeg commented 4 years ago

This is not prioritized for me right now so Ill wait.

MansMeg commented 4 years ago

I have now included tests for the README and CONTRIBUTING markdown files. Not exactly the same, but the same purpose.

stan-dev / posteriordb

WIP: Add test for checking that generated readme is up-to-date #89

Codecov Report