search

KentonWhite / ProjectTemplate

A template utility for R projects that provides a skeletal project.
http://projecttemplate.net
GNU General Public License v3.0
623 stars 159 forks source link

tests fail in master #186

Closed connectedblue closed 7 years ago

connectedblue commented 7 years ago

I just synced to the latest master and I have some tests fail:

Error in rsqlite_send_query(conn@ptr, statement) : 
  file is encrypted or is not a database
1. Failure: ignored data files are not loaded (@test-load.R#186) -----------------------------
exists("test", envir = .TargetEnv) isn't false.

2. Failure: ignored data files are not loaded (@test-load.R#193) -----------------------------
exists("test", envir = .TargetEnv) isn't false.

These are run locally, straight off the master branch with no modifications.

I'm not sure how this can happen as the travis checks must have picked this up at the last merge. Does anyone else see this failure?

Hugovdberg commented 7 years ago

It seems to me that testthat isn't always completely catching the errors, especially that RSQLite error. I also noticed that sometimes tests seem to succeed or fail depending on whether you run devtools::test, testthat::test_package or devtools::check. Did you experience differences between those? Perhaps this has something to do with the fact we're messing with the global environment during testing? Since (almost) every function we write uses .TargetEnv (I fixed a couple succeeding 'by accident' in an upcoming PR) we could perhaps use a clean .TargetEnv in the tests, with only some specific tests for the difference between .TargetEnv <- GlobalEnv and .TargetEnv <- .TestEnv?

Hugovdberg commented 7 years ago

Also, testthat::test_package('ProjectTemplate', 'create') produces some failed tests on my system that are listed along the way but not in the final results. Does that happen with you to?

KentonWhite commented 7 years ago

I'm seeing the this error:

Error in sqliteSendQuery(con, statement, bind.data) : 
  error in statement: file is encrypted or is not a database

But testthat is not showing it as a failure. Reading the comments this seems to be expected behavior:

 # Should trigger error
  # reload the project, now with an illegal Thumbs.db

So the illegal Thumbs.db is throwing and error, which is printed to the console -- but it is expected that there is an error.

connectedblue commented 7 years ago

I just do source(tests/run-all.R).

I didn't know there were multiple ways to run the tests. Even if there are different environment issues, it still seems odd why just those tests fail, and moreover why Travis passed them.

Either way, my own development is at a standstill. I don't want to add new features when tests fail, because you can't see whether it's because of the test, or because of a change I have made.

Hugovdberg commented 7 years ago

Give testthat::test_package a try instead of the test_check in tests/run-all.R, it is recommended for interactive testing (see the help page). test_check is recommended for run-all.R, but then to be run by R CMD check.

connectedblue commented 7 years ago

When I run `test_package("ProjectTemplate") it throws out the following error:

Failed ---------------------------------------------------------------------------------------------------
1. Failure: ignored data files are not loaded (@test-load.R#186) -----------------------------------------
exists("test", envir = .TargetEnv) isn't false.

2. Failure: ignored data files are not loaded (@test-load.R#193) -----------------------------------------
exists("test", envir = .TargetEnv) isn't false.

@Hugovdberg When you check out master and run this command do all your tests pass?

Hugovdberg commented 7 years ago

@connectedblue I checked out upstream/master and ran the command R -q -e "devtools::load_all(); testthat::test_package('ProjectTemplate')" > /tmp/testlog_upstream_master.log (runs devtools::test() and logs everything to a textfile in case you're not a linux guy). The log reports this:

> devtools::load_all(); testthat::test_package('ProjectTemplate')
AddExtension: .....
Testing Autogeneration: ..
Caching: ............................................
Clearing from memory: ..........................
Configuration: ......
Create project: ...................................................1
Failed -------------------------------------------------------------------------
1. Error: Full project (@test-create.R#76) -------------------------------------
attempt to apply non-function
1: suppressMessages(test.project()) at /home/hugo/Projects/ProjectTemplate/tests/testthat/test-create.R:76
2: withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3: test.project()
4: lapply(files, testthat::test_file, env = environment()) at /home/hugo/Projects/ProjectTemplate/R/test.project.R:28
5: FUN(X[[i]], ...)
6: with_reporter(reporter = reporter, start_end_reporter = start_end_reporter, {
       lister$start_file(basename(path))
       source_file(path, new.env(parent = env), chdir = TRUE)
       end_context()
   })
7: force(code)
8: source_file(path, new.env(parent = env), chdir = TRUE)
9: eval(exprs, env)
10: eval(expr, envir, enclos)
11: expect_that(1, equals(1)) at tests/1.R:3
12: condition(object)
13: expect_equal(x, expected, ..., expected.label = label)
14: expect(comp$equal, sprintf("%s not equal to %s.\n%s", lab_act, lab_exp, comp$message), 
       info = info)
15: withRestarts(if (expectation_broken(exp)) {
       stop(exp)
   } else {
       signalCondition(exp)
   }, continue_test = function(e) NULL)
16: withOneRestart(expr, restarts[[1L]])
17: doWithOneRestart(return(expr), restart)
18: signalCondition(exp)
19: (function (e) 
   {
       register_expectation(e, frame + 11, sys.nframe() - 6)
       invokeRestart("continue_test")
   })(structure(list(message = "1 not equal to 1.\nEqual\n", srcref = NULL), .Names = c("message", 
   "srcref"), class = c("expectation_success", "expectation", "condition")))
20: register_expectation(e, frame + 11, sys.nframe() - 6)
21: get_reporter()$add_result(context = get_reporter()$.context, test = test, result = e)
22: o_apply(self$reporters, "add_result", context = context, test = test, result = result)
23: lapply(objects, function(x) x[[method]](...))
24: FUN(X[[i]], ...)
25: x[[method]](...)

DONE ===========================================================================
...................................................................................1
Failed -------------------------------------------------------------------------
1. Error: Test full project into existing directory (@test-create.R#111) -------
attempt to apply non-function
1: suppressMessages(test.project()) at /home/hugo/Projects/ProjectTemplate/tests/testthat/test-create.R:111
2: withCallingHandlers(expr, message = function(c) invokeRestart("muffleMessage"))
3: test.project()
4: lapply(files, testthat::test_file, env = environment()) at /home/hugo/Projects/ProjectTemplate/R/test.project.R:28
5: FUN(X[[i]], ...)
6: with_reporter(reporter = reporter, start_end_reporter = start_end_reporter, {
       lister$start_file(basename(path))
       source_file(path, new.env(parent = env), chdir = TRUE)
       end_context()
   })
7: force(code)
8: source_file(path, new.env(parent = env), chdir = TRUE)
9: eval(exprs, env)
10: eval(expr, envir, enclos)
11: expect_that(1, equals(1)) at tests/1.R:3
12: condition(object)
13: expect_equal(x, expected, ..., expected.label = label)
14: expect(comp$equal, sprintf("%s not equal to %s.\n%s", lab_act, lab_exp, comp$message), 
       info = info)
15: withRestarts(if (expectation_broken(exp)) {
       stop(exp)
   } else {
       signalCondition(exp)
   }, continue_test = function(e) NULL)
16: withOneRestart(expr, restarts[[1L]])
17: doWithOneRestart(return(expr), restart)
18: signalCondition(exp)
19: (function (e) 
   {
       register_expectation(e, frame + 11, sys.nframe() - 6)
       invokeRestart("continue_test")
   })(structure(list(message = "1 not equal to 1.\nEqual\n", srcref = NULL), .Names = c("message", 
   "srcref"), class = c("expectation_success", "expectation", "condition")))
20: register_expectation(e, frame + 11, sys.nframe() - 6)
21: get_reporter()$add_result(context = get_reporter()$.context, test = test, result = e)
22: o_apply(self$reporters, "add_result", context = context, test = test, result = result)
23: lapply(objects, function(x) x[[method]](...))
24: FUN(X[[i]], ...)
25: x[[method]](...)

DONE ===========================================================================
.......................................................
Load project: ...........W...W.W.W.WWW.WWW..WWW.WWW..WWW..WWW..WWW..WWW..
Migration: .............WW....W.......
Readers: ...................................................................................................................................W...........................................................................
Require package: .......
Clean variable names: .....
Version field: ......

Warnings -----------------------------------------------------------------------
1. auto loaded data is not cached when cached_loaded_data is FALSE (@test-load.R#95) - cannot open compressed file 'cache/test.RData', probable reason 'No such file or directory'

2. auto loaded data from an R script is cached correctly (@test-load.R#135) - cannot open compressed file 'cache/test_data11.RData', probable reason 'No such file or directory'

3. auto loaded data from an R script is cached correctly (@test-load.R#137) - cannot open compressed file 'cache/test_data21.RData', probable reason 'No such file or directory'

4. ignored data files are not loaded (@test-load.R#155) - Coercing LHS to a list

5. ignored data files are not loaded (@test-load.R#168) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

6. ignored data files are not loaded (@test-load.R#168) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

7. ignored data files are not loaded (@test-load.R#168) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

8. ignored data files are not loaded (@test-load.R#173) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

9. ignored data files are not loaded (@test-load.R#173) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

10. ignored data files are not loaded (@test-load.R#173) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

11. ignored data files are not loaded (@test-load.R#181) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

12. ignored data files are not loaded (@test-load.R#181) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

13. ignored data files are not loaded (@test-load.R#181) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

14. ignored data files are not loaded (@test-load.R#185) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

15. ignored data files are not loaded (@test-load.R#185) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

16. ignored data files are not loaded (@test-load.R#185) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

17. ignored data files are not loaded (@test-load.R#191) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

18. ignored data files are not loaded (@test-load.R#191) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

19. ignored data files are not loaded (@test-load.R#191) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

20. ignored data files are not loaded (@test-load.R#198) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

21. ignored data files are not loaded (@test-load.R#198) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

22. ignored data files are not loaded (@test-load.R#198) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

23. ignored data files are not loaded (@test-load.R#205) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

24. ignored data files are not loaded (@test-load.R#205) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

25. ignored data files are not loaded (@test-load.R#205) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

26. ignored data files are not loaded (@test-load.R#213) - Your configuration file is missing the following entries: version, data_loading, data_ignore, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

27. ignored data files are not loaded (@test-load.R#213) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

28. ignored data files are not loaded (@test-load.R#213) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

29. projects without the cached_loaded_data config have their migrated config set to FALSE  (@test-migration.R#77) - Your configuration file is missing the following entries: cache_loaded_data. Defaults will be used.

30. projects without the cached_loaded_data config have their migrated config set to FALSE  (@test-migration.R#78) - cannot open compressed file 'cache/test.RData', probable reason 'No such file or directory'

31. projects without the cached_loaded_data config have their migrated config set to FALSE  (@test-migration.R#91) - cannot open compressed file 'cache/test.RData', probable reason 'No such file or directory'

32. Example 29: SQLite3 Support with .sql Extension with query = "SELECT * FROM ..." (@test-readers.R#562) - RSQLite::dbGetException() is deprecated, please switch to using standard error handling via tryCatch().

DONE ===========================================================================
> 
>

As I said, I get a lot of warnings, apparently here because I forgot to checking the call to .read.config in #178. Also there are some errors within testthat from test-create.R which are not reported as failures, so I'm not sure all the tests work exactly as we expect them to work.

KentonWhite commented 7 years ago

Interesting. Can you do a git bisect and see at which commit these errors started?

connectedblue commented 7 years ago

I checked out 71eafd8a47cf3 on master and ran devtools::test() This delivered no test failures, but only 11 warnings:

Warnings -------------------------------------------------------------------------------------
1. auto loaded data is not cached when cached_loaded_data is FALSE (@test-load.R#85) - Coercing LHS to a list

2. auto loaded data is not cached when cached_loaded_data is FALSE (@test-load.R#88) - Your configuration file is missing the following entries: version, data_loading, cache_loading, recursive_loading, munging, logging, logging_level, load_libraries, libraries, as_factors, data_tables, attach_internal_libraries, sticky_variables. Defaults will be used.

3. auto loaded data is not cached when cached_loaded_data is FALSE (@test-load.R#88) - Your configuration contains the following unused entries: X.0.7., X.TRUE., X.TRUE..1, X.FALSE., X.TRUE..2, X.FALSE..1, X.INFO., X.FALSE..2, X.reshape..plyr..dplyr..ggplot2..stringr..lubridate., X.TRUE..3, X.FALSE..3, X.FALSE..4, X.TRUE..4, X.NONE.. These will be ignored.

4. auto loaded data is not cached when cached_loaded_data is FALSE (@test-load.R#88) - Your configuration is compatible with version 0.5 of the ProjectTemplate package.
  Please run ProjectTemplate::migrate.project() to migrate to the installed version 0.7.

5. auto loaded data is not cached when cached_loaded_data is FALSE (@test-load.R#91) - cannot open compressed file 'cache/test.RData', probable reason 'No such file or directory'

6. auto loaded data from an R script is cached correctly (@test-load.R#131) - cannot open compressed file 'cache/test_data11.RData', probable reason 'No such file or directory'

7. auto loaded data from an R script is cached correctly (@test-load.R#133) - cannot open compressed file 'cache/test_data21.RData', probable reason 'No such file or directory'

8. projects without the cached_loaded_data config have their migrated config set to FALSE  (@test-migration.R#77) - Your configuration file is missing the following entries: cache_loaded_data. Defaults will be used.

9. projects without the cached_loaded_data config have their migrated config set to FALSE  (@test-migration.R#78) - cannot open compressed file 'cache/test.RData', probable reason 'No such file or directory'

10. projects without the cached_loaded_data config have their migrated config set to FALSE  (@test-migration.R#91) - cannot open compressed file 'cache/test.RData', probable reason 'No such file or directory'

11. Example 29: SQLite3 Support with .sql Extension with query = "SELECT * FROM ..." (@test-readers.R#562) - RSQLite::dbGetException() is deprecated, please switch to using standard error handling via tryCatch().

By way of comparison, when you checkout 497d2d574a6421f from last August, there are no warnings from this command

I think quite a few errors have slipped into the regression pack in last few months which I take responsibility for, and some more have slipped into recent regressions (perhaps as a result of cutting and pasting test scripts already with warnings embedded).

I think we should look to eradicate errors in master before adding new features.

connectedblue commented 7 years ago

I added #188 to address some of the warnings from this tool.

I still experience local issues running the test scripts, but the warnings have disappeared at least.