Move the markdown package from Imports to Suggests

[yihui]

For package authors who use R Markdown vignettes based on the vignette engine knitr::knitr (or knitr::docco_linear, knitr::docco_classic, etc.), you should declare the (soft) dependency on markdown, e.g., in the package DESCRIPTION:

Suggests: markdown
VignetteBuilder: knitr

Previously they didn't need to do this because markdown has been a hard dependency of knitr (in Imports), so the availability of markdown is guaranteed by knitr, but I want to make markdown a soft dependency of knitr in the future.

I think most packages use the vignette engine knitr::rmarkdown to write vignettes now, but there may still be a few vignettes using the knitr::knitr engine, which depends on markdown. If I move markdown to Suggests in knitr, users have to make sure markdown is available to R CMD check. To achieve that, they have to add markdown to Suggests to their DESCRIPTION files.

Similarly, if your vignette is based on the vignette engine knitr::rmarkdown, you have to declare the dependency on rmarkdown, e.g.,

Suggests: rmarkdown
VignetteBuilder: knitr

The error message when building the source package via R CMD build should tell you if you need to add markdown or rmarkdown to Suggests, which looks like this:

* checking for file ‘PKG/DESCRIPTION’ ... OK
* preparing ‘PKG’:
* checking DESCRIPTION meta-information ... OK
* installing the package to build vignettes
* creating vignettes ... ERROR
--- re-building ‘doc.Rmd’ using rmarkdown_notangle
Error: processing vignette 'doc.Rmd' failed with diagnostics:
The 'rmarkdown' package should be declared as a dependency of
the 'PKG' package (e.g., in the  'Suggests' field of DESCRIPTION),
because the latter contains vignette(s) built with the 'rmarkdown'
package. Please see https://github.com/yihui/knitr/issues/1864
for more information.
--- failed re-building ‘doc.Rmd’

SUMMARY: processing the following file failed:
  ‘doc.Rmd’

Error: Vignette re-building failed.
Execution halted

If you are a package author affected by this issue, but not clear about what you need to do, please reply below and I'll be glad to help. I recommend that you skip the replies below. Thanks!

[yihui]

@mkoohafkan I just made the error more accurate in case Pandoc is not installed. Thanks!

[samuel-rosa]

I have found a possibly related issue and would like to confirm this before opening a new one.

The Description file of the febr package Imports knitr and Suggests rmarkdown because they are used to build the vignette. When running CRAN checks, the following NOTE is issued: Including not using Suggests conditionally.

For example https://win-builder.r-project.org/X079z5f4WZ1T/00check.log

I could not find a solution for using rmarkdown conditionally. If you think that this is not a related issue, I can open a new one.

The package source code is available on https://github.com/samuel-rosa/febr-package/

[yihui]

@samuel-rosa I don't think the NOTE was from the current version of your package, but from the archived version instead.

* checking CRAN incoming feasibility ... NOTE
Maintainer: 'Alessandro Samuel-Rosa <alessandrosamuelrosa@gmail.com>'

New submission

Package was archived on CRAN

CRAN repository db overrides:
  X-CRAN-Comment: Archived on 2020-10-24 as problems were not corrected
    in time.

  Including not using Suggests conditionally.

That is, "not using Suggests conditionally" was the reason (or one of the reasons) why your previous version of package was archived on CRAN. As long as you have addressed this problem, and R CMD check --as-cran doesn't reveal new problems, I think you are good to go.

[samuel-rosa]

@yihui it seems that this is another CRAN message that needs improvement. :-| Many thanks!!!

[hpages]

Hi @yihui ,

Now that the dust of the last Bioconductor release has settled down, we would like to be able to control this behavior on the Bioconductor build machines. Would you be amenable to introduce a dedicated environment variable for this? E.g. something like:

diff --git a/R/utils.R b/R/utils.R
index 515f9638..c1e7b907 100644
--- a/R/utils.R
+++ b/R/utils.R
@@ -1062,7 +1062,7 @@ test_desc_dep = function(pkg, dir = '.') {

 # TODO: remove this hack in the future when no CRAN/BioC packages have the issue
 test_vig_dep = function(pkg) {
-  if (skip_cran_bioc() || test_desc_dep(pkg, '..')) return()
+  if (no_test_vig_dep() || test_desc_dep(pkg, '..')) return()
   p = read.dcf(file.path('..', 'DESCRIPTION'), fields = 'Package')[1, 1]
   stop2(
     "The '", pkg, "' package should be installed and declared as a dependency of the '", p,
@@ -1072,4 +1072,8 @@ test_vig_dep = function(pkg) {
   )
 }

-skip_cran_bioc = function() is_R_CMD_check() || Sys.getenv('BBS_HOME') != ''
+no_test_vig_dep = function() is_R_CMD_check() || tolower(Sys.getenv("KNITR_NO_TEST_VIGNETTE_DEP", "true")) == "true"

I can turn this into a PR if you want. Let me know.

Thanks! H.

[yihui]

@hpages Yes, I'll be more than happy to merge the PR, or just push the commit myself since it is simple enough. Should Sys.getenv("KNITR_NO_TEST_VIGNETTE_DEP", "true") be just Sys.getenv("KNITR_NO_TEST_VIGNETTE_DEP")? It seems you want to set KNITR_NO_TEST_VIGNETTE_DEP to true on Bioconductor to skip the test.

[hpages]

Should Sys.getenv("KNITR_NO_TEST_VIGNETTE_DEP", "true") be just Sys.getenv("KNITR_NO_TEST_VIGNETTE_DEP")?

Oops! Yes, absolutely. This should not be true by default. Thanks for the catch.

[yihui]

Okay. Just to make sure, this env var has already been set on Bioconductor, right? (meaning I no longer need to test Sys.getenv('BBS_HOME') != '' in future versions of knitr)

[hpages]

Correct. You no longer need to test Sys.getenv('BBS_HOME') != '' in future versions of knitr. Using BBS_HOME to control whether to skip the vignette dep test or not is not flexible and makes reproducibility difficult. Using a dedicated env var like KNITR_NO_TEST_VIGNETTE_DEP addresses that. Thanks!

[yihui]

That makes perfect sense. I just pushed the commit. Thanks!

[vnijs]

@yihui This change, surprisingly, caused an error in my shiny app. The fix is straightforward enough but now CRAN will not accept the new package version due to an issue in the vignette. Or rather, it seems there is an issue with how CRAN windows builds parse URLs for CSS in Rmarkdown. The URL for the CSS file in the header shown below clearly has two /s. However, when processed by CRAN on windows, one of those /s is magicked away (see message and cran-check link below). I have sent a few emails to CRAN but I'm not making any progress. Students using my app at different schools are getting errors so there is some urgency to get the new version on CRAN.

Do you have any suggestions on how I could change the url specification in the vignette header below to pass the CRAN windows builds?

pandoc.exe: Could not fetch http:/netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css

See also: https://www.r-project.org/nosvn/R.check/r-release-windows-ix86+x86_64/radiant-00check.html

---
title: Programming with Radiant
author: "Vincent R. Nijs, Rady School of Management (UCSD)"
date: "`r Sys.Date()`"
output: 
  rmarkdown::html_vignette:
    css: https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css
vignette: >
  %\VignetteIndexEntry{Programming with Radiant}
  %\VignetteEngine{knitr::rmarkdown}
  \usepackage[utf8]{inputenc}
---

[yihui]

@vnijs I don't know where the URL http:/netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css came from. This is the css option you used in the current version of radiant (v1.3.2):

output: 
  rmarkdown::html_vignette:
    css: http://netdna.bootstrapcdn.com/font-awesome/4.0.3/css/font-awesome.css

I don't know if the change to https://cdnjs.cloudflare.com/ajax/libs/font-awesome/5.15.3/css/all.min.css will help, since I don't know the Pandoc version CRAN uses. I guess the best you could do for now is to see if the problem is reproducible on win-builder (try both v1.3.2 and your current development version, and see if the former throws the same error and the latter passes).

[cderv]

@vnijs this particular "could not fetch" issue is a bug in rmarkdown that we unfortunately missed in last release in our tests, and in reverse dependencies checks. I'll fix that today (rstudio/rmarkdown#2163)

[jameskozubek]

I am no expert in R and I got the following error. How do I add the markdown to suggest?

devtools::install_github("kenhanscombe/ukbtools", build_vignettes = TRUE, dependencies = TRUE) Downloading GitHub repo kenhanscombe/ukbtools@HEAD ✔ checking for file ‘/tmp/RtmprNPa9Z/remotes73b52d1262aa/kenhanscombe-ukbtools-4015440/DESCRIPTION’ ... ─ preparing ‘ukbtools’: ✔ checking DESCRIPTION meta-information ... ─ installing the package to process help pages E creating vignettes (4.5s) --- re-building ‘explore-ukb-data.Rmd’ using rmarkdown Warning in engine$weave(file, quiet = quiet, encoding = enc) : Pandoc (>= 1.12.3) not available. Falling back to R Markdown v1. Error: processing vignette 'explore-ukb-data.Rmd' failed with diagnostics: The 'markdown' package should be installed and declared as a dependency of the 'ukbtools' package (e.g., in the 'Suggests' field of DESCRIPTION), because the latter contains vignette(s) built with the 'markdown' package. Please see https://github.com/yihui/knitr/issues/1864 for more information. --- failed re-building ‘explore-ukb-data.Rmd’

SUMMARY: processing the following file failed: ‘explore-ukb-data.Rmd’

Error: Vignette re-building failed. Execution halted Error: Failed to install 'ukbtools' from GitHub: System command 'R' failed, exit status: 1, stdout + stderr (last 10 lines): E> Pandoc (>= 1.12.3) not available. Falling back to R Markdown v1. E> Error: processing vignette 'explore-ukb-data.Rmd' failed with diagnostics: E> The 'markdown' package should be installed and declared as a dependency of the 'ukbtools' package (e.g., in the 'Suggests' field of DESCRIPTION), because the latter contains vignette(s) built with the 'markdown' package. Please see https://github.com/yihui/knitr/issues/1864 for more information. E> --- failed re-building ‘explore-ukb-data.Rmd’ E> E> SUMMARY: processing the following file failed: E> ‘explore-ukb-data.Rmd’ E> E> Error: Vignette re-building failed. E> Execution halted

[jangorecki]

@jameskozubek relevant sections of manual https://cran.r-project.org/doc/manuals/R-exts.html#The-DESCRIPTION-file https://cran.r-project.org/doc/manuals/R-exts.html#Package-Dependencies

[yihui]

@jameskozubek If you do not install the package inside RStudio, you have to install Pandoc: https://pandoc.org/installing.html Your ran into the error because Pandoc was not available:

Warning in engine$weave(file, quiet = quiet, encoding = enc) : Pandoc (>= 1.12.3) not available. Falling back to R Markdown v1.

The error could be circumvented by adding markdown to Suggests in DESCRIPTION, but I don't recommend that. BTW, it seems you are not the package author of ukbtools, so it will not be straightforward for you to do that.

[jameskozubek]

Thank you. Installing pandoc worked.

[aharmer]

The error could be circumvented by adding markdown to Suggests in DESCRIPTION, but I don't recommend that.

Hi Yihui, can you please elaborate on why adding markdown to suggests is not a good idea if using knitr::rmarkdown. A colleague who tested this for me gets the install error telling them about adding rmarkdown to Suggests, but I already have it there. He is not using RStudio but native R, and so maybe this is a pandoc issue? Can you recommend the best way to handle this?

My package repository is aharmer/morphoBlocks.

Thanks in advanced. Aaron.

[yihui]

why adding markdown to suggests is not a good idea if using knitr::rmarkdown.

@aharmer If you use the vignette engine knitr::rmarkdown, you mean to use Pandoc to build your vignettes. Adding markdown to Suggests means to use the markdown package to build vignettes, which is a workaround. Since markdown is not Pandoc and doesn't cover Pandoc's Markdown features, you could end up with an unexpected vignette.

He is not using RStudio but native R, and so maybe this is a pandoc issue? Can you recommend the best way to handle this?

He needs to install Pandoc if he wants to build the package from source with vignettes. Your instructions are perfectly accurate: https://github.com/aharmer/morphoBlocks#installation

[aharmer]

Thanks so much @yihui, this really clarifies things for me about the difference between the two packages. In the end it turned out the issue was I didn't include dependencies = TRUE in the install_github instructions on my README. I wrongly assumed specifying build_vignettes = TRUE would install the Suggests. Thanks again, Aaron.

[lshep]

@yihui We just installed the latest version of R on the Bioconductor builders end of last week and installed knitr version 1.33.8 and the ERROR no longer occurs for packages that are using knitr but do not have rmarkdown/markdown in Suggest. It seems like you have decided not to throw this ERROR anymore or there is a bug in your code?

[yihui]

@lshep Bioconductor is a different story. This issue only affects R CMD check --as-cran. As @hpages replied above, Bioc doesn't set the env var _R_CHECK_SUGGESTS_ONLY_, so it's not affected by this R CMD check issue (it was affected once but the problem has been fixed in knitr long ago).

[lshep]

@yihui this may have been the wrong issue to post on but what I'm concerned with it how to reproduce the ERROR we were seeing on the Bioconductor build report since it has disappeared with knitr1.33.8? As this seemed to be a real ERROR that you wanted corrected and we were able to see the ERROR with knitr 1.33.3. My understanding was the work around was a temporary fix since the original updated was right before the Bioconductor release and breaking numerous packages; now we wanted the ERROR to appear so maintainers would correct?

[yihui]

@lshep Looking at the changes from 1.33.3 to 1.33.8 (https://github.com/yihui/knitr/compare/ae6ab6e2a781c071626c3b1d04ed802a2ee4afc2...fad3ead77209bb6cabbc86a50be21fa9fb52705d), I can't think of a possible reason why the error could have disappeared in 1.33.8. The only possibly relevant commit is 93720d3eea3483c5cb37e3ea1c0441f9cfe1e14d, which was @hpages's suggestion from https://github.com/yihui/knitr/issues/1864#issuecomment-852405935, but that shouldn't make the error disappear. I don't know why 1.33.3 could cause the error, either...

[hpages]

@yihui Correction: we were using knitr 1.33.5 on our build machines, which was raising the R CMD build error if either markdown or rmarkdown was not declared as a dep. With the latest knitr (1.33.8), we only see the error if markdown is not declared as a dep but we don't see it anymore for rmarkdown. The change is in commit c56fd61814d3b99242ba496dbb13280e6980eba4 from Aug 25.

So for example, a package like RMassBank (which uses rmarkdown to build its vignette) was getting an R CMD build error with knitr 1.33.5 but not anymore with knitr 1.33.8. Is this intended?

We've contacted many developers to let them know about these errors and ask them to fix their package. Now they're getting back to us telling us they don't know what we're talking about because they don't see the error on our daily build report. What should we tell them? Thanks

[yihui]

@hpages I'm confused now. Neither v1.33.5 nor v1.33.8 should cause an error on Bioc, because both markdown and rmarkdown are available on Bioc. If I understood you correctly last time (93720d3), you have set KNITR_NO_TEST_VIGNETTE_DEP=true on Bioc, right? With that, no Bioc packages should need to do anything, and R CMD build should not throw an error.

That said, I just committed 0dcd462eefd21086ef20ea33220f2964c45fb7ea, which completely removed the possibility of build error as long as markdown is available (which is true on Bioc).

[hpages]

@yihui We only use KNITR_NO_TEST_VIGNETTE_DEP=true for our release builds (BioC 3.13). We don't set this variable for our devel builds (BioC 3.14) because we want BioC developers to fix whatever error knitr detects in the devel version of their package.

I'm confused now. Neither v1.33.5 nor v1.33.8 should cause an error on Bioc, because both markdown and rmarkdown are available on Bioc.

Both markdown and rmarkdown have always been available on our build machines and yet we were getting these R CMD build errors. The error message was:

The 'markdown' package should be installed and declared as a dependency of the 'foo' package (e.g., in the 'Suggests' field of DESCRIPTION), because the latter contains vignette(s) built with the 'markdown' package.

The message complains that markdown should be declared as a dependency. AFAIK the fact that a package needs to declare another package as a dependency has nothing to do with what's installed on the system where the package is being tested. So if knitr considers it is a mistake that markdown or rmarkdow are not listed as a dependency, this is a fact that remains true whether these packages are installed or not.

With knitr v1.33.9, R CMD build RMassBank still doesn't give the error we used to get with knitr 1.33.5, despite the fact that RMassBank still uses rmarkdown to build its vignette but does not list it as a dep.

[yihui]

With knitr v1.33.9, R CMD build RMassBank still doesn't give the error we used to get with knitr 1.33.5, despite the fact that RMassBank still uses rmarkdown to build its vignette but does not list it as a dep.

Okay, that's the (new) behavior that I expect on Bioc now. Once CRAN submission is re-opened, I'll try to submit a new version of knitr to CRAN. Thanks!

[hpages]

Okay, that's the (new) behavior that I expect on Bioc now.

Focusing on the behavior on Bioc is not that helpful. When Bioc developers run R CMD build on their package, they don't do it on the Bioconductor build machines. They do it on their own machine and with a stock knitr installed from CRAN (knitr 1.33). This version of knitr gives them an error about markdown or rmarkdown not being in Suggests. This has been the case for months now (since knitr 1.33 was released on CRAN), and during all these months we've been using a setup on our devel build machines that was consistent with what people get on their own machine.

Even though this new check by knitr came as a surprise in April and broke hundreds of Bioconductor packages a few days before the BioC 3.13 release (see https://github.com/yihui/knitr/issues/1864#issuecomment-822848655), we embraced it in BioC 3.14 and have been working all these months with our developers to help them fix their package.

But now it seems that you've changed your mind. Are you saying that knitr will no longer consider that it's an error when a package uses markdown or rmarkdown to build its vignette and doesn't depend on it? Just want to make sure so we can clarify the situation with our developers. Thanks!

[yihui]

Are you saying that knitr will no longer consider that it's an error when a package uses markdown or rmarkdown to build its vignette and doesn't depend on it?

Yes. I'm following your original suggestion in https://github.com/yihui/knitr/issues/1864#issuecomment-822848655, i.e., "let things happen naturally." I will no longer throw an error on my side. If the build/check platform (e.g., CRAN) thinks it is an error not to declare the dependency, I'll just let the platform throw the error. The previous error on my side was meant to serve as a precaution and avoid future breakages on CRAN.

Now that I have gotten the permission from CRAN (and was even strongly encouraged by them) that it's time to break packages that still haven't made the correction, I'll just let them break, although personally I really hope to give them more time.

[hpages]

Thanks for clarifying. I know, it was my suggestion to let things happen naturally. More generally I don't think a package should meddle too much in what its reverse deps do with their Suggests field. We just need to know where you are going so we can follow, anticipate, and avoid miscommunication/confusion with our developers. Thanks again.

[stefanoborini]

For the record, I continue getting this message, despite both rmarkdown and markdown being declared in Suggests or Imports of my package, as well as having both of them installed in the environment.

[yihui]

@stefanoborini If you get the message even if you declare them in Imports, then there must be something else wrong. Chances are your .libPaths() is not properly configured on your computer and you installed rmarkdown/markdown to a custom library path that R is not able to find during R CMD build or R CMD check. You may try to configure R_LIBS_USER in ~/.Renviron.

[stefanoborini]

@yihui I use .Rprofile to extend the libPaths indeed, but then I don't understand why devtools::build works and R CMD build does not. Why would they behave differently? Don't they work exactly in the same way?

[yihui]

@stefanoborini I don't know why (I'm not the author of either of them), but my experience is that R commands (such as R CMD build or check) works better with .Renviron in general. If you want to set a custom library path, I recommend that you set it via R_LIBS_USER in .Renviron instead.

[github-actions[bot]]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.