Compile quarto document in temporary directory in R fails when using `output_file = ...`

rkrug commented 1 year ago

Bug description

# This will create a file named "template.qmd" in the current folder.
# Used in: the "template.qmd" is a parametrized report and in a package. The report
# will be compiled in a temporary directly which will be cleaned afterwards.

f <- c("---", "title: \"Test\"", "format: html", "---", "", "## Set a variable",
  "", "```{r}", "OK <- FALSE", "```", "", "## Running Code depending on value of `OK`",
  "", "```{r}", "#| eval: !expr OK", "print(\"OK is TRUE - otherwise you won't see anything here.\")",
  "```", "")
writeLines(f, "template.qmd")

name <- "Test_Report"

output_format <- "html"
output_file <- paste0(name, ".html")

tmpdir <- tempfile()
dir.create(tmpdir)
template <-  file.path(tmpdir, "template.qmd")

file.copy(
  file.path(".", "template.qmd"),
  template
)

report <- quarto::quarto_render(
  input = template,
  output_format = output_format,
  output_file = output_file
)

# Here the error occurs:
#
# processing file: template.qmd
# |..............                                                        |  20%
# ordinary text without R code
#
# |............................                                          |  40%
# label: unnamed-chunk-1
# |..........................................                            |  60%
# ordinary text without R code
#
# |........................................................              |  80%
# label: unnamed-chunk-2 (with options)
# List of 1
# $ eval: logi FALSE
#
# |......................................................................| 100%
# ordinary text without R code
#
#
# output file: template.knit.md
#
# pandoc --output ../../../../../../../Users/rainerkrug/tmp/test/Test_Report.html
# to: html
# standalone: true
# section-divs: true
# html-math-method: mathjax
# wrap: none
# default-image-extension: png
#
# metadata
# document-css: false
# link-citations: true
# date-format: long
# lang: en
# title: Test
#
# pandoc: ../../../../../../../Users/rainerkrug/tmp/test/Test_Report.html: openFile: does not exist (No such file or directory)
# Error in `processx::run(quarto_bin, args, echo = TRUE)`:
#   ! System command 'quarto' failed
# ---
#   Exit status: 1
# stdout & stderr: <printed>
#   ---
#   Type .Last.error to see the more details.

# And I would like to continuye as follows:

file.copy(
  file.path(tmpdir, output_file),
  file.path(".", output_file),
)
unlink(tmpdir)

> sessionInfo()
R version 4.2.2 (2022-10-31)
Platform: x86_64-apple-darwin17.0 (64-bit)
Running under: macOS Ventura 13.1

Matrix products: default
LAPACK: /Library/Frameworks/R.framework/Versions/4.2/Resources/lib/libRlapack.dylib

locale:
[1] en_US.UTF-8/en_US.UTF-8/en_US.UTF-8/C/en_US.UTF-8/en_US.UTF-8

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

loaded via a namespace (and not attached):
 [1] Rcpp_1.0.9      ps_1.7.2        digest_0.6.30   later_1.3.0     R6_2.5.1        jsonlite_1.8.4 
 [7] evaluate_0.19   rlang_1.0.6     cli_3.6.0       rstudioapi_0.14 rmarkdown_2.18  tools_4.2.2    
[13] yaml_2.3.6      xfun_0.36       fastmap_1.1.0   compiler_4.2.2  processx_3.8.0  htmltools_0.5.3
[19] knitr_1.41      quarto_1.2     

> quarto::quarto_version()
[1] ‘1.2.313’
> pandoc::pandoc_version()
[1] ‘2.19.2’

The example runs on Ubuntu, but not on a Mac.

Checklist

[X] Please include a minimal, fully reproducible example in a single .qmd file? Please provide the whole file rather than the snippet you believe is causing the issue.
[X] Please format your issue so it is easier for us to read the bug report.
[ ] Please document the RStudio IDE version you're running (if applicable), by providing the value displayed in the "About RStudio" main menu dialog?
[X] Please document the operating system you're running. If on Linux, please provide the specific distribution.

cderv commented 1 year ago

Solutions ?

# And I would like to continuye as follows:
file.copy(
 file.path(tmpdir, output_file),
 file.path(".", output_file),
)

Currently your output file is created in the working directory where quarto_render() is ran and not in the temp dir. (see below in details)

I think you need to explicitly run your document in the temp directory. You should try this

xfun::in_dir(dirname(template), 
            # run Quarto in the directory of the input file
             report <- quarto::quarto_render(
              # run the input file
               input = basename(template),
               output_format = output_format,
              # output file will be created in the temp directory
               output_file = output_file
             )
)

# You can then copy it. 
file.copy(
  file.path(tmpdir, output_file),
  file.path(".", output_file),
)

So I think this is a matter of workflow and the right pattern.

Details about the issue

Currently, I believe passing a relative pass to --output will mean relative to directory where quarto is ran, and not input dir. So this is equivalent to running

quarto render full/path/to/temp/file.qmd --output file-relative-to-current-working-dir.html

However, Pandoc converter works based on input directory. so it will transform file-relative-to-current-working-dir.html to a relative path to file.qmd

On my Windows environment, this is working ok, as the relative path is build correctly

Output created: ..\..\..\..\..\Documents\Test_Report.html

for you I believe the relative path is not ok

pandoc: ../../../../../../../Users/rainerkrug/tmp/test/Test_Report.html: openFile: does not exist (No such file or directory)

rkrug commented 1 year ago

Details about the issue

Currently, I believe passing a relative pass to --output will mean relative to directory where quarto is ran, and not input dir. So this is equivalent to running
quarto render full/path/to/temp/file.qmd --output file-relative-to-current-working-dir.html

The parameter --output must not contain any path, just the filename. So if I use `./somename.html' I =get the following error 'ERROR: --output option cannot specify a relative or absolute path'

However, Pandoc converter works based on input directory. so it will transform file-relative-to-current-working-dir.html to a relative path to file.qmd

On my Windows environment, this is working ok, as the relative path is build correctly
Output created: ..\..\..\..\..\Documents\Test_Report.html

it is also working on Ubuntu.

So there is a bug in the Mac section when getting the path for pandoc.

for you I believe the relative path is not ok

pandoc: ../../../../../../../Users/rainerkrug/tmp/test/Test_Report.html: openFile: does not exist (No such file or directory)

At the moment I am using the approach of renaming the input .qmd file to the name I want to have finally when copying the template to the temp directory, and not using the output_file (--output) parameter / argument.

cderv commented 1 year ago

The parameter --output must not contain any path, just the filename. So if I use `./somename.html' I =get the following error 'ERROR: --output option cannot specify a relative or absolute path'

Yes it must be the output filename, which will be created in the working where quarto render is run. So in your R code example, using xfun::in_dir() to change working dir for the render to the temp dir should work.

But you mean the piece of code with xfun::in_dir() does not work for you ? It does for me on Windows. So somehing on MacOS 🤔

At the moment I am using the approach of renaming the input .qmd file to the name I want to have finally when copying the template to the temp directory, and not using the output_file (--output) parameter / argument.

Can you confirm that you have issue outside of R too, using quarto at command line ?

rkrug commented 1 year ago

Yes:

15:41 $ quarto render /var/folders/5k/g8z_7q3d6vb54m8k0cjy940r0000gq/T//RtmpkrZjRl/file139dc75bc40f8/template.qmd --output Report.html

processing file: template.qmd
  |..............                                                        |  20%
  ordinary text without R code

  |............................                                          |  40%
label: unnamed-chunk-1
  |..........................................                            |  60%
  ordinary text without R code

  |........................................................              |  80%
label: unnamed-chunk-2 (with options)
List of 1
 $ eval: logi FALSE

  |......................................................................| 100%
  ordinary text without R code

output file: template.knit.md

pandoc --output ../../../../../../../Users/rainer/Report.html
  to: html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png

metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Test

pandoc: ../../../../../../../Users/rainer/Report.html: openFile: does not exist (No such file or directory)

The path points back to the current directory, but the file Report.html is not there (neither template.html.

15:43 $ ls -la /var/folders/5k/g8z_7q3d6vb54m8k0cjy940r0000gq/T//RtmpkrZjRl/file139dc75bc40f8/
total 8
drwxr-xr-x@ 4 rainer  staff  128 Jan 17 15:43 .
drwx------@ 7 rainer  staff  224 Jan 17 15:41 ..
-rw-r--r--@ 1 rainer  staff  214 Jan 17 15:41 template.qmd
drwxr-xr-x  3 rainer  staff   96 Jan 17 15:43 template_files

cderv commented 1 year ago

Thanks a lot for the non-R example.

I believe this could be related to the fact that you are not having input and output on same "mount", first is on /var and last is on /Users (I think?). I think this could be the issue seen. I am not sure Pandoc even supports that. Are the tests you've done on Linux having the same situation ?

Correct pattern I think would be to change the current directory

# being in your user directory (/Users/rainer/ ? )
cd /var/folders/5k/g8z_7q3d6vb54m8k0cjy940r0000gq/T/RtmpkrZjRl/file139dc75bc40f8/
quarto render template.qmd --output Report.html
cp Report.html /Users/rainer/Report.html
cd /Users/rainer/

@dragonstyle as a Mac user, do you have more insights regarding using the remp dir to render ?

rkrug commented 1 year ago

They are on the same volume - the Mac only has one HDD, and as far as I understand, the OS is on a different read-only volume volume than the data (on a read-write mount). So the /var directly is presumably on the same volume as the data

cderv commented 1 year ago

They are on the same volume - the Mac only has one HDD, and as far as I understand, the OS is on a different read-only volume volume than the data (on a read-write mount). So the /var directly is presumably on the same volume as the data

Sorry what I meant is that creating a relative path from one to the other is maybe not possible ? I feel that somewhere a relative path to output is built from input.qmd and it created this ../../../../../../../Users/rainerkrug/tmp/test/Test_Report.html which may not be valid it seems;

I don't use Mac so I'll let @dragonstyle help with that - Thanks for all the information

rkrug commented 1 year ago

Executing ls -la ../../../../../../../Users/rainer/ is showing me the content of the current directory - so on Bash it is working.

It is just that the file 'Test_Report.html' is not there. So it is possibly a problem in quarto, that the copying / renaming is not working?

rkrug commented 1 year ago

Oh: using debug gives:

16:14 $ quarto render /var/folders/5k/g8z_7q3d6vb54m8k0cjy940r0000gq/T//RtmpkrZjRl/file139dc75bc40f8/template.qmd --output Report.html --debug

processing file: template.qmd
  |..............                                                        |  20%
  ordinary text without R code

  |............................                                          |  40%
label: unnamed-chunk-1
  |..........................................                            |  60%
  ordinary text without R code

  |........................................................              |  80%
label: unnamed-chunk-2 (with options)
List of 1
 $ eval: logi FALSE

  |......................................................................| 100%
  ordinary text without R code

output file: template.knit.md

pandoc --output ../../../../../../../Users/rainer/Report.html
  to: html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png

metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Test

pandoc: ../../../../../../../Users/rainer/Report.html: openFile: does not exist (No such file or directory)
✔ ~
16:19 $ ls -la /var/folders/5k/g8z_7q3d6vb54m8k0cjy940r0000gq/T//RtmpkrZjRl/file139dc75bc40f8/
total 16
drwxr-xr-x@  5 rainer  staff  160 Jan 17 16:19 .
drwx------@ 16 rainer  staff  512 Jan 17 16:17 ..
-rw-r--r--   1 rainer  staff  258 Jan 17 16:19 template.md
-rw-r--r--@  1 rainer  staff  214 Jan 17 15:41 template.qmd
drwxr-xr-x   3 rainer  staff   96 Jan 17 15:43 template_files
✔ ~
16:20 $

cderv commented 1 year ago

Thanks for the additional information !

It is just that the file 'Test_Report.html' is not there. So it is possibly a problem in quarto, that the copying / renaming is not working?

Possibly with Pandoc which get this relative file path for its --output argument I think 🤔

RegalPlatypus commented 1 year ago

Hello, I believe I'm encountering the same bug. I'm using quarto::quarto_render to pass a param entered in a shiny app to the qmd, the output of which is returned via downloadHandler().

I receive the following error:

ERROR: --output option cannot specify a relative or absolute path
Warning: Error in processx::run(quarto_bin, args, echo = TRUE): ! System command 'quarto.exe' failed

Note: This same format has worked when rendering an Rmarkdown file rather than a Quarto document.

The app:

library(shiny)
library(showtext)
#> Loading required package: sysfonts
#> Loading required package: showtextdb
library(gt)
library(lubridate)
#> 
#> Attaching package: 'lubridate'
#> The following objects are masked from 'package:base':
#> 
#>     date, intersect, setdiff, union
library(qbr)
library(tidyverse)
library(quarto)

ui <- fluidPage(

    titlePanel("My Title"),

    sidebarLayout(
      sidebarPanel(width = 3,
        textInput(inputId = "user.string", label = "Enter a string:"),
        br(),
        downloadButton(outputId = "report", label = "Generate Report:"),
      ),
      mainPanel(

      )
    )
)

server <- function(input, output) {
  dynamic.filename <- paste0("Cages_", format(Sys.Date(), "%y%m%d"), format(Sys.time(), "%H%M%S"), ".html")

  output$report <- downloadHandler(
    filename = dynamic.filename,
    content = function(file) {

      temp_file <- tempfile(fileext = ".html")

      quarto::quarto_render(
        "checklist.qmd",
        output_format = "html",
        output_file = temp_file,
        execute_params = list(userkey = input$user.string)
      )

      file.copy(temp_file, file)

    }
  )
}

# Run the application 
shinyApp(ui = ui, server = server)
#> 
#> Listening on http://127.0.0.1:5216

^{Created on 2023-07-03 with reprex v2.0.2}

The QMD file "checklist.qmd":

---
format:
  html
params:
  user.string: NA
---

```{r}
#| label: setup
#| include: false
knitr::opts_chunk$set(echo = TRUE, dev = "png", cache = FALSE)

#| label: head
#| message: false
#| warning: false
#| include: false

rendered.string <- params$user.string

# Generic Heading

My string was: `r rendered.string`

cderv commented 1 year ago

@RegalPlatypus The error you get (--output option cannot specify a relative or absolute path) is on purpose. --output flag in quarto cannot take paths, relative nor absolute. Only a file name. So output-file in Quarto and output_file equivalent in R function can only take a filename.

This has been added in https://github.com/quarto-dev/quarto-cli/issues/2440 to fix some issues.

output-dir with Quarto Project is the way to store output in another location.

What you can do is render the document with Quarto and then move it to another location.

Regarding usage in Shiny, the pattern has always been to copy the report and its resources to a temporary directory and then render there (See doc with R Markdown example: https://shiny.posit.co/r/articles/build/generating-reports/)

I would suggest trying this.

BTW this is slightly different than this initial issue - please next time open a new issue or discussion (with maybe a link to another issue you think is related). We prefer for issue to be scoped in order to keep discussion clean and not mix topic. Thank you !

RegalPlatypus commented 1 year ago

@cderv Thanks for the timely, polite, and excellent help! I was able to publish the app and have it render and save as expected with one exception: quarto isn't successfully using my CSS file. But, as you suggested, I'll open a discussion for that.

benyamindsmith commented 3 weeks ago

This is a fix that I have for doing this in my package. Seems to work pretty well.

temp_file<- tempfile(fileext = ".docx") |> basename()
quarto::quarto_render(qmd_file,output_format = "docx", output_file=temp_file)
unlink(temp_file)

quarto-dev / quarto-cli