Bash code chunks with profile enabled don't work when cache enabled

[mdehollander]

I have a Quartro Notebook (knitr flavor) with bash code chunks. I run them with the profile option enabled in order to run specific commands like mamba. Some parts take long to execute so I enabled caching. With a global enabled engine.opts='-l' and caching enabled, rendering fails with:

==> quarto preview conda.qmd --to html --no-watch-inputs --no-browse

processing file: conda.qmd
  |..........................................          |  80% [unnamed-chunk-1]running: bash  -c 'mamba env list'
Error in `system2()`:
! error in running command
Backtrace:
  1. global .main()
  2. execute(...)
  3. rmarkdown::render(...)
  4. knitr::knit(knit_input, knit_output, envir = envir, quiet = quiet)
  5. knitr:::process_file(text, output)
     ...
 13. knitr (local) engine(options)
 14. base::tryCatch(...)
 15. base (local) tryCatchList(expr, classes, parentenv, handlers)
 16. base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 17. value[[3L]](cond)

Quitting from lines 16-17 [unnamed-chunk-1] (conda.qmd)

Execution halted

This is the quatro document:

---
title: "Conda"
format: html
editor: visual
engine: knitr
execute: 
  cache: true
---

{r setup} library(knitr) opts_chunk$set(engine.opts='-l')

{bash} mamba env list

Either setting cache: false or explicitly adding engine.opts = '-l' to the code chunk works ok:

{bash, engine.opts='-l'} mamba env list

Ideally I would like to setup the bash profiling only once, and just use plain bash code blocks without specifying the engine.opts every time. I guess there is a bug here when caching is enabled.

[cderv]

Can you format the code part correctly ? Thank you ! https://quarto.org/bug-reports.html#formatting-make-githubs-markdown-work-for-us

This will help copy and paste for reproduction.

I am wondering here is this is Quarto or rather knitr limitation.

[mdehollander]

Reformatted! Let me know if this is an issue for quarto or knitr.

[cderv]

You will need to provide more. I can't reproduce with latest version, but I did try with Mamba; I don't have it.

Can you provide quarto check ?

Though, if knitr caches, then it shouldn't call system2 again. and the error comes from there.

So it seems something related to your specific tools or context. We'll need more.

[mdehollander]

I tried with quarto check​ and that works indeed. I think the problem is when using programs that require the profile to be activate with the engine.opts -l and otherwise don't work.

---

From: Christophe Dervieux @.> Sent: Tuesday, June 4, 2024 4:36 PM To: quarto-dev/quarto-cli @.> Cc: Hollander, Mattias de @.>; Author @.> Subject: Re: [quarto-dev/quarto-cli] Bash code chunks with profile enabled don't work when cache enabled (Issue #9852)

You will need to provide more. I can't reproduce with latest version, but I did try with Mamba; I don't have it.

Can you provide quarto check ?

Though, if knitr caches, then it shouldn't call system2 again. and the error comes from there.

So it seems something related to your specific tools or context. We'll need more.

— Reply to this email directly, view it on GitHubhttps://github.com/quarto-dev/quarto-cli/issues/9852#issuecomment-2147702676, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACAFWGVPKXLBSGIOATQD6LZFXGIBAVCNFSM6AAAAABIWWFBMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXG4YDENRXGY. You are receiving this because you authored the thread.Message ID: @.***>

[cscheid]

I tried with quarto check​ and that works indeed

What @cderv is asking you is for your output from quarto check.

[mdehollander]

Here you go:

Quarto 1.4.553

[✓] Checking versions of quarto binary dependencies... Pandoc version 3.1.11: OK Dart Sass version 1.69.5: OK Deno version 1.37.2: OK

[✓] Checking versions of quarto dependencies......OK

[✓] Checking Quarto installation......OK Version: 1.4.553 Path: /usr/lib/rstudio-server/bin/quarto/bin

(|) Checking tools.................... [✓] Checking tools....................OK TinyTeX: (not installed) Chromium: (not installed)

 [✓] Checking LaTeX....................OK Tex: (not detected)

(|) Checking basic markdown render.... (/) Checking basic markdown render.... (-) Checking basic markdown render.... () Checking basic markdown render.... (|) Checking basic markdown render.... (/) Checking basic markdown render.... (-) Checking basic markdown render.... () Checking basic markdown render.... [✓] Checking basic markdown render....OK

 [✓] Checking Python 3 installation....OK Version: 3.8.10 Path: /usr/bin/python3 Jupyter: (None)

  Jupyter is not available in this Python installation.
  Install with python3 -m pip install jupyter

  There is an unactivated Python environment in .venv. Did you forget to activate it?

(|) Checking R installation..........

(/) Checking R installation........... (-) Checking R installation........... () Checking R installation........... [✓] Checking R installation...........OK Version: 4.4.0 Path: /opt/R/4.4.0/lib/R LibPaths:

• /home/XXXX/R/x86_64-pc-linux-gnu-library/4.4
• /opt/R/4.4.0/lib/R/library knitr: 1.47 rmarkdown: 2.27

(|) Checking Knitr engine render...... (/) Checking Knitr engine render...... (-) Checking Knitr engine render...... () Checking Knitr engine render...... (|) Checking Knitr engine render...... (/) Checking Knitr engine render...... (-) Checking Knitr engine render...... () Checking Knitr engine render...... (|) Checking Knitr engine render...... (/) Checking Knitr engine render...... (-) Checking Knitr engine render...... () Checking Knitr engine render...... (|) Checking Knitr engine render...... [✓] Checking Knitr engine render......OK

---

From: Carlos Scheidegger @.> Sent: Tuesday, June 4, 2024 5:23 PM To: quarto-dev/quarto-cli @.> Cc: Hollander, Mattias de @.>; Author @.> Subject: Re: [quarto-dev/quarto-cli] Bash code chunks with profile enabled don't work when cache enabled (Issue #9852)

I tried with quarto check​ and that works indeed

What @cdervhttps://github.com/cderv is asking you is for your output from quarto check.

— Reply to this email directly, view it on GitHubhttps://github.com/quarto-dev/quarto-cli/issues/9852#issuecomment-2147812075, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AACAFWA6QDIQLXW22RNMEZLZFXLXJAVCNFSM6AAAAABIWWFBMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCNBXHAYTEMBXGU. You are receiving this because you authored the thread.Message ID: @.***>

[cderv]

Thanks. I don't see anything suspicious in your quarto check results regarding this specific issue.

I believe if there is an issue here, this would be with knitr directly. For this caching feature, and bash execution in quarto, it relies directly on knitr.

Here is what to test

• test.md

library(knitr)
opts_chunk$set(engine.opts='-l')

mamba env list

Then to run with `knitr::knit()` in your R console. It should give you: 
* First pass an execution
* Second pass no execution of bash

````R
> knitr::knit("test.md")

processing file: test.md
  |.........................................................................................................................| 100% [unnamed-chunk-1]running: bash -l -c "mamba env list"

output file: test.txt

[1] "test.txt"
> knitr::knit("test.md")

processing file: test.md

output file: test.txt

[1] "test.txt"

If this does not work in this context, then it is directly related to knitr

Be sure to clean you cache environment before doing the testing.

Thanks

[github-actions[bot]]

Thank you for using Quarto and reporting an issue!

Unfortunately, this issue is now considered stale because it has been opened since 14 days without providing a "working" reproducible example to help us investigate. If you are still facing the issue, please review the "Bug Reports" guide on how to provide a fully reproducible example as a self-contained Quarto document or a link to a Git repository. Without a reproducible example, it is unlikely that the issue will be addressed.

You can share a Quarto document using the following syntax, i.e., using more backticks than you have in your document (usually four ````).

````qmd
---
title: "Reproducible Quarto Document"
format: html
---

This is a reproducible Quarto document using `format: html`.
It is written in Markdown and contains embedded R code.
When you run the code, it will produce a plot.

```{r}
plot(cars)

The end.

[mdehollander]

Sorry, this issue missed my attention. Here is the output of knitr::knit() after I removed the cache dirs. It seems to work because the markdown output file has the expected output:

> knitr::knit("quarto-conda.qmd")

processing file: quarto-conda.qmd
  |...................................................................................................................| 100% [unnamed-chunk-1]running: bash -l -c 'mamba --version'

output file: quarto-conda.md

[1] "quarto-conda.md"
> knitr::knit("quarto-conda.qmd")

processing file: quarto-conda.qmd
  |...................................................................................................................| 100% [unnamed-chunk-1]running: bash -l -c 'mamba --version'

output file: quarto-conda.md

[1] "quarto-conda.md"

>

The produced Markdown files contains the bash output. I changed the command to show the version of the tool, since the output is shorter and the effect the same:

```r
library(knitr)
opts_chunk$set(engine.opts='-l')

mamba --version

## mamba 1.5.6
## conda 23.11.0

Clicking Render in Rstudio fails. Setting cache to false in the preamble and doing the render again works:

execute: cache: false

[cderv]

Thanks for trying.

Here is the output of knitr::knit() after I removed the cache dirs. It seems to work because the markdown output file has the expected output:

Does quarto-conda.qmd has cache enabled as I shared in example in my previous comment https://github.com/quarto-dev/quarto-cli/issues/9852#issuecomment-2149348985 ? Not the cache option on the cell

I see two times the progress bar so It seems that the cache is not taken into account.

I still feel this is something to understand if it works ok at knitr level, and only if so then we can look at Quarto level. The cache feature used by Quarto is directly the one by knitr

Can you confirm me the example you ran ?

[mdehollander]

So cache can be configured either globally or per cell. Before I had it only globally. Now I did it per cell so,

{bash, cache = TRUE} mamba --version

This show caching is working because the second time there is not progress bar

> knitr::knit("quarto-conda.qmd")

processing file: quarto-conda.qmd
  |...................................................................................................................| 100% [unnamed-chunk-1]running: bash -l -c 'mamba --version'

output file: quarto-conda.md

[1] "quarto-conda.md"
> knitr::knit("quarto-conda.qmd")

processing file: quarto-conda.qmd

output file: quarto-conda.md

[1] "quarto-conda.md"

I only get an error when I click the Render button in RStudio when the cache is globally enabled:

---
title: "quarto-conda"
execute: 
  cache: true
---

[mcanouil]

What if you render from the command line? (to rule out any possibility that it is an RStudio bug)

[mdehollander]

Good one. Same error:

> quarto::quarto_render("quarto-conda.qmd")

processing file: quarto-conda.qmd
  |..........................................          |  80% [unnamed-chunk-1]running: bash  -c 'mamba --version'
Error in `system2()`:
! error in running command
Backtrace:
  1. global .main()
  2. execute(...)
  3. rmarkdown::render(...)
  4. knitr::knit(knit_input, knit_output, envir = envir, quiet = quiet)
  5. knitr:::process_file(text, output)
     ...
 13. knitr (local) engine(options)
 14. base::tryCatch(...)
 15. base (local) tryCatchList(expr, classes, parentenv, handlers)
 16. base (local) tryCatchOne(expr, names, parentenv, handlers[[1L]])
 17. value[[3L]](cond)

Quitting from lines 17-18 [unnamed-chunk-1] (quarto-conda.qmd)
Execution halted

[cderv]

quarto::quarto_render("quarto-conda.qmd")

Did you run this outside of RStudio in a R console ? That would be the ultimate thing to look to rule out RStudio IDE behavior. R console or terminal inside RStudio IDE could inherit some configurations from the IDE itself. So using an external R console to test is better.

I'll look if we are doing anything specific that I missed when doing

execute: 
  cache: true

[mcanouil]

To be clear, run outside of RStudio and not in R:

quarto render quarto-conda.qmd

[mdehollander]

If I use the terminal command quarto render quarto-conda.qmd it always works, with or without the _cache or _files` dirs being present. Note I use the quarto binary provided by rstudio, but I run it inside the terminal inside RStudio.

If I remove the _cache or _filesdirs, the first time I use thequarto::quarto_render("quarto-conda.qmd")` command in the R console, it works as well. If I run it again, when it should start to to use the cache, it fails.

Same with the Render button in RStudio. Without any cache dirs it works. If they are present, it fails.

[mcanouil]

So, it only fails in RStudio environment?

[mdehollander]

Well, if I use the quarto R library.

Strange thing is that if I run the quarto in the terminal first, let it create the cache dirs, and then use quarto in R, it works, also for more times after it.

(weekend starts here) Thanks for the debugging help.

[cderv]

Thanks for the information. I'll investigate more with these new informations. Won't be an easy one to understand 🤔