Filter extensions fail to load if source file is in a folder that begins with underscore

[corcoted]

Bug description

If a .qmd source file is in a project subfolder that begins with an underscore (e.g. ./_folder/file.qmd), then quarto filter extensions do not load because quarto looks for the extensions in the wrong place. If the folder name is changed, the file does render properly. If extensions are omitted in the qmd file, then rendering occurs normally. The failure does not seem to depend on which extension is being loaded.

quarto render ./_folder/file.qmd

To obvious workaround is to change the folder name, but I'm using the underscore to prevent the contents of that specific folder from rendering during a batch render of the whole project.

Steps to reproduce

• Put the file below in the project subfolder _folder.

---
title: "Doesn't render"
format: pdf
filters:
  - latex-environment
environments:
  - center
---

:::{.center}
This file does not render if located in a folder whose name begins with _.
:::

• Install the latex-environment filter from the project base folder: quarto add quarto-ext/latex-environment
• Attempt to render using CLI: quarto render ./_folder/file.qmd

Expected behavior

The file should render to file.pdf

Actual behavior

The following error message (picking out the relevant part after the "pandoc" and "metadata" blocks)

Error running filter /opt/quarto/share/filters/main.lua:
Error running filter ~/project/_folder/latex-environment:
Could not find executable ~/project/_folder/latex-environment
stack traceback:
        /opt/quarto/share/filters/main.lua:4030: in function </opt/quarto/share/filters/main.lua:4009>
        [C]: in ?
        [C]: in method 'walk'
        /opt/quarto/share/filters/main.lua:171: in function 'run_emulated_filter'
        /opt/quarto/share/filters/main.lua:449: in local 'callback'
        /opt/quarto/share/filters/main.lua:454: in upvalue 'run_emulated_filter_chain'
        /opt/quarto/share/filters/main.lua:495: in function </opt/quarto/share/filters/main.lua:476>
stack traceback:
        /opt/quarto/share/filters/main.lua:171: in function 'run_emulated_filter'
        /opt/quarto/share/filters/main.lua:449: in local 'callback'
        /opt/quarto/share/filters/main.lua:454: in upvalue 'run_emulated_filter_chain'
        /opt/quarto/share/filters/main.lua:495: in function </opt/quarto/share/filters/main.lua:476>

Note in the second line of the error message that its looking in the wrong folder for the filter.

• Looking in ~/project/_folder/latex-environment
• Should look in ~/project/_extensions/quarto-ext/latex-environment

To reiterate, if the .qmd file is moved to a different subfolder the rendering succeeds (e.g. ./folder instead of ./_folder).

Your environment

• IDE: none
• OS: Ubuntu 22.04.3 LTS (jammy) in WSL
• LaTeX: XeLaTeX, Version 3.141592653-2.6-0.999995 (TeX Live 2023)

Quarto check output

[✓] Checking versions of quarto binary dependencies... Pandoc version 3.1.1: OK Dart Sass version 1.55.0: OK [✓] Checking versions of quarto dependencies......OK [✓] Checking Quarto installation......OK Version: 1.3.450 Path: /opt/quarto/bin

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK Version: 3.11.4 (Conda) Path: ~/project/env/bin/python Jupyter: 5.3.1 Kernels: python3

(-) Checking Jupyter engine render....0.00s - Debugger warning: It seems that frozen modules are being used, which may 0.00s - make the debugger miss breakpoints. Please pass -Xfrozen_modules=off 0.00s - to python to disable frozen modules. 0.00s - Note: Debugging will proceed. Set PYDEVD_DISABLE_FILE_VALIDATION=1 to disable this validation. [✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........(None)

[cscheid]

Thanks for the report, I can repro it.

I'm pretty sure that what's happening here is that if a file is in a location that is ignored, then we treat that location as if it weren't in the project. Which means that we resolve the extensions as if the file were a standalone file. You can verify that this behavior goes away by adding

  render:
    - "*.qmd"
    - "_test/test.qmd"

to your project. (Of course, that will mean that _test/test.qmd will now be rendered every time you call quarto render.)

I understand that this is counterintuitive (I'll let @jjallaire or @dragonstyle confirm), but I believe this is by design, and that _folder targets in general should not be used as a "please only render this when I ask you to". That's what the caching and freezing options are for.

Quarto (by design) treats folders with underscores as "does not belong to project", so that those can be used indirectly as eg. targets of {{< include _includes/test.md >}} shortcodes.

[corcoted]

It sounds like the take home message is that I should explicitly state my render targets in the project _quarto.yml file. Not a problem, although the _folder loophole was nice when it worked.

[jjallaire]

Yes, the _folder is considered "not part of the project" so rendering an individual file within it does not use project config or extensions. You could either explicitly list it or you could install extensions locally into the _folder directory.

[corcoted]

Update. Adding explicit render targets in the _quarto.yml file did not work. I just want to double check if I'm getting the expected behavior.

I added the following:

project:
  render:
    - includedfolder1/*.qmd
    - includedfolder2/*.qmd

Then put my "hidden" files in another folder unlistedfolder. I can individually render targets in one of the included folders, quarto render includedfolder1/test.qmd, but rendering targets in unlistedfolder fails like before: quarto render unlistedfolder/test2.qmd. Quarto is still looking for extensions inside unlistedfolder.

I think all of this is exposing bad project design on my part. My intention was to put files in unlistedfolder that I want to render locally, but that I do not want pushed to the remote server when I quarto publish gh-pages. It's convenient to have all of these in one project because the public and private files share common assets, but I understand if that's contrary to the intended use case.

This is a long way of asking, is there a way to prevent certain targets from being published?

[cscheid]

This is a long way of asking, is there a way to prevent certain targets from being published?

I don't think that's currently the case; "publish" for quarto just means "take the rendered output and upload it to a service". I think that what you're trying to do in general --- to mix individual and project pages while sharing assets --- is not a use-case that we originally considered.

It's convenient to have all of these in one project because the public and private files share common assets

It's possible that project profiles do exactly what you want: https://quarto.org/docs/projects/profiles.html