Support symlinked extension directory in `_extensions/`

jimjam-slam commented 4 months ago

Bug description

In https://github.com/quarto-dev/quarto-cli/discussions/8579 I have been discussing the best way to structure a repository for a Quarto extension alongside a documentation website that uses that extension. @coatless has had success symlinking the doc website's extension folder (whether for a single extension or several) to the parent folder (eg. so if the documentation website is in /docs, /docs/_extensions/testproject -> /_extensions/testproject).

I'm trying to pursue this option, but I find that if my extension involves a custom project type (see reprex extension), I get an error when I render the documentation website.

Steps to reproduce

Clone the reprex extension, then:

$ cd docs
$ quarto render
ERROR: Unsupported project type testproject

Stack trace:
    at projectType (file:///Users/rensa/Applications/quarto/bin/quarto.js:37943:15)
    at projectContext (file:///Users/rensa/Applications/quarto/bin/quarto.js:72786:30)
    at eventLoopTick (ext:core/01_core.js:183:11)
    at async render (file:///Users/rensa/Applications/quarto/bin/quarto.js:81779:19)
    at async Command.fn (file:///Users/rensa/Applications/quarto/bin/quarto.js:81954:32)
    at async Command.execute (file:///Users/rensa/Applications/quarto/bin/quarto.js:8104:13)
    at async quarto (file:///Users/rensa/Applications/quarto/bin/quarto.js:114999:5)
    at async file:///Users/rensa/Applications/quarto/bin/quarto.js:115017:9

Expected behavior

If I delete the symlink and manually copy the files into the doc website's extension folder, it renders fine:

$ cd _extensions
$ rm testproject
$ cp -r ../../_extensions/testproject .
$ quarto render
[1/2] index.qmd
[2/2] about.qmd

Output created: _site/index.html

Your environment

IDE: none (using terminal)
macOS: 14.2.1
Quarto: have tested on both 1.4.550 and pre-release (at commit 581c7e54d)

Quarto check output

Using the pre-release:

Quarto 99.9.9
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.1.11: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.41.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 99.9.9
      Path: /Users/rensa/code/quarto-cli/package/dist/bin

Check file:///Users/rensa/code/quarto-cli/src/resources/vendor/deno-land/x/puppeteer@9-0-2/mod.ts
[✓] Checking tools....................OK
      TinyTeX: (external install)
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/rensa/Library/TinyTeX/bin/universal-darwin
      Version: 2021

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.9.7 (Conda)
      Path: /Users/rensa/miniforge3/bin/python
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with conda install jupyter

[✓] Checking R installation...........OK
      Version: 4.2.1
      Path: /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources
      LibPaths:
        - /Users/rensa/Library/R/arm64/4.2/library
        - /Library/Frameworks/R.framework/Versions/4.2-arm64/Resources/library
      knitr: 1.42
      rmarkdown: 2.21

[✓] Checking Knitr engine render......OK

cderv commented 4 months ago

It is not just related to project configuration. We just currently do not read an extension as a symlink - it is not resolved and we expect a directory as we check extensionDir.isDirectory https://github.com/quarto-dev/quarto-cli/blob/78e7f847d49dfd029252a8c4b2fc80be33456436/src/extension/extension.ts#L409-L448

So currently extensions as a symlink is not supported.

Issue can be reproduced above by just

cd docs/
quarto list extensions
ERROR: Unsupported project type testproject

As a quick test, this could be a way to handle the reading for the symlink (and solves this specific issue)

diff --git a/src/extension/extension.ts b/src/extension/extension.ts
index ac081998c..47b5f8036 100644
--- a/src/extension/extension.ts
+++ b/src/extension/extension.ts
@@ -415,10 +415,14 @@ export async function readExtensions(
   const extensions: Extension[] = [];
   const extensionDirs = Deno.readDirSync(extensionsDirectory);
   for (const extensionDir of extensionDirs) {
-    if (extensionDir.isDirectory) {
-      const extFile = extensionFile(
-        join(extensionsDirectory, extensionDir.name),
-      );
+    if (extensionDir.isDirectory || extensionDir.isSymlink) {
+      let extDir = extensionDir.name;
+      if (extensionDir.isSymlink) {
+        extDir = Deno.readLinkSync(
+          join(extensionsDirectory, extensionDir.name),
+        );
+      }
+      const extFile = extensionFile(join(extensionsDirectory, extDir));
       if (extFile) {
         // This is a directory that contains an extension
         // This represents an 'anonymous' extension that doesn't

But I don't know enough about the extension context and resource handling to know if having other extensions files as a symlink would be problematic (thinking about templates, resources, theme files, ...)

I believe in general we do try to prevent any path handling for files that escape the root directory. Anything should be local to project usually.

Related topic on symlink

https://github.com/quarto-dev/quarto-cli/issues/2001 (old discussion about copying symlinked resources)
https://github.com/quarto-dev/quarto-cli/issues/4167 (some issues with symlinked content and quarto render)

jimjam-slam commented 4 months ago

Makes sense! I might experiment with a project pre-render script to copy the extension in from the repo root (the extension also supports single document renders, but that wouldn't be important for this documentation site). Just trying to find a solution for an extension documentation site using that extension that keeps a single source of truth within the repo!

jimjam-slam commented 4 months ago

Unfortunately the project type is rejected before any pre-render script is executed, so I can't bolt one on to the extension site. I'd probably have to use something like cp -r ../_extensions/sverto /_extensions/sverto && quarto render in that case.

mcanouil commented 4 months ago

@jimjam-slam Did you try with Quarto 1.5? There are some changes in Quarto related to pre-render scripts. (see the highlight section of the changelog)

cscheid commented 4 months ago

@mcanouil he did:

Quarto: have tested on both 1.4.550 and pre-release (at commit https://github.com/quarto-dev/quarto-cli/commit/581c7e54dbf5ecc6638553d28592f6388fe6ec65)

mcanouil commented 4 months ago

Was referring to the last comment about pre-render.

Unfortunately the project type is rejected before any pre-render script is executed, so I can't bolt one on to the extension site. I'd probably have to use something like cp -r ../_extensions/sverto /_extensions/sverto && quarto render in that case.

jimjam-slam commented 4 months ago

Adding the pre-render script I've tried with the prerelease above but not with the 1.4 stable 🙂

I should add that extension documentation is a fairly narrow use case, and if you decide that the benefits of not following symlinks outweigh the downsides, I can definitely make do 😊

coatless commented 4 months ago

Huh, that's an interesting development. Tis' always good to know I'm using an edge case 😉

Thinking out loud, we recently added a make sync directive in {quarto-countdown} that could be used to copy the extension to relative directories. That would address the local development environment portion until the pre-render script hiccup could be fixed in 1.5 dev builds.

Outside of that, another approach would be for the CI run to copy the development extension over to other directories, e.g.

      - name: Copy extension for demos
        run: |
          mkdir -p demos/_extensions/
          cp -r ./_extensions demos/_extensions/

(Note, you could employ the make directive inside of the CI script as well.)

jimjam-slam commented 4 months ago

Yeah, having the CI copy the extension is what I've settled on! Note that I specifically had to specify shell: bash in order to get it to work on a Windows runner (it defaulted to PowerShell otherwise, which has a different syntax).

I should also note that my project pre-render script tries to avoid running duplicate files, so I'm also learning how much easier it is when you assume no symlinks in your relative paths 😅

quarto-dev / quarto-cli