ERROR: Include directive failed.

[kdheepak]

I tried to clone this repository and run quarto preview and I get an include directive failed error. I get the same error from the docs folder and if I try to follow the instructions in my personal project.

$ quarto preview
Preparing to preview
ERROR: Include directive failed.
  in file /Users/USERNAME/gitrepos/quarto-blog/sverto/docs/index.qmd,
  could not find file /Users/USERNAME/gitrepos/quarto-blog/sverto/docs/.sverto/index.qmd.

Error: Include directive failed.
  in file /Users/USERNAME/gitrepos/quarto-blog/sverto/docs/index.qmd,
  could not find file /Users/USERNAME/gitrepos/quarto-blog/sverto/docs/.sverto/index.qmd.
    at retrieveInclude (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:83837:23)
    at Object.directive (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:83873:15)
    at handleLanguageCells (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:57793:83)
    at async renderContexts (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:93180:45)
    at async renderFormats (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:95567:26)
    at async inputTargetIndex (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:92959:21)
    at async resolveInputTarget (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:93038:19)
    at async serveFiles (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:107848:24)
    at async serveProject (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:107486:30)
    at async Command.fn (file:///Users/USERNAME/Applications/quarto/bin/quarto.js:108047:13)

Do you know what might be happening here?

[jimjam-slam]

Hi @kdheepak! Generally, you won't want to run quarto preview on this repository's root directly—it's a template, and you'll want to use quarto use template 360-info/sverto to start a new Quarto project using it.

That said, it ought to work inside the docs folder, which is a Sverto project itself, as well as in your personal project if you have set Sverto up.

Can you verify that you've done the following steps when setting up your personal project?

1. quarto use template 360-info/sverto to import the Sverto files
2. npm install to install the npm packages needed by Sverto (and potentially your visualisation)?
3. quarto render to render the project? (quarto preview should run the render too, but it's possible that there's some sort of edge case which is causing the render not to happen).

Let me know how you go!

[jimjam-slam]

I've also noticed that the docs folder is inadvertently being brought in to new Sverto projects. I'll update the .quartoignore file to stop that from happening (tracking in #12), but you can delete that folder in the mean time if you're setting up a new project!

[kdheepak]

Thanks for the quick reply.

quarto use template 360-info/sverto

this doesn’t appear to do anything in an existing project? Do you mean add extension instead?

[kdheepak]

Okay, I'm able to do a fresh quarto use template 360-info/sverto, delete the docs folder and run quarto render. And that doesn't give the error any more.

But,

1) I'm not see any svelte visualizations?

I'm getting this error in the console.

2) When I try to copy over to an existing project the package.json file, package-lock.json file, the _extensions folder, and create a new .qmd file, I still get the same error.

I changed the shortcode in the file to this:

:::{}
{{< include /.sverto/svelte-and-pandoc/index.qmd >}}
:::

I even tried changed the quarto project from type: website to type: sverto and that didn't do anything.

[jimjam-slam]

@kdheepak You may need to use add extension first for an existing project, sorry! I haven't played too much with adding Sverto to existing projects.

You should end up with https://github.com/360-info/sverto/tree/main/_extensions/sverto copied into your project directory, plus:

• Circles.svelte
• examples.qmd (these first two are an example page and component)
• package.json
• package-lock.json
• _quarto.yml

You'll definitely need project type set to sverto in your quarto.yml (it inherits from the website project type, so it shouldn't muck up other aspects of your website).

Did you also run npm install? You should have a node_modules folder in your project directory once that's run.

What version of Quarto are you running? (quarto --version)

[jimjam-slam]

Is your index.qmd in a subfolder called svelte-and-pandoc? The include directive should be whatever your document's path is within the project, but with /.sverto prefixed.

So if you're in a home page at /index.qmd, the directive will be to /.sverto/index.qmd. If your page is at /posts/animals/cows.qmd, the directive will be for /.sverto/posts/animals/cows.qmd.

[kdheepak]

I'm also new to quarto, so maybe I'm doing something dumb?

If I do quarto install extension 360-info/sverto, I get an extensions folder that has 360-info/sverto:

$ tree _extensions/360-info
_extensions/360-info
└── sverto
   ├── _extension.yml
   ├── cleanup-transform.lua
   ├── compile-imports.ts
   ├── create-imports.lua
   ├── refresh.ts
   ├── rollup.config.js
   └── setupTypeScript.js

But I get no package.json or package-lock.json file.

[jimjam-slam]

I believe quarto install extension installs the content of the extension in _extensions/[org]/[name], but it doesn't copy template content in the extension repo root (like example.qmd or package.json) across.

Template content is generally optional, but in Sverto's case we need package.json, and it has to go in the project root, not in the extension folder due to the way npm works (you'll notice most of the other under-the-hood stuff is thrown inside the extension folder).

quarto use template should additionally copy this other content over, which is why I recommend using that command 🙂

[kdheepak]

quarto use template doesn't work in a existing folder though, which is why I was having problems.

I did use it in an empty folder, and copied over the files into my project and now I'm not getting this error about the include directive any more.

I'm getting a different error when it renders though:

[jimjam-slam]

Good to know! Are you able to show me the terminal output you're getting when you do quarto render now?

(I agree that we need a better way to add Sverto to existing projects, but I'm not sure that Quarto currently supports it in the case of starter templates! I'll update the Sverto docs to more specifically outline the install procedure in this case.)

[kdheepak]

Here's the output of quarto render

$ quarto render
_extensions/sverto/refresh.ts
_extensions/sverto/create-imports.lua

_extensions/sverto/compile-imports.ts

> svelte-app@1.0.0 build
> rollup -c /Users/kd/gitrepos/blog/_extensions/sverto/rollup.config.js

ℹ No Svelte imports to process; skipping compilation

[ 1/52] mac-osx-tweaks/index.qmd
[ 2/52] spotify-and-tmux/index.qmd
[ 3/52] welcome-to-my-blog/index.qmd
[ 4/52] raspberry-pi-powered-led-lamp/index.qmd

[kdheepak]

I copied over the exact example.qmd file. There are 2 things happening.

1) The entire content of the markdown file is duplicated.

2) In the second duplicated section of the rendered output in the HTML, I'm getting these errors:

[jimjam-slam]

I think I might need to see the source of your Quarto document here! Can you copy it in here? The rendered chunk doesn't look quite right to me. (Indeed, you shouldn't see import_svelte in the rendered document at all, because Sverto swaps it out internally to import the compiled Svelte JavaScript bundle.) The fact that it's still there, plus the ℹ No Svelte imports to process; skipping compilation message, tells me that the Svelte component isn't being detected properly.

[kdheepak]

Here's the branch where I just committed all the changes: https://github.com/kdheepak/blog/tree/sverto

This link should show the diff: https://github.com/kdheepak/blog/compare/sverto?expand=1

[kdheepak]

I'm not sure if this is important but the quarto use template 360-info/sverto creates a _extension/360-info/sverto extension but I copied it over to _extension/sverto like you have in the repo. I didn't change anything else. I copied the files over and then ran npm install.

[jimjam-slam]

Thanks! So yeah, I think the problem you're having is that you've put example.qmd in a subfolder:

https://github.com/kdheepak/blog/blob/8c8d3a8227f17d5d14c493934d861cd7cc6b0f9f/svelte-and-pandoc/index.qmd#L19

you'll need to update the path in the include directive, as it needs to match the path of the doc it's in (but with /.sverto prefixed). So instead of:

````markdown
:::{}
{{{< include /.sverto/example.qmd >}}}
:::

You want:

:::{}
{{{< include /.sverto/svelte-and-pandoc/example.qmd >}}}
:::

How does that work for you?

(You _may_ need to add the include directive to your other Quarto docs, even if you're not putting a Svelte component in them, but I'm not 100% sure. Let's see how we go with this change first!)

[kdheepak]

Isn't that what I already have here in this line?

https://github.com/kdheepak/blog/commit/8c8d3a8227f17d5d14c493934d861cd7cc6b0f9f#diff-6c031527ee06381635fc88707232815f66eded4468580bff5a3d392f56cd1fdaR8

[kdheepak]

The line you linked to is the Markdown example of what the shortcode include should look like. The line before what you linked to does what the documentation says (this is the very first line in the document).

[jimjam-slam]

Oops, sorry! You're right—the markdown block is for my documentation and isn't executed. Let me pull the project and see if I can reproduce.

[jimjam-slam]

Ah, it might take me some time to reproduce with your blog, as I don't have Julia installed. I'll try to as soon as I can, though!

[jimjam-slam]

Are you able to tell me which Quarto version you're on?

[kdheepak]

I'm using the latest pre-release!

$ quarto --version
1.3.260

[jimjam-slam]

Okay, I'm not sure if something has changed about the way Quarto's Lua filters work (possibly work related to Custom AST nodes?), but I also can no longer render my docs site on the latest pre-release either. Digging into the create-imports.lua filter, it looks like code blocks are being mis-identified.

I'm not sure if this is a regression on Quarto's part, or if I've just been overly presumptuous about the Lua filters. You might find more success if you go back to an earlier pre-release, but I'll see if I can produce a more minimal example for the Quarto folks!

[kdheepak]

I can help debug too! Can you give me a quick explanation of how this works to get me oriented?

[jimjam-slam]

Sure! There's a bit fiddliness with how Sverto works at present just because OJS isn't directly editable by regular Lua filters. So the way it works is this:

1. refresh.ts is a pre-project script that just wipes the .sverto folder if it's a full (whole-project) render
2. create-imports.lua is a pre-project script that does a few things:
   • firstly, it runs a filter that transforms the Quarto doc. The filter has two parts:
     1. it traverses code blocks, substituting import_svelte("path/Y.svelte") into import(path/Y.js)
     2. it removes the first element of the doc body (which ought to be the include directive). The doc with both of these transformations present is written out to .sverto/[doc path]
   • secondly, it adds any Svelte file paths it finds in these statements (like path/Y.svelte) to a text file in .sverto/, to send to the Svelte compiler later
3. compile-imports.ts is a pre-project script that sends all the found Svelte paths (de-deuplicated) to Rollup (with rollup.config.js to produce compiled JavaScript bundles.
4. cleanup-transform.lua is a regular Lua filter that basically substitutes the whole doc with just its first doc (so essentially it replaces the doc with its transformed version in .sverto/.

The problem is with create-imports.lua—when I drop a print at the top of the CodeBlock filter, it's not correctly identifying the code blocks :(

[kdheepak]

At least one problem is that the import_svelte line is being read as an inline Code element instead of a CodeBlock:

{
  "pandoc-api-version": [
    1,
    23
  ],
  "meta": {
    "categories": {
      "t": "MetaList",
      "c": [
        {
          "t": "MetaInlines",
          "c": [
            {
              "t": "Str",
              "c": "svelte"
            }
          ]
        },
        {
          "t": "MetaInlines",
          "c": [
            {
              "t": "Str",
              "c": "pandoc"
            }
          ]
        }
      ]
    },
    "date": {
      "t": "MetaInlines",
      "c": [
        {
          "t": "Str",
          "c": "2022-01-11T23:03:31-06:00"
        }
      ]
    },
    "title": {
      "t": "MetaInlines",
      "c": [
        {
          "t": "Str",
          "c": "Svelte"
        },
        {
          "t": "Space"
        },
        {
          "t": "Str",
          "c": "and"
        },
        {
          "t": "Space"
        },
        {
          "t": "Str",
          "c": "Pandoc"
        }
      ]
    }
  },
  "blocks": [
    {
      "t": "Para",
      "c": [
        {
          "t": "Code",
          "c": [
            [
              "",
              [],
              []
            ],
            "{ojs} Circles = import_svelte(\"Circles.svelte\")"
          ]
        }
      ]
    }
  ]
}

[kdheepak]

I added these two utility functions:

function string.starts_with(str, starts)
  return str:sub(1, #starts) == starts
end

function string.ends_with(str, ends)
  return ends == "" or str:sub(- #ends) == ends
end

And changed the preprocess_qmd_filter to the following:

local preprocess_qmd_filter = {
  Code = function(block)
    if string.starts_with(block.text, "{ojs}") then
      local svelte_import_syntax =
      "import%_svelte%(\"([%w;,/%?:@&=%+%$%-_%.!~%*'%(%)#]+)%.svelte\"%)"

      local block_text = block.text

      -- get the qmd_path from disk
      local current_qmd_path = read_file(".sverto/.sverto-current-qmd-folder")

      -- first, extract .svelte paths in import_svelte() statements
      for svelte_path in block_text:gmatch(svelte_import_syntax) do
        append_to_file(".sverto/.sverto-imports",
          current_qmd_path .. svelte_path .. ".svelte\n")
      end

      -- now change `import_svelte("X.svelte")` refs to `import("X.js")`
      -- TODO - neaten up relative paths instead of assuming we're going from
      -- /site_libs/quarto-ojs
      block.text = block_text:gsub(
        svelte_import_syntax,
        "import(\"./../../" .. current_qmd_path .. "%1.js\")")
    end
    return block
  end,
  -- search for `import_svelte("X.svelte")` refs in codeblocks and switch them
  -- to `import("X.js")`
  CodeBlock = function(block)

This does something now but still not the right thing.

For this input /svelte-and-pandoc/index.qmd file:

```{ojs}
Circles = import_svelte("Circles.svelte")

:::{#mycircles} :::

myCircles = new Circles.default({
  target: document.querySelector("#mycircles"),
  props: {
    data: [5, 15, 25, 17, 8]
  }
});

// here are some datasets...
allDatasets = new Map([
    ["Dataset A", [5, 15, 25, 17, 8]],
    ["Dataset B", [25, 5, 5]],
    ["Dataset C", [12, 5, 8, 21, 5]]
  ]);

viewof selectedDataset =
  Inputs.select(allDatasets, { label: "Selected dataset" });

myCircles.data = selectedDataset

this is the file that's in the `.sverto/svelte-and-pandoc/index.qmd` file

::: {#mycircles} :::

{ojs} myCircles = new Circles.default({ target: document.querySelector("#mycircles"), props: { data: [5, 15, 25, 17, 8] } });

{ojs} // here are some datasets... allDatasets = new Map([ ["Dataset A", [5, 15, 25, 17, 8]], ["Dataset B", [25, 5, 5]], ["Dataset C", [12, 5, 8, 21, 5]] ]);

{ojs} viewof selectedDataset = Inputs.select(allDatasets, { label: "Selected dataset" });

{ojs} myCircles.data = selectedDataset

[jimjam-slam]

Good pick! I wonder what changed on the Quarto (or Pandoc) side to cause the code block to be represented as an inline Code instead of a CodeBlock?

@cscheid might be able to advise here, since he helped me understand Quarto's render process in the first place.

[kdheepak]

It seems to only be doing it for {ojs} blocks.

[cscheid]

The easiest way to figure out when the change happened would be to bisect on the different quarto releases (or to install from git and explicitly git bisect).

I won't have time to do it this week, unfortunately.

[cscheid]

With that said, if the goal is simply to emit code blocks instead of identifying why quarto is now seeing Code elements instead of CodeBlock elements (I suspect it was the move to Pandoc 3; the {lang} syntax is no longer supported and we had to add workarounds), then this looks like it might work:

local preprocess_qmd_filter = {
  Code = function(block)
    -- ....
    -- ....
    -- instead of `return block`, call this:

    return pandoc.CodeBlock(code_text, code_attr) -- whatever the right thing is here
  end,
  -- search for `import_svelte("X.svelte")` refs in codeblocks and switch them
  -- to `import("X.js")`
  CodeBlock = function(block)

[cscheid]

We just learned what this bug is! We'll fix it soon.

[jimjam-slam]

Thanks @cscheid! If it's an ID'd bug with an incoming fix, I might sit tight, but if it does turn out that it's not feasible to fix soon we can always implement your workaround 😊 Thanks for taking a look (and for the git bisect tip—I'll be sure to have a good look at it soon)!

[cscheid]

More information (unfortunately, now I think that there's no simple way for us to fix it in quarto). Here's the source of the bug. It is ultimately related to how Pandoc 3 changes the way it parses code blocks. It turns out that Pandoc 3 no longer supports

```{foo}
code

as a syntax for code blocks. This used to work in Pandoc 2, and you'd get a `CodeBlock` node with class `{foo}` (_with_ the brackets). We depended on this, but it turns out the author of Pandoc 2 considered this behavior a bug, and in Pandoc 3, that's not how this works anymore; what you get is a `Para` node with a `Code` node inside, with `{foo}` in the content. This is, I believe, what you're seeing.

The way to fix this bug for you is to identify that `Para [Code ...]` structure in the Pandoc AST and replace it with the `CodeBlock` structure. That means my Lua suggestion was right; but you'll need to do a little bit of the work to find a `Para` with a `Code` that has source starting with `{ojs}` inside it, and then replacing that whole thing with a `CodeBlock` that has a class `{ojs}`, and a `text` field that is the rest of the code in the `Code` block. Hopefully that makes sense?

[kdheepak]

For what it is worth, I also noticed that when you have a new line in the code block pandoc 3 does something weird too.

With this in a test.md file:

```{ojs}
require("d3")

Here's what I get for the JSON:

$ quarto pandoc test.md --to json {"pandoc-api-version":[1,23],"meta":{},"blocks":[{"t":"Para","c":[{"t":"Code","c":[["",[],[]],"{ojs} require(\"d3\")"]}]}]}

But with this:

require("d3")

I get this instead:

$ quarto pandoc test.md --to json {"pandoc-api-version":[1,23],"meta":{},"blocks":[{"t":"Para","c":[{"t":"Str","c":"{ojs}"}]},{"t":"Para","c":[{"t":"Str","c":"require("},{"t":"Quoted","c":[{"t":"DoubleQuote"},[{"t":"Str","c":"d3"}]]},{"t":"Str","c":")"}]},{"t":"Para","c":[{"t":"Str","c":""}]}]}

Notice in the second version there's no `Code` at all.

[cscheid]

@kdheepak That's right, it's ultimately the same bug.

You could try to work around it in the same way that we do, by running a custom Reader.

$ quarto pandoc test.md --to json --from qmd-reader.lua

This should give you an AST that looks like the one from Pandoc 2. You might have to locate that file (or, if you're ok with it, you can just grab a local copy).

A final note of warning: this file has been the source of 2 or 3 recent quarto regressions (you can see the log). Charles and I spent an hour doing a careful joint review today, but I wouldn't be surprised if this file is still subtly incorrect.

[kdheepak]

Without changing the syntax that quarto supports (maybe a 2.0 discussion?), it seems like using the custom reader is the only solution here.

[jimjam-slam]

If we're using the usual Quarto render (just with pre-render scripts and a filter bolted on), then I would imagine that we're already using the custom reader that you've developed (at least if we're on latest pre-release)? Or do you only invoke that custom reader in certain circumstances?

[cscheid]

@jimjam-slam The problem is that you're calling pandoc.read yourself, here. That call to pandoc.read will need a custom reader.

[jimjam-slam]

Ahh, of course. I'll see what I can do there! Thanks again!

[jimjam-slam]

Sopandoc.read() takes a format argument, but it looks like it accepts a string from a table of built-in formats. it doesn't looks like you can just give it the path to a lua file, like the qmd-reader.lua that Quarto ships with. I wonder if there's another way to pass a custom format in here? The docs seem to suggest it takes a string from a table of built-ins.

[jimjam-slam]

EDIT: copying qmd-reader.lua into the extension directory and running require "_extensions.sverto.qmd-reader" (extension namespacing aside) gets us that file's Reader function in scope, but unfortunately it doesn't look like the format argument takes a function 🤔

[cscheid]

qmd-reader.lua is a parameter, like pandoc -f qmd-reader.lua, for what's worth (in case you're calling pandoc directly)

[jimjam-slam]

Thanks very much for helping me through this, @cscheid!

I've had a crack at integrating the custom reader code on #14, and it looks like it's working so far.

I've largely ignored the parts of qmd-reader.lua's Reader function (commented out in my commit) that check and pass user-defined extensions and options in to pandoc.read, so it's possible this solution isn't general enough yet. But it is getting us back to a working state, so I think we're mostly there!

@kdheepak, does this solution work for your purposes so far?

[cscheid]

I've had a crack at integrating the custom reader code on #14, and it looks like it's working so far.

Great!!

I've largely ignored the parts of qmd-reader.lua's Reader function (commented out in my commit) that check and pass user-defined extensions and options in to pandoc.read, so it's possible this solution isn't general enough yet.

Maybe that's not necessary for your usecase, but just to explain: you'll be missing two classes of things:

1. Explicit extensions written like

   from: markdown+emoji

2. Extensions that we turn on by default on all documents. These might mean that some features break, and it's worth making sure you don't need them. We do turn on a number of extensions by default.

[jimjam-slam]

Fix is now merged and released with 1.3! 🥳

[kdheepak]

Hi @jimjam-slam, thanks for taking a stab at this.

It still doesn't appear to be working for me though. I have a very simple example that I'm testing:

[jimjam-slam]

Hey @kdheepak, thanks for taking a look! Are you an me to point me to this sm example so that I can reproduce? It could be that this is a separate issue and I need to work on the file path identification.

Also keep in mind that if you're running Sverto on an existing project, like your blog, you need to add the magic block to all your posts and pages, not just the ones actually including Svelte components!

[kdheepak]

Oh, thanks for the update. I pushed the code that I was testing in this commit:

https://github.com/kdheepak/kdheepak.github.io/commit/71b6d2b80db26f46488a5bcaab367ad7b6d512a9

I might skip on using this project for my blog because I already have a number of posts that I don't want to mess with. I'll maybe try it out on a new project :)

[jimjam-slam]

Sorry for the pain there! I'd love to get to the point where we don't need to have the magic block at all, but it's currently a necessity with the way Quarto is designed. If you're not importing any components then Sverto should leave the other posts alone, but I understand if you're not keen to touch them!