File Transclusion doesn't work when reading stdin

[willhains]

cat file1.mmd file2.mmd | multimarkdown

File transclusion doesn't work, even if Transclude Base is included at the top of file1.mmd with an absolute path.

[willhains]

I'm using the macOS build of 6.5.1 from here.

[willhains]

Is there a way to set the transclude base as a command-line argument? That would avoid the need to create a temporary file to work around this.

[fletcher]

Transclude base can be a relative path, which would still not work from stdin. (In fact, some of the most useful (and challenging!) ways of transcluding makes use of relative paths.) And the files that you pull in from the "virtual file" on stdin could, themselves, transclude other files which could use relative paths, even if you forced an absolute path the first time.

Transclusion without starting from a fixed location on disk opens up a lot of edge cases that I worry would not work properly.

I'll keep this open as a reminder to reconsider periodically, but for the moment I still think of transclusion as a feature that should require working from a fixed file on disk.

[willhains]

Thanks for the quick response, @fletcher! I understand the dilemma. Please feel free to put this on the someday/maybe list. I will use temporary files as a workaround for now.

Just a suggestion: could you mention this in the docs? I spent quite a while trying to figure out why it wasn't working. I'd hope other users don't run into the same problem.

[fletcher]

That's fair -- can add to documentation when I get a chance.

Thanks!

[siegel]

We just caught a customer case about this; BBEdit 13.1 and later allow specifying a custom Markdown processor for previewing and HTML exports. There are several built-in options that are explicitly supported, including Multimarkdown.

In BBEdit, preview generation works by supplying the document's text on stdin (for various reasons), so transclusion unexpectedly (to the user) doesn't occur.

I won't pretend to know how Multimarkdown works internally; but I think there's probably a viable solution that includes one or more of the following strategies (any or all of which might only apply when reading from stdin):

• use $PWD as the base location for relative references. (getcwd(3) will do this for you.) I think this might address most of your concerns about where to "anchor" transclusion for files provided on stdin. (Currently, BBEdit does not set $PWD to the document's directory for preview text generation via Unix process, but it will.)

• support a "--transclude-base" argument on the command line, which specifies a directory to use instead of $PWD.

• look for a "MULTIMARKDOWN_TRANSCLUDE_BASE" environment variable. (I mention this purely for completeness; if the other solutions are viable it may not be worth the effort.)

[fletcher]

I'll need to keep considering this. Something about the proposed solutions doesn't "feel right" -- I worry about there being further edge case bugs if done incorrectly.

Based on the way that MultiMarkdown handles absolute and relative path determination and prevents infinite loops, what might be workable would be a way to specify a "virtual" path to the file when it passed on stdin. It should be the path to a specific file (that doesn't necessarily have to exist), NOT a directory. In the case described in BBEdit, it should be the path to the user file, even if the file text is sent via stdin instead of being read directly from the file.

• Using PWD isn't right. The path used for the file and determination of relative references is not necessarily the same as the current working directory. Conflating the two is likely to lead to other issues down the line.

• Setting transclude-base via the command line doesn't really solve the complete problem. Again, it might work for simplistic use cases, but I suspect there will be other edge case issues.

• Same as setting transclude-base via command line

[alphatan]

As a reference, the combination of multimarkdown and markdown-mode.el could not handle the TRANSCLUSION section described in MultiMarkdown-6-Syntax-Guide.

Cause: markdown-mode.el sends info to multimarkdown by stdin , where is not graceful for relative path.

[daaaaaaaaaniel]

After I read the Transclusion Preprocessing section in the MultiMarkdown User Guide, I tried configuring multimarkdown as a preprocessor in Marked 2 (because Marked 2 uses its own special syntax for transclusion, and I would prefer to use the standard MMD transclusion syntax). Since preprocessors in Marked 2 must take input on STDIN, I wasn't able to make this work. Rather than replacing the {{included_file.md}} with the content of the file, the preprocessor just returned the same text as the original input. I tried setting an absolute path to the folder containing the files I want to include using the transclude base metadata, but even that wouldn't solve it.