Using ${USERDATA} or ${HOME} in Defaults Bibliography Metadata

[iandol]

Explain the problem:

Given this defaults file:

verbosity: INFO # verbose by default
citeproc: true
cite-method: citeproc
reference-links: true
metadata:
  csl: ${USERDATA}/csl/apa.csl
  bibliography: ${USERDATA}/Core.json # JSON faster than BIB, symlinked into Pandoc data dir
  citation-abbreviations: ${USERDATA}/cite-abbr.json # my journal abbreviations
  reference-section-title: Bibliography
  notes-after-punctuation: false
  link-citations: true
  link-bibliography: true

When I try to compile I get an error:

▶︎ pandoc -d refs -t latex -o tuftebook.tex ~/Code/dotpandoc/test-tufte.md
[INFO] Running filter citeproc
[INFO] Loaded apa-ian.csl from /Users/ian/.local/share/pandoc/csl/apa-ian.csl
File ${USERDATA}/cite-abbr.json not found in resource path

So ${USERDATA} is not expanding, however by moving those fields out of metadata:

verbosity: INFO # verbose by default
citeproc: true
cite-method: citeproc
reference-links: true
csl: ${USERDATA}/csl/apa.csl
bibliography: ${USERDATA}/Core.json #JSON faster than BIB, symlinked into Pandoc data dir
citation-abbreviations: ${USERDATA}/cite-abbr.json # my journal abbreviations
metadata:
  reference-section-title: Bibliography
  notes-after-punctuation: false
  link-citations: true
  link-bibliography: true

So bibliography and citation-abbreviations are top level, now Pandoc finds the files:

▷ pandoc -d refs -t latex -o tuftebook.tex ~/Code/dotpandoc/test-tufte.md
[INFO] Running filter citeproc
[INFO] Loaded apa-ian.csl from /Users/ian/.local/share/pandoc/csl/apa-ian.csl
[INFO] Loaded /Users/ian/.local/share/pandoc/cite-abbr.json from /Users/ian/.local/share/pandoc/cite-abbr.json
[INFO] Loaded /Users/ian/.local/share/pandoc/Core.json from /Users/ian/.local/share/pandoc/Core.json
[INFO] Completed filter citeproc in 245 ms

According to the manual, we should use metadata for csl & bibliography but when I do ${USERDATA} fails to expand. My workaround (don't put them in metadata) is fine for this use case although the behaviour is unexpected.

The manual does state:

This environment variable interpolation syntax only works in fields that expect file paths.

So this is probably only tested for at the top-level?

In fact the manual example fails if you use the proposed position. e.g.

In fields that expect a file path (or list of file paths), the following syntax may be used to interpolate environment variables:

csl: ${HOME}/mycsldir/special.csl https://pandoc.org/MANUAL.html#defaults-files

--csl ieee.csl metadata:    csl: ieee.csl https://pandoc.org/MANUAL.html#citation-rendering-1

csl should be in metadata, but the example shows csl used with ${HOME}

Expected Result:

I would ideally hope we could use ${USERDATA} within metadata bibliography fields too.

Hoped for extensions:

Apart from bibliography I would also wish we could use ${USERDATA} or ${HOME} for other custom metadata fields, for example I have a copyright-image field I'd like to link to my pandoc data directory so my defaults files do not contain absolute paths, but I cannot do this at present. Why is the use of ${USERDATA} or ${HOME} so restrictive?

Pandoc version?

▶︎ pandoc --version
pandoc 3.1.9
Features: +server +lua
Scripting engine: Lua 5.4
User data directory: /Users/ian/.local/share/pandoc
Copyright (C) 2006-2023 John MacFarlane. Web: https://pandoc.org
This is free software; see the source for copying conditions. There is no
warranty, not even for merchantability or fitness for a particular purpose.

[jgm]

As documented, we expand these for the defaults file fields that take file paths as arguments. metadata doesn't -- it returns a structured metadata object. As you note, components of metadata might be file paths, but it's hard to know which of them are intended that way.

I think the example from the manual https://pandoc.org/MANUAL.html#citation-rendering-1 should be changed to use top-level bibliography etc.

[iandol]

... but it's hard to know which of them are intended that way.

Why can't ${...} interpolate for other items? I assume Pandoc has a good reason to add in this limitation; are you worried about collision with possible content, i.e. maybe a user uses ${.} to mean something in metadata; this does seem highly unlikely... How about "quoting" to handle this? I would really like to have relative paths expanded for resources that get passed into pandoc templates using metadata, and this restriction blocks this, requiring absolute paths and/or some script to rewrite yaml file paths across different machines etc. I'd assume there are more potential users who benefit from path interpolation than would be affected by any syntax collision?

I think the example from the manual should be changed to use top-level bibliography etc.

Yes, if you do not relax the interpolation rules than that makes sense.

[jgm]

I've updated the manual. Making this work in metadata would require significant changes, and we'd have to drop the restriction to fields that are meant to be file paths, because there'd be no way to know which these are. That would mean that you couldn't include a literal ${HOME} in metadata. For now I don't want to make that change, but we can leave the issue open.

[iandol]

OK, thank you for your consideration!