--data-dir is ignored in defaults files' search path

[tstenner]

With pandoc 2.13, the data dir is used to the search path for default files supplied with the --defaults parameter but not in other default files:

$ mkdir -p test/defaults
$ printf 'to: markdown' > test/defaults/a.yaml
$ printf 'defaults: a' > test/defaults/b.yaml
$ echo 'Test' | pandoc --data-dir=test --defaults=a -  # loads test/defaults/a.yaml
Test
$ echo 'Test' | pandoc --data-dir=test --defaults=b -  # loads test/defaults/b.yaml, but doesn't find a.yaml in the same directory
pandoc: a.yaml: openBinaryFile: does not exist (No such file or directory)
$ cd test/defaults
$ echo 'Test' | pandoc --defaults=b -   # a.yaml in the working directory is found
Test

[jgm]

See https://github.com/jgm/pandoc/pull/6924 which explains the motivation for the way it is now (under "user data directories"). I'm not sure about this.

[tstenner]

defsA not inheriting the data dir from defsB is a bit surprising, but I can see that there are valid reasons for it.

Both files are in the same directory, so I'd expect that if b.yaml is found, so should a.yaml. Also, when I put both files in the default --data-dir path (~/.local/share/pandoc/defaults), it works just fine.

The docs suggest that --data-dir will change the user data directory and a defaults file won't override it, but just use it as search path for templates etc.

--data-dir=DIRECTORY

Specify the user data directory to search for pandoc data files.

--defaults The file will be searched for first in the working directory, and then in the defaults subdirectory of the user data directory (see --data-dir).

# Include options from the specified defaults files.
# The files will be searched for first in the working directory
# and then in the defaults subdirectory of the *user data directory*.
# The files are included in the same order in which they appear in
# the list. Options specified in this defaults file always have
# priority over the included ones.
defaults:
- defsA
- defsB

[tstenner]

Some more rough edges: both data-dir: and defaults: don't interpolate the environment variables ${.} etc., so neither

data-dir: ${.}
defaults: a

nor

defaults: ${.}/a.yaml

includes a.yaml.

[jgm]

I wonder if @davidmrt98 (who implemented the code allowing default files to be specified within default files) has a comment on this issue?

[jgm]

On the second "rough edge," judging from the code, data-dir should accept the interpolation syntax. I just tried it and confirmed that it works.

defaults does not currently accept the interpolation syntax. It should, probably.

[tstenner]

On the second "rough edge," judging from the code, data-dir should accept the interpolation syntax. I just tried it and confirmed that it works.

I just checked and it doesn't:

$ pandoc --version
pandoc 2.13 [..]
$ mkdir -p testdir/defaults
$ printf 'to: markdown' > testdir/defaults/a.yaml
$ printf 'data-dir: ${USERDATA}\ndefaults: a' > testdir/defaults/b.yaml
$ echo '# test' | pandoc --data-dir=testdir -d b -
pandoc: a.yaml: openBinaryFile: does not exist (No such file or directory)
$ printf 'data-dir: ${.}\ndefaults: a' > testdir/defaults/b.yaml
$ echo '# test' | pandoc --data-dir=testdir -d b -
pandoc: a.yaml: openBinaryFile: does not exist (No such file or directory)
$ printf 'data-dir: .\ndefaults: a' > testdir/defaults/dot.yaml
echo '# test' | pandoc --data-dir=testdir -d dot -

But for some reason, this does:

$ cd testdir/
$ echo '# test' | pandoc --data-dir=. -d dot -
# test

[jgm]

I'm not happy with the complexity of the code that handles this; I may try to rewrite it in a simpler way.

[davidmrt98]

I agree that the behaviour in the first example is odd. I already have a guess as to what the cause could be, but I will take a closer look.

On the interpolation issue I can not comment, as I was not involved in the implementation of this feature.

[7596ff]

I'm currently building something that is affected by this issue, and I wanted to see if I had the current status of it correct.

There are in fact two bugs discussed in this issue:

• Per the docs, a defaults file specified with --defaults is searched for in the current working directory and the defaults subdirectory of the data directory. If that defaults file includes other defaults files with defaults: (per #6924), a search for them will not inherit the data-dir value passed from either the command line or the first defaults file itself.
• In defaults files, ${.} is not interpreted correctly.

Similar problems have been brought up in the following issues: #8024 #8780 #8826 #8841

As for the first problem, in my project, I can work around it by programmatically inserting the correct data-dir in each defaults file, but this is not an ideal solution.

I don't want to ask for any guarantees or ETAs, but is this an accurate summation of the status of this problem?

[mboyea]

defaults does not currently accept the interpolation syntax. It should, probably.

This would be an extremely useful feature for me. Judging by the half-dozen issues about the feature, I think it would be useful for others too. When I find time, I may contribute towards implementing this feature.

The use case of having a dependency tree in the data/defaults directory is pretty huge! The ${.} syntax feels like a must-have.

EDIT: The ${USERDATA} syntax is not supported either, so ${USERDATA}/defaults/adefault.yaml also doesn't work in the defaults: list.