5.2.0 update and --shell-escape

[Diapente]

I write gabc snippets in vim for creating plainsong psalms. Previously I was able to run :! lualatex xxx.tex to create a PDF of my snippets. The recent update causes this to fail unless I run: :! lualatex --shell-escape xxx.tex Other than downgrading to the previous release is there a way to fix this behaviour?

[rpspringuel]

Yes and no.

LuaTeX (and all other TeX engines) restrict the possible command line tools which can be called from within a document as a security measure. If you're using a TeX Live based distribution, then the version of gregorio which comes bundled with that years distribution is on the list, but upgrades are not (upgraded executables have a different filename in order to allow multiple versions to live side-by-side you your computer). If you wish to upgrade Gregorio outside of the TeX Live update system then you have to do one of the following:

1. Use --shell-escape to allow that compilation to call any shell command. Obviously this makes the compilation less secure, but that drop in security applies only to that document compilation, allowing for case-by-case differentiation in compilation behavior. It is our default recommendation because it is the simplest for most people to implement. Since the is the option you don't like, then perhaps you're willing to pursue one of the more involved options below.
2. Make the --shell-escape mode the default by setting shell_escape=t in texmf.cnf. This will make all compilations less secure, but means that you don't have to specify the --shell-escape flag on any given run.
3. Add the new executable to the shell_escape_commands list in texmf.cnf. New versions of the executable install with a name which includes the version number by default. I.e. the executable for the 5.2.0 executable is gregorio-5_2_0. This is probably the most secure option, in that it simply lets the new executable, and only the new executable.
4. Create a symlink on your system which points gregorio to gregorio-5_2_0 and which has higher priority in PATH than the location of texbin (where the executables from TeX Live live). This is has a similar level of security to option 3, assuming that the symlink can't be hijacked.
5. Abandon the autocompilation feature and manually compile your gabc files yourself from the command line. This doesn't work for \gabcsnippet (only \gregorioscore) and is damned inconvenient generally, but it is possible (indeed, it's how things used to work all the time before we implemented the autocompilation feature).

[Diapente]

Ok - trying to wrap my head around this change. I'm on arch and I installed gregorio from the AUR - so my install of gregorio is outside of the TEX live distribution. Due to this I now add the version number to texmf.cnf when I upgrade (option 3) to get similar behaviour to before the update?

[rpspringuel]

I think so, but that's a question best asked of @avbop or @eschwab, both of whom are far more familiar with that platform.

[eschwab]

Option 3 is the safest. Locate your texmf.cnf with kpsewhich texmf.cnf (probably at /usr/local/texlive/2018/texmf.cnf) and add:

shell_escape_commands = gregorio-5_2_0

For each gregorio update you will need to change the 5_2_0 to correspond to the new version number. This will allow you to continue using:

$ lualatex xxx.tex

IF you run into problems with bibtex or others that suddenly stop working, then copy the value of shell_escape_commands from /usr/local/texlive/2018/texmf-dist/web2c/texmf.cnf into the /usr/local/texlive/2018/texmf.cnf and append gregorio-5_2_0 to the list i.e.:

shell_escape_commands = \
bibtex,bibtex8,\
extractbb,\
kpsewhich,\
makeindex,\
repstopdf,\
texosquery-jre8,\
gregorio-5_2_0,\

[avbop]

I did some more exploration, and here's what I think is going on.

with gregorio 5.1.* and no --shell-escape:

1. gregoriotex looks for gregorio-5_1_*, but is blocked from finding it since it's not on the permitted executables list.
2. So, it falls back to looking for gregorio. Since this is on the list, it finds it. Since the version number happens to match right now (TexLive 2018 has 5.1.1), this works.

with gregorio 5.2.0 and no --shell-escape:

1. gregoriotex looks for gregorio-5_2_0, but is blocked from finding it since it's not on the permitted executables list.
2. So, it falls back to looking for gregorio. This works, but it's version 5.1.1, which is incompatible with 5.2.0, so it stops and outputs the error message about --shell-escape.

This is all unrelated to running gregorio without --shell-escape, though; for that, see the above. I had to create a new texmf.cnf, containing shell_escape_commands = gregorio-5_2_0, at /usr/share/texmf-local/web2c/texmf.cnf and run mktexlsr to get this method to work. (At least for me, Arch doesn't package TeXLive under the year, but directly at /usr/share/texmf-dist.) You can run kpsewhich -var-value=shell_escape_commands to see whether it has picked up on your changes.

[eschwab]

Interesting about needing to run mktexlsr, I did not need to do that. Simply saving my "local" texmf.cnf was sufficient. The paths I provided are based on what I have, but alas I am not using the texlive-core et.al. packages from arch, since I roll my own install using the install-tl script from tug.org.

[rpspringuel]

I did some more exploration, and here's what I think is going on. [snip...]

That is the logic which is built-in to gregoriotex.lua. As I mentioned before, it's designed for instances when multiple versions of gregorio are installed so that the program can tell the difference and use the one that will work with the TeX files being used.

My only concern was making sure that the suggested modifications to texmf.cnf would work on arch. I use Ubuntu for my Linux VMs and, like @eschwab, install TeX Live rather than use the package repositories (which tend to be out of date). As a result, I wasn't sure if arch's package management altered the way that TeX finds and uses texmf.cnf. I'm glad to hear there isn't any real problem there. Knowing that, perhaps I'll work up an addition for the website that explains how to configure the security restrictions without disabling them entirely.

[Diapente]

Ok, great - thanks for all your help, and for this software!

[rpspringuel]

Closing as this has now been more thoroughly documented and the one identified bug fixed.