Shell escape for multiple file creation?

[communistkiro]

My tex package https://github.com/communistkiro/dmlb has an option to enable graphing of the data. When I first started using tectonic, shell-escape wasn't a thing, but now it is, and get the following error:

! Package tikz Error: Sorry, the system call 'xelatex -shell-escape -halt-on-er
ror -interaction=batchmode -jobname "./graphs/88" "\def\tikzexternalrealjob{gra
phmedaddy}\input{graphmedaddy}"' did NOT result in a usable output file './grap
hs/88' (expected one of .pdf:.jpg:.jpeg:.png:.bmp:). Please verify that you hav
e enabled system calls. For pdflatex, this is 'pdflatex -shell-escape'. Sometim
es it is also named 'write 18' or something like that. Or maybe the command sim
ply failed? Error messages can be found in './graphs/88.log'. If you continue n
ow, I'll try to typeset the picture.

1--87 have been already created, hence the skip. I do not know if it is possible to run tectonic seperately for every file in specific way so as to get the .pdf, or if I'm missing something. Little-to-nothing of note has changed within the .tex and .sty files, I house locally; the command I'm running is tectonic --keep-intermediates --keep-logs --print --reruns 0 -Z shell-escape graphmedaddy.tex && tectonic --keep-intermediates --keep-logs --print --reruns 0 -Z shell-escape fmsuf.tex; 88.dat I've verified is proper and valid, as much as the previous 87 are.

I'll provide additional information if required to aid diagnosis.

[pkgw]

Let me see if I'm understanding your setup correctly. Is it correct that you're basically running TeX 88 times to create 88 graphs (with 87 results saved from previous runs), and then you run it one more time to assemble them all into one big output document?

Based on the output you pasted, it seems that tikz is running xelatex to generate the 88th graph. And if I'm understanding you correctly, that's coming from a Tectonic run? Since Tectonic is derived from, and basically masquerades as, XeLaTeX, that's understandable, but not what we would want — Tectonic should be able to keep things self-contained as much as it can.

If my understanding of the overall process is correct, I think it would be better to separate the generation of the 88th graph and the compilation of the overall document. (Traditional TeX techniques kind of encourage people to combine all of these steps in one big document build, but from an engineering standpoint I think it's really not desirable.) This may be the intention of tectonic ... && tectonic ... commands you pasted. If you generate that 88th graph using xelatex, does the overall document build work correctly in Tectonic?

If that first tectonic command of yours is intended to emulate tikz's invocation of xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "./graphs/88" "\def\tikzexternalrealjob{graphmedaddy}\input{graphmedaddy}", it won't quite work due to the "jobname" specification: at a minimum, the output file(s) won't get the right name. That could be patched up pretty easily with a rename command, though.

Sorry this is a bit meandering — it's a bit hard to know what to focus on without more information about your specific document. Are you able to reduce your workflow into a minimal example that seems to reproduce the same problem?

[communistkiro]

Not quite, TeX is run n times, but I run a single command once to create the graphs. Starting without any other files than the two .tex and the .sty files, one would have to:

• run xelatex once without my [graph] option to create all the .dat files and the basic document;
• run xelatex once with shell-escape option on the graphing .tex, this created a .pdf from every .dat without one;
• run xelatex once with [graph] option, wherein .pdf's are merely included, not not created (I cannot remember why I'd split this into two files, but I wrote it awhiles back).

If you generate that 88th graph using xelatex, does the overall document build work correctly in Tectonic? Deleting all .dat files after 88, the graphme.pdf including all separate graphs is generated, however, since .dat files are regenerated on compilation, building the main .tex with the [graph] option results in everything up to the 88 graph being rendered correctly based on the output of tectonic: ...yaddayadda.. Output written on fmsuf.xdv (229 pages, 24122436 bytes)., halting on the exact same tikz error as in the OP, with fmsuf being main document name.

Using shell shenanigans, probably everything could be automated, but the idea behind the package was to be standalone and without fuss: enter data, run xelatex in the above order with the relevant options, gets graphs from the data. However, I don't know how to map the xelatex -shell-escape -halt-on-error -interaction=batchmode -jobname "./graphs/88" "\def\tikzexternalrealjob{graphmedaddy}\input{graphmedaddy}" command to tectonic, like mentioned the jobname spec is one of the things.

Regarding a MWE:

• clone my repo
• edit template.tex to delete the % CALENDAR section right after \begin{document}; remove the [graph] option;
• run tectonic --keep-intermediates --keep-logs --print --reruns 1 template.tex; with the 2 files of 20-something days plus as many in the many .tex, you should get 3 .dat files in ./graphs/: 0, 1, and 81 (each dat files spans 28 days as per the option in template.tex, so a date skip to 2006-02-01 is ~ 88*28 days);
• run tectonic --keep-intermediates --keep-logs --print --reruns 0 -Z shell-escape graphmedaddy.tex && tectonic --keep-intermediates --keep-logs --print --reruns 0 -Z shell-escape template.tex to get the OP's error but for 1 instead of 88.

[kpym]

This is related to #852