tikzexternalize and graphicscache

Painkilla commented 2 years ago

I use tikzexternalize to reduce buildtime for TikZ, now i want to use graphicscache to reduce file size and build time of pictures as well. I dont´t know how this may interfere with #12 , so i started a different issue. It may be merged if necessary. My execution command: latexmk -f -pdf -synctex=1 -interaction=nonstopmode -output-directory='./conf/tmp/' '$./conf/main/main.tex'

From my /conf/main/main.tex i use this configuration to achieve that tikz uses a directory within my output-directory.

\usetikzlibrary{external}
\tikzsetexternalprefix{tikz/}
\tikzexternalize[shell escape=-shell-escape]

\makeatletter
    \tikzset{%
        external/system call={%
            pdflatex %
                \tikzexternalcheckshellescape %
                -halt-on-error %
                -interaction=batchmode %
                -output-directory="conf/tmp" %
                -jobname "\image" %
                "\string\def\string\tikzexternalrealjob{\tikzexternal@realjob}\string\input{conf/main/\tikzexternal@realjob}"%
        },
        /pgf/images/include external/.code={%
            \includegraphics{conf/tmp/#1}%
        },%
    }
\makeatother
\tikzset{external/up to date check=simple}

My latexmkrc is custom to be able to achieve this and located in the top directory

$latex = 'latex -shell-escape %O %S';
$pdflatex = 'pdflatex -shell-escape %O %S';
$pdftex = 'pdftex -shell-escape %O %S';
$makeindex = "makeindex -s ./conf/main/common/index_style.ist %S";

the last lines from the log /conf/tmp/tikz/main-figure0.log say

Package pdftexcmds Info: LuaTeX not detected.
Package pdftexcmds Info: \pdf@primitive is available.
Package pdftexcmds Info: \pdf@ifprimitive is available.
Package pdftexcmds Info: \pdfdraftmode found.
) (/usr/share/texmf/tex/generic/oberdiek/catchfile.sty
Package: catchfile 2016/05/16 v1.7 Catch the contents of a file (HO)
)
runsystem(uname -s > "tikz/main-figure1.w18")...executed.

! Package catchfile Error: File `tikz/main-figure1.w18' not found.

See the catchfile package documentation for explanation.
Type  H <return>  for immediate help.
 ...                                              

l.93     \CatchFileDef\@tempa{\ip@file}{}

Here is how much of TeX's memory you used:
 3610 strings out of 492995
 54121 string characters out of 6141053
 127254 words of memory out of 5000000
 7084 multiletter control sequences out of 15000+600000
 4702 words of font info for 16 fonts, out of 8000000 for 9000
 1141 hyphenation exceptions out of 8191
 59i,0n,51p,434b,38s stack positions out of 5000i,500n,10000p,200000b,80000s

!  ==> Fatal error occurred, no output PDF file produced!

The figures are not build correctly.

Whereas here is the output from the Build without graphicscache say:

LaTeX Warning: There were undefined references.

Package biblatex Warning: Please (re)run Biber on the file:
(biblatex)                tikz/main-figure0
(biblatex)                and rerun LaTeX afterwards.

Package logreq Info: Writing requests to 'tikz/main-figure0.run.xml'.
\openout1 = `tikz/main-figure0.run.xml'.

 ) 
Here is how much of TeX's memory you used:
 75420 strings out of 492995
 1750355 string characters out of 6141053
 2492709 words of memory out of 5000000
 77539 multiletter control sequences out of 15000+600000
 566782 words of font info for 120 fonts, out of 8000000 for 9000
 1145 hyphenation exceptions out of 8191
 82i,19n,123p,10454b,1983s stack positions out of 5000i,500n,10000p,200000b,80000s
{/usr/share/texmf/fonts/enc/dvips/cm-super/cm-super-t1.enc}</usr/share/texmf/
fonts/type1/public/cm-super/sfbx1200.pfb></usr/share/texmf/fonts/type1/public/c
m-super/sfrm1200.pfb>
Output written on conf/tmp/tikz/main-figure0.pdf (1 page, 20290 bytes).
PDF statistics:
 131 PDF objects out of 1000 (max. 8388607)
 23 compressed objects within 1 object stream
 0 named destinations out of 1000 (max. 500000)
 109 words of extra memory for PDF output out of 10000 (max. 10000000)

The figures are build correct again.

xqms commented 2 years ago

Hey, this seems like a complex setup. Can you please provide a full example that demonstrates the problem?

Painkilla commented 2 years ago

execute make cld-exm-thr-hyb-cmp-one-max-emce within the ./templates directory from this repo https://gitlab.com/omn-dev-team/conf/-/tree/painkilla/templates

the graphicscache package needs to be activated at /conf/main/common/package.tex first (Uncomment right at the top) The graphicscache.sty is at /conf/main/graphicscache.sty with my first attempts to hard code the directory and to remove the dependency from tikz by changing the renewcommand to something new. The current state is NOT working.

Painkilla commented 2 years ago

It is maybe a slightly confusing setup to understand. This is only a "conf" repo used within many other repos to achieve homogeneous output files. The templates directory is only a "sneak peak" what is possible and certainly not a productive file. The confusing command from the make is explained (in german) in Build.md. It is used to build different versions of the same file according to the needs of the student. Some repos currently contain more than 200 questions and many pictures (and increasing) and the buildtime and filesize totally exploded.

xqms commented 2 years ago

Yeah, it would be helpful if you could reduce it down to a single .tex file, maybe including a single .tikz file to demonstrate the problem. Otherwise I have to drill down into your build process, which looks quite complicated ;)

Painkilla commented 2 years ago

okay, i try to make a mwe.

Painkilla commented 2 years ago

I created a mwe on the graphicscache branch on the conf repo: https://gitlab.com/omn-dev-team/conf/-/tree/graphicscache

Steps to reproduce:

Using a linux machine with symlinks
cd ./templates
make test
Build will fail

Steps to generate only with tikzexternalize

Go into /conf/main/test.tex
Comment line 2
cd ./templates
make test
Build will generate an output file in /tmp/test.pdf
tikz temporary files are stored in /conf/tmp/tikz

Steps to generate only with graphicscache

Go into /conf/main/test.tex
Comment line 3, Uncomment line 2
cd ./templates
make test
Build will generate an output file in /tmp/test.pdf
NO tikz temporary files are stored in /conf/tmp/tikz
graphicscache temporary files are stored in /graphicscache

Is it possible to add something like a compatible to tikz option where the includegraphics command is not overwritten, maybe rather a new command, maybe includecachedgraphics is defined. Looking into the logs it seams there is a problem with exactly this command. It is very easy to change in al repos the includegraphics command against the new includecachedgraphics command with search and replace.

As mentioned in the other thread a path=/conf/tmp Option to change the temporary path would be really helpful to keep the repo clean.

Painkilla commented 2 years ago

I am sorry, but unfortunately i could not reduce it any further since including files from different locations and using it in combination with make and latexmk is crucial for this repo.

xqms commented 2 years ago

I followed your steps to reproduce the problem and for me everything works. So probably it's an issue that is version-specific. What kind of Linux distribution are you using?

xqms commented 2 years ago

ah sorry, I was using the wrong branch. Now I could reproduce it.

xqms commented 2 years ago

Ok, I managed to track the problem down.

You set \tikzsetexternalprefix{tikz/}, which results in pdflatex being called for each figure XYZ with the -jobname tikz/XYZ option.
Since you also manually patched the tikzexternal call to have -output-directory="conf/tmp", it creates the output pdf, log, aux files for the figure in conf/tmp/tikz/.
I'm using the ifplatform package to decide whether we are on Windows or Linux. On Linux, it tries to execute uname -s > \jobname.w18, which expands to uname -s > tikz/XYZ.w18. Since shell commands are run in the current directory, not conf/tmp, this fails since there is no tikz directory to create the XYZ.w18 file in.

Possible solutions:

Create a tikz directory in the place where pdflatex is executed. That fixes the problem for me.
Remove the \tikzsetexternalprefix{tikz/}. Also fixes the problem for me.
Replace the \tikzsetexternalprefix{tikz/} with -output-directory="conf/tmp/tikz" in tikz.tex). I couldn't make that work right now for some reason.
Before \usepackage{graphicscache}, set \jobname to something without slashes. After the \usepackage, reset it to the initial value. Haven't tried this.

Anyway, this is a problem with the way you patched TikZ to output its files to another directory and the ifplatform package. I cannot think of a good way to deal with this inside graphicscache. If you have a suggestion, please comment. Otherwise I'm going to close this issue.

(Global suggestion: It might be better to think of a way to avoid the -output-directory option altogether. Maybe you can symlink your main tex file into the conf/tmp directory and run pdflatex there? With TEXINPUTS set to the main directory it could work...)

Painkilla commented 2 years ago

Hi, thank you for your help!

with your help i managed to get everything to work. graphicscache is now at /conf/tmp/graphicscache tikz is now at /conf/tmp/tikz

I used this super dirty symlink hack: i placed at /conf/tmp/tikz/conf/tmp/tikz a symlink to ./../../../tikz So in the first run the images are placed at /conf/tmp/tikz/conf/tmp/tikz which is due to the symlink equal to /conf/tmp/tikz. The \tikzsetexternalprefix{conf/tmp/tikz/} command then looks then only in the lower path and everything is fine.

This is not very clean but who will look into a tmp folder if everything is super clean? Whereas a tikz folder in the main repo makes me uncomfortable.

xqms / graphicscache

tikzexternalize and graphicscache #13