Gnuplot native support by default

[dflvunoooooo]

The environment gnuplot is not working. Is it possible to cache gnuplot code? The package is called gnuplottex and to convert gnuplot plots to tikz and include it inside Latex the package gnuplot-lua-tikz is needed, even if one is not using lua. For this package the --shell-escape option has to be set as well. Ther error is Package pgfkeys Error: I do not know the key '/robExt/gnuplot' and I am going to ignore it. Perhaps you misspelled it. ^^I\end.

For this a caching would be especially great, since the calculation of the plot can take some time and it is calculated every time.

Here is a minimalistic example which runs without CacheMe:

%%%%%%%%%%%%%%%%%%%%%%%                 Klassen-Optionen
\documentclass{scrreport}

\usepackage{amsmath}                                % Unverzichtbare Mathe-Befehle.
\usepackage{tikz}                                   % Für Zeichnungen direkt in LaTeX

\usepackage{gnuplottex}                             % Um Gnuplot direkt in Latex erzeugen zu lassen
\usepackage{gnuplot-lua-tikz}                       % Für die tikz-Umgebung in gnplottex. Ermöglicht Export von Variablen

\usepackage{robust-externalize}
\robExtConfigure{
    enable fallback to manual mode,
}

%%%%%%%%%%%%%%%%%%%%%%%                 Beginn des eigentlichen Dokuments
\begin{document}

\begin{figure}
    \centering 
    \begin{gnuplot}[terminal=tikz]
        plot sin(x)
    \end{gnuplot}
    \caption{Test}
\label{gnu:CuRT}
\end{figure}

\end{document}

[tobiasBora]

So I think your problem is like in here where you need to load gnuplottex in the cached file, like:

\robExtConfigure{
  add to preset={latex}{
    add to preamble={\usepackage{gnuplottex}},
  },
}

But I can't test because I first need to make sure gnuplot-lua-tikz is accessible on my system, while it is not for now.

Another option is to directly do everything with this library without using gnuplottex at all. Here is an example with the pdf backend (again, can't test with the tikz one):

%%%%%%%%%%%%%%%%%%%%%%%                 Klassen-Optionen
\documentclass{scrreport}

\usepackage{amsmath}                                % Unverzichtbare Mathe-Befehle.
\usepackage{tikz}                                   % Für Zeichnungen direkt in LaTeX
\usepackage{robust-externalize}

\begin{PlaceholderFromCode}{__GNUPLOT_TEMPLATE__}
__GNUPLOT_TERMINAL__
__ROBEXT_MAIN_CONTENT__  
\end{PlaceholderFromCode}

\begin{PlaceholderFromCode}{__GNUPLOT_TERMINAL__}
set terminal pdf
set output "__ROBEXT_OUTPUT_PDF__"
\end{PlaceholderFromCode}

\robExtConfigure{
  enable fallback to manual mode,
  new preset={gnuplot}{
    set compilation command={gnuplot -c "__ROBEXT_SOURCE_FILE__"},
    set template={__GNUPLOT_TEMPLATE__},
    % For the tikz background, you might need also a custom include file, see `include graphics`
    print verbatim if no externalization,
  },
}

%%%%%%%%%%%%%%%%%%%%%%%                 Beginn des eigentlichen Dokuments
\begin{document}

\begin{figure}
    \centering 
\begin{CacheMeCode}{gnuplot,set includegraphics options={width=.5\linewidth}}
plot sin(x)
\end{CacheMeCode}
    \caption{Test}
\label{gnu:CuRT}
\end{figure}
\end{document}

It should not be too hard to adapt this to use the tikz background, just you will need a different include command, cf the doc. I will try to take some time to create a module just for gnuplot as it seems to be a quite common use case.

[dflvunoooooo]

Great, thank you. Will test this in a minute.

[dflvunoooooo]

The first proposal is not working or I am doing something wrong. With the setting

\robExtConfigure{
    add to preset={tikz}{
        add to preamble={\usepackage{gnuplottex} \usepackage{gnuplot-lua-tikz}},
    },
}

I still get the error Package pgfkeys Error: I do not know the key '/robExt/gnuplot' and I am going to ignore it. Perhaps you misspelled it. ^^I\end. Regardless if used with or without gnuplot-lua-tikz.

The second proposal is sadly not working either. Run in command line it says:

[robExt]We will start the compilationusing: cd robustExternalize/ && gnuplot -c
 "robExt-48723AF68C8154175B9C365809604861.tex".
^^I^^Iset terminal pdf
^
"robExt-48723AF68C8154175B9C365809604861.tex" line 1: invalid command

system returned with code 256

! Package robExt Error: The pdf file
(robExt)               
robustExternalize/robExt-48723AF68C8154175B9C365809604861.pdf
(robExt)                is not present. The compilation command "cd
(robExt)                robustExternalize/ && gnuplot -c
(robExt)                "robExt-48723AF68C8154175B9C365809604861.tex"" used
(robExt)                to compile the environment on line 39 certainly
(robExt)                failed, see logs above or in
(robExt)               
robustExternalize/robExt-48723AF68C8154175B9C365809604861.log.

[tobiasBora]

Sorry, for the first one I meant

\robExtConfigure{
    add to preset={latex}{ %% Use the latex preset
        add to preamble={\usepackage{gnuplottex} \usepackage{gnuplot-lua-tikz}},
    },
}

and you should use it like:

\begin{CacheMe}{latex}
\begin{gnuplot}[terminal=tikz]
plot sin(x)
\end{gnuplot}
\end{CacheMe}

(you can also create a new preset that wraps it automatically, but let's start simple)

For the second example, it is weird, seems like it added a TAB-like characters in front of the set terminal pdf line. Can you make sure that you have removed all symbols/space in front of set terminal pdf and all other lines? Ideally, copy/paste the above file as it, and make sure your editor does not re-indent it (I removed any space for safety). If it is not working, can you print me the exact content of the robExt-48723AF68C8154175B9C365809604861.tex file, possibly like hexdump -C robExt-48723AF68C8154175B9C365809604861.tex?

[tobiasBora]

Ok I managed to find how to get the .sty file (cf comments), so here is the version for the tikz version:

\documentclass{scrreport}

\usepackage{amsmath}
\usepackage{tikz}

%% Generate the .sty file with:
% ## First, gnuplot does not come with lua/tikz by default, you can get it with:
% $ nix-shell -p '(gnuplot.override { withLua = true; })'
% ## Then, you need to generate the .sty file with:
% $ gnuplot -e "set terminal tikz createstyle"
\usepackage{gnuplot-lua-tikz}

\usepackage{robust-externalize}

\begin{PlaceholderFromCode}{__GNUPLOT_TEMPLATE__}
__GNUPLOT_TERMINAL__
__ROBEXT_MAIN_CONTENT__  
\end{PlaceholderFromCode}

\begin{PlaceholderFromCode}{__GNUPLOT_TERMINAL__}
set terminal tikz  
set output "__ROBEXT_OUTPUT_PDF__"
\end{PlaceholderFromCode}

\robExtConfigure{
  enable fallback to manual mode,
  new preset={gnuplot}{
    set compilation command={gnuplot -c "__ROBEXT_SOURCE_FILE__"},
    set template={__GNUPLOT_TEMPLATE__},
    custom include command={\input{\robExtAddCachePathAndName{\robExtFinalHash.pdf}}},
    print verbatim if no externalization,
  },
}

\begin{document}

\begin{figure}
    \centering 
\begin{CacheMeCode}{gnuplot,set includegraphics options={width=.5\linewidth}}
plot sin(x)
\end{CacheMeCode}
    \caption{Test}
\label{gnu:CuRT}
\end{figure}

\end{document}

[dflvunoooooo]

Thank you for your work!

Ah, I was fiddeling with the latex part myself, after reading your documentation again, but didn't manage it. Your code does compile without errors, but it does not generate the plot. There is no fig-1 file, the latex log doesn't complain, but the robust-externalize log says:

Opening gnuplot stream ./robExt-506330CDE3109BC2E222DC5B95DE3CBF-gnuplottex-fig
1.gnuplot

Package gnuplottex Warning: Please convert ./robExt-506330CDE3109BC2E222DC5B95D
E3CBF-gnuplottex-fig1.gnuplot manually.

LaTeX Warning: Characters dropped after `\end{gnuplot}' on input line 15.

I think in the latex environment is the -shell-escape missing. That is needed for gnuplottex. Is it possibile to add that?

The second is sadly not working either. Still the error Package robExt Error: The pdf file. The latex log also mentions TAB-like characters. And it is the same log when run in the command line:

\openout6 = `test_gnuplotrobust-externalize-robExt-tmp-file-you-can-remove.tmp'
.

runsystem(mkdir -p robustExternalize/)...executed.

\openout6 = `robustExternalize/robExt-1A013C164428F72D4F69B1AC60747555.deps'.

\openout6 = `robustExternalize/robExt-1A013C164428F72D4F69B1AC60747555.tex'.

Source saved in robustExternalize/robExt-1A013C164428F72D4F69B1AC60747555.tex.
[robExt]We will start the compilationusing: cd robustExternalize/ && gnuplot -c
 "robExt-1A013C164428F72D4F69B1AC60747555.tex".
runsystem(cd robustExternalize/ && gnuplot -c "robExt-1A013C164428F72D4F69B1AC6
0747555.tex")...executed.

! Package robExt Error: The pdf file
(robExt)               
robustExternalize/robExt-1A013C164428F72D4F69B1AC60747555.pdf
(robExt)                is not present. The compilation command "cd
(robExt)                robustExternalize/ && gnuplot -c
(robExt)                "robExt-1A013C164428F72D4F69B1AC60747555.tex"" used
(robExt)                to compile the environment on line 41 certainly
(robExt)                failed, see logs above or in
(robExt)               
robustExternalize/robExt-1A013C164428F72D4F69B1AC60747555.log.

Type <return> to continue.
 ...                                              

l.41 ^^I^^I\end{CacheMeCode}

LaTeX does not know anything more about this error, sorry.

[tobiasBora]

Ok so I added support for gnuplot in Master (next v2.3). I was able to succesfully compile this file, at least on linux:

%%%%%%%%%%%%%%%%%%%%%%%                 Klassen-Optionen
\documentclass{scrreport}

\usepackage{amsmath}
\usepackage{tikz}

%% Generate the .sty file with:
% ## First, gnuplot does not come with lua/tikz by default, you can get it with:
% $ nix-shell -p '(gnuplot.override { withLua = true; })'
% ## Then, you need to generate the .sty file with:
% $ gnuplot -e "set terminal tikz createstyle"
\usepackage{gnuplot-lua-tikz}

\usepackage{robust-externalize}

%%%%%%%%%%%%%%%%%%%%%%%                 Beginn des eigentlichen Dokuments
\begin{document}

\begin{figure}
  \centering 
\begin{CacheMeCode}{gnuplot}
plot sin(x)
\end{CacheMeCode}
  \caption{Basic gnuplot call}
\end{figure}

\begin{figure}
  \centering 
\begin{CacheMeCode}{gnuplot,pdf terminal={size 5cm,4cm}}
plot sin(x)
\end{CacheMeCode}
  \caption{Gnuplot with different size}
\end{figure}

\begin{figure}
  \centering 
\begin{CacheMeCode}{gnuplot,set includegraphics options={width=5cm,height=4cm}}
plot sin(x)
\end{CacheMeCode}
\caption{Gnuplot with include graphics options}
\end{figure}

\begin{figure}
  \centering
  % Make sure to have:
  % \usepackage{gnuplot-lua-tikz} %% Generate with gnuplot -e "set terminal tikz createstyle"
\begin{CacheMeCode}{gnuplot, tikz terminal={size 8cm,5cm}}
plot cos(x)
\end{CacheMeCode}
\caption{Using tikz terminal + caching gnuplot but not tikz}
\end{figure}

\robExtConfigure{
  add to preset={tikz}{
    % Run gnuplot -e "set terminal tikz createstyle" in the robustExternalize folder first
    % to create this file:
    add to preamble={\usepackage{gnuplot-lua-tikz}},
  },
  tikz terminal/.append style={
    cache tikz,
  }
}
\begin{figure}
  \centering 
\begin{CacheMeCode}{gnuplot, tikz terminal={size 8cm,5cm}}
plot cos(x)
\end{CacheMeCode}
  \caption{Using tikz terminal + caching both gnuplot and tikz}
\end{figure}

\begin{figure}
  \centering 
\begin{CacheMeCode}{gnuplot, set terminal={png size 400,300}}
plot cos(x)
\end{CacheMeCode}
\caption{Using a custom png terminal and size}
\end{figure}
\end{document}

Hopefully, it will work for you as well? Let me know, I'm really curious to understand what is wrong.

For the approach based on gnuplot-lua-tikz, yes, indeed, you can add -shell-escape in the sub command using:

\robExtConfigure{
  add to preset={latex}{ %% Use the latex preset
    add to preamble={\usepackage{gnuplottex} \usepackage{gnuplot-lua-tikz}},
    % In v2.3, I add "add to compilation command options={-shell-escape}" for simplicity.
    add to placeholder={__ROBEXT_LATEX_COMPILATION_COMMAND_OPTIONS__}{-shell-escape},
  },
}

This gives me the (tested and working) code:

%%%%%%%%%%%%%%%%%%%%%%% Klassen-Optionen
\documentclass{scrreport}

\usepackage{amsmath}                                % Unverzichtbare Mathe-Befehle.
\usepackage{tikz}                                   % Für Zeichnungen direkt in LaTeX

\usepackage{gnuplottex}                             % Um Gnuplot direkt in Latex erzeugen zu lassen
%%% 
\usepackage{gnuplot-lua-tikz}                       % Für die tikz-Umgebung in gnplottex. Ermöglicht Export von Variablen

\usepackage{robust-externalize}
\robExtConfigure{
  add to preset={latex}{ %% Use the latex preset
    add to preamble={\usepackage{gnuplottex} \usepackage{gnuplot-lua-tikz}},
    % In v2.3, I add "add to compilation command options={-shell-escape}" for simplicity.
    add to placeholder={__ROBEXT_LATEX_COMPILATION_COMMAND_OPTIONS__}{-shell-escape},
  },
}

%%%%%%%%%%%%%%%%%%%%%%% Beginn des eigentlichen Dokuments
\begin{document}

\begin{figure}
  \centering 
  \begin{CacheMe}{latex}
    \begin{gnuplot}[terminal=tikz]
      plot sin(x)
    \end{gnuplot}
  \end{CacheMe}
  \caption{Test}
  \label{gnu:CuRT}
\end{figure}
\end{document}

Regarding your error with the ^^J I don't understand at all what is happening, as these tabs MUST come from somewhere… Are you 100% sure you have not inserted 2 TABs right before \end{CacheMeCode} for instance as the error suggests? I actually tried to add TABS there, and the error I get is basically identical to yours. Weirdly enough, the error you have mentions line 41 while the document only has 39 lines… so it seems like you are not using exactly the file I copied above. If you did, I would need:

• a fresh run from the command line (and not from tex studio) of pdflatex -shell-escape yourmainfile.tex
• the content of: hexdump -C yourmainfile.tex
• the content of: hexdump -C robustExternalize/robExt-1A013C164428F72D4F69B1AC60747555.pdf

If you don't know how to run these commands in a terminal, let me know what is your OS and I can tell you more. Meanwhile, attach these files to this conversation.

[dflvunoooooo]

This is awesome, thank you for all your work! The master does still say it is version 2.2, is this intended?

Your version brought things a big step further! :) But the indent problem is still there. At least for me. If I remove all indents and every leading spaces in every line and compile in command line with xelatex -synctex=1 -shell-escape -interaction=nonstopmode it works but for the last figure. In TexStudio it shows the same two errors. Without the last figure, everything works :) :) :) The error is:

! Unable to load picture or PDF file 'robustExternalize/robExt-3AF5356E64975153
7B1FDADAC49C40F0.pdf'.
<to be read again> 
                   }
l.76 \end{CacheMeCode}

! Unable to load picture or PDF file 'robustExternalize/robExt-3AF5356E64975153
7B1FDADAC49C40F0.pdf'.
<to be read again> 
                   \GPT@clipend 
l.76 \end{CacheMeCode}

I don't know why the indents are a problem. With a simple copy of your code I get two errors like this: Unable to load picture or PDF file 'robustExternalize/robExt-3C22317B38CE39D8456BD692113A6A05.pdf'. \end{CacheMeCode}, four like this Unable to load picture or PDF file 'robustExternalize/robExt-CE2984A86330560290F2B67532163B61.pdf'. ^^I\end{CacheMeCode}, and one like this Package graphics Error: Division by 0. ^^I\end{CacheMeCode}. So still some ident problem.

You second approach is working now without any problem :) Splendid! I will try and investigate the indent error in the original code.

Edit: I can no longer reproduce the former errors with version 2. Unfortunately I did not safe the old version. Do you want me to investigate further? Then I would need the old version.

[dflvunoooooo]

There seems to be problems with tabs in gnuplot. I can run

set key right bottom# Test comment
plot cos(x)

without problems. But with one ore more tabs bevore the gnuplot comment # Test comment it won't run

set key right bottom    # Test comment
plot cos(x)

Wit one ore more spaces on the other hand, it is working.

set key right bottom             # Test comment
plot cos(x)

[dflvunoooooo]

I just noticed that, if I want to plot data from a file, that file has to be in the robustExternalize folder, otherwise the plot fails, the latex compilaiton works and the figure stays empty. I don't know if this is intended, but it is a difference to gnuplottex, where the data has to be in the main folder.

Code example:

plot 'test.txt' using 1:2

test.txt

[tobiasBora]

For the folder, by default the file must be in robustExternalize (unless you change this folder to the same folder using no cache folder in the configuration) indeed just because gnuplot runs from this folder. But you should be able to add __ROBEXT_WAY_BACK__ that will be replaced with ../ (you can also directly add ../ but it would break if you change the folder of robustExternalize), like:

plot "__ROBEXT_WAY_BACK__yourfile.dat"`

I advise you not to directly put stuff in the cache folder as it might be removed if you clean it.

[tobiasBora]

I will try to automatically add set loadpath {"__ROBEXT_WAY_BACK__"} in the gnuplot template to fix this issue.

[tobiasBora]

And for your other issues, I’ll get back to you later today, I have some stuff to do before.

[dflvunoooooo]

For the folder, by default the file must be in robustExternalize (unless you change this folder to the same folder using no cache folder in the configuration) indeed just because gnuplot runs from this folder. But you should be able to add __ROBEXT_WAY_BACK__ that will be replaced with ../ (you can also directly add ../ but it would break if you change the folder of robustExternalize), like:

plot "__ROBEXT_WAY_BACK__yourfile.dat"`

I advise you not to directly put stuff in the cache folder as it might be removed if you clean it.

This workaround is perfekt. Thank you again for your quick response!

[tobiasBora]

The fact that the command succeeds is a bit more annoying. I guess that gnuplot creates the file even if the command fails, will try to find a solution.

[dflvunoooooo]

And for your other issues, I’ll get back to you later today, I have some stuff to do before.

No worries. You also implemented a working gnuplot environment and I am very happy. If you need further information from me, please tell.

[dflvunoooooo]

The fact that the command succeeds is a bit more annoying. I guess that gnuplot creates the file even if the command fails, will try to find a solution.

There is a pdf file created, but it is not a valid pdf, I can not open it. If I run the gnuplot code inside the robEx-xxxxx.tex file, the correct pdf is created.

[dflvunoooooo]

Is it possible to the set size to the latex length \textwidth? Or omit the size option? If I remove the terminal={} part, I get the error:

\writeRobExt=\write4
\openout4 = `robExt-15087A88FA1BD45424DC9265369B1364-out.tex'.

\boxRobExt=\box67

! Missing $ inserted.
<inserted text> 
                $
l.15 ^^Iset decimalsign locale "de_

Edit: It is working if I write terminal={}.

Edit2: To set the size to \textwidth with gnuplottex is rather complicated. It can be set with size \expandafter\cm\the\widthgnuplot,\expandafter\cm\the\heightgnuplot}. But for that one has to define in the preamble:

\newlength\textwidthcm
\newlength\widthgnuplot
\newlength\heightgnuplot
\textwidthcm=.03514598035146\textwidth             %-- conversion from pt to cm
\widthgnuplot=1\textwidthcm                                        %-- conversion to desired width
\heightgnuplot=0.75\widthgnuplot                   %-- keep aspectratio (prefactor) for height  
{\catcode`p=12 \catcode`t=12 \gdef\cm#1pt{#1cm}}% 

This solution is from here. But this does not work with robust-externalize. In the created gnuplot tex file is the size not replaced with the length, but written as \expandafter \cm \the \widthgnuplot, \expandafter \cm \the \heightgnuplot}. With a space before every .

[dflvunoooooo]

Really great work. Even the providevars feature of the tikz implementation of gnuplot-lua-tikz is working. :)

[tobiasBora]

There is a pdf file created, but it is not a valid pdf, I can not open it. If I run the gnuplot code inside the robEx-xxxxx.tex file, the correct pdf is created.

The fact that an invalid pdf is created even if the gnuplot command fails has been fixed in master.

This workaround is perfekt. Thank you again for your quick response!

I pushed a change so that you don't even need this workaround anymore. Now, you can just do:

\begin{CacheMeCode}{gnuplot, dependencies={gnuplot_data.dat}}
plot "gnuplot_data.dat"
\end{CacheMeCode}

(dependencies is needed to recompile the file if the data changes: if you do not add it it will not recompile automatically if you change the dependency file)

Or omit the size option?

You can omit the size, it will just default to the size in gnuplot (this line is basically sent as it to set terminal).

Is it possible to the set size to the latex length \textwidth?

In the very latest version, I just pushed a modification allowing you to write this directly (do not forget the expanded):

\begin{CacheMeCode}{gnuplot, tikz terminal/.expanded={size \lenToCm{\textwidth},\lenToCm{.3\textwidth}}}
plot cos(x)
\end{CacheMeCode}

[tobiasBora]

In the (even more recent) latest version, I also added support for cairolatex:

\begin{CacheMeCode}{gnuplot, cairolatex terminal/.expanded={size \lenToCm{\textwidth},\lenToCm{.3\textwidth}}}
plot cos(x)
\end{CacheMeCode}

So I guess I consider now the native support of gnuplot quite complete. If you have any other issue, please recreate a new issue as this one starts to be a bit of a mess.

[dflvunoooooo]

Absolutely great. Thank you for all your wok! Everything works as you described, beautiful! Is there some way to support you?

[tobiasBora]

No problem, thanks for your reports!

Is there some way to support you?

Sure:

• If you want, you can cite the arXiv version of this package (possibly using nocite, it should hopefully be online soon if arXiv does not consider this documentation as off-topic).
• I also just created a github sponsor page here if you want to support me financially ;-) (I can also send a Bitcoin address if it is easier)

Thanks a lot!

Btw, I added an option copy file to cache that you might like as it allows to copy a file to the cache folder. This might be practical to avoid manually copying the files generated by the gnuplot tikz terminal.