lol768 / regexp-refinement-types

Dissertation project: Regular expression refinement types
GNU General Public License v3.0
2 stars 1 forks source link

Compiling the document with these \PYG... commands #4

Closed PHPirates closed 5 years ago

PHPirates commented 5 years ago

Hey, I was using your document as a test document, but when I tried to compile it (using xelatex and shell escape) your use of these \PYGidseven commands confused me, are those internal minted commands? I don't know enough of pygments :/ They get me Undefined control sequence. ...ED\PYGidsevenZus {}META}\PYGidseven

I did find that idseven is the pygments style. I found https://github.com/gpoore/minted/issues/230 but it didn't get any clearer.

I can reproduce the problem when I compile the following

\documentclass{article}

\usepackage{minted}
\usemintedstyle{idseven}

% Compile using XeLaTeX with -shell-escape
\begin{document}
        \begin{Verbatim}[commandchars=\\\{\}]
            \PYGidseven{k+kd}{public}
        \end{Verbatim}
\end{document}

[Edit] I hope it's no problem for your supervisor that I am trying to compile it... I won't distribute the result, of course.

lol768 commented 5 years ago

Hey @PHPirates, sorry for the belated reply! I've been busy with some exams and some travel.

You're quite right in that they're internal minted commands. As part of this dissertation work, I wrote a style which I'm guessing you already found over at https://github.com/lol768/id7-pygments-style/

https://github.com/gpoore/minted/issues/230 is indeed the issue motivating the TeX I ended up writing. The basic problem is:

My code samples are not just code, they use a bunch of LaTeX inside the code sample to highlight things. For example, here's a SQL injection vulnerability with the vulnerable part of the code highlighted:

Screenshot from 2019-05-19 22-16-54

The red box is a \colorbox and an \faIcon within the code sample. Unfortunately, only some lexers provided by Pygments correctly handle this. There is a long-standing bug in Pygments that was raised 4 years ago which means that I can't use escapeinside for languages like C# or Java. Instead, I manually run Minted on just the bare source code, copy the generated output (from _minted-document) and then manually paste it into the LaTeX document already-highlighted. From here, I can then add my annotations and other custom code.

But! This doesn't "just work". The problem is that Pygments lazy-loads the \PYG<x> commands. When you do what I just did and try and paste the generated output and remove the original minted environment, nothing is loaded! Hence, the compile fails because it doesn't understand any of the commands.

The solution is to include the code twice. Once with no annotations, in a way that's hidden from the final document (imagine display: none, but for LaTeX, if you're familiar with CSS) in order to trick Minted into thinking the commands are necessary and loading them. And then another time, which appears in the final document, with all the necessary annotations and modifications.

The custom id7 Pygments style (which is named after the university's 7th brand iteration) complicates things a bit further, but ensures that the code samples look pretty and are consistent with the rest of the document. You need to have the style installed for any of this to work, since we use \usemintedstyle{idseven} to load it in the preamble.

The reason your example doesn't work is because the \PYGidseven{k+kd} definition isn't loaded. In Pygments land, k is a keyword and kd is a Keyword.Declaration. So we can make your minimal example compile by doing something like this:

\documentclass{article}

\usepackage{minted}
\usemintedstyle{idseven}

% Compile using XeLaTeX with -shell-escape
\begin{document}
        \begin{minted}{java}
            public class MyClass {
                // exists solely to pre-load the k+kd definition
            }
        \end{minted}

        \begin{Verbatim}[commandchars=\\\{\}]
            \PYGidseven{k+kd}{public}
        \end{Verbatim}
\end{document}

In this example, we first have minted compile a simple java class. The public keyword in this code sample is enough for minted to realise that it needs to load the k+kd style, and the subsequent Verbatim environment works just fine.

In addition to this fun hackery, you'll notice that some code samples are highlighted using an environment like this:

\begin{minted}{rrtlex.py:RrtLexer -x}
    % ...
\end{minted}

This one is hilarious - minted doesn't support custom lexers. So, we take advantage of a command injection 'vulnerability' in the minted LaTeX package to inject an additional argument into the pygmentize command that's invoked to do the actual nitty-gritty highlighting work. This allows us to specify our own lexer file to use, which was written specifically for the dissertation.

lol768 commented 5 years ago

[Edit] I hope it's no problem for your supervisor that I am trying to compile it... I won't distribute the result, of course.

@mbg - for context, @PHPirates contributes to the LaTeX IntelliJ plugin I used to write most of the report (which you should totally check out one day :stuck_out_tongue:). I added him here so we could try and reproduce some bugs I'd been experiencing.

From my perspective, I don't see a problem with you doing what you want with the report, @PHPirates. I intend on open sourcing all of this work later on.

mbg commented 5 years ago

I see no problem with this!

PHPirates commented 5 years ago

No worries about any delay, I'm not in any hurry, and your exams are more important!

Oh, wow. That's some advanced minted use :smile: Thanks for the in-depth explanation, I see the general idea of what you're doing now, I had not realised that the \PYG commands were right there in the generated output, but I see that when I copy them back they indeed start working when the original minted environment is also there (hidden or not).

I guess this would be useful information if you want to open source things :)

But, I'm still not sure I am using the custom style properly. I just installed it to my system-wide Python, but I get the following error when trying to run the example file you give:

*** Error while highlighting:
AttributeError: module 'id7' has no attribute 'IDSevenStyle'
   (file "/usr/lib/python3.7/site-packages/id7-1.1-py3.7.egg/id7/__init__.py", line 3, in <module>)
*** If this is a bug you want to report, please rerun with -v.

So it seems to my it does find id7-1.1-py3.7.egg but somehow it's not doing what it expects. And the pygments output still contains only \PYG and no \PYGidseven.

Yeah, collaborator of TeXiFy and author of https://github.com/PHPirates/travis-ci-latex-pdf ... not sure as which of these persons I am interested in your dissertation. But it's proving to be one of the better test files I've seen so far! (testing using avenir next and ubuntu mono on Travis atm, I think I should be able to make that work somehow)

lol768 commented 5 years ago

https://github.com/PHPirates/travis-ci-latex-pdf - ah, this is cool! I wanted CI -> PDF to work for this project. I did try using Tectonic but eventually gave up because I don't think it does -shell-escape. Will investigate your repo a bit more thoroughly in due course.

The error you're seeing seems to be related to the Python version. I can reproduce with v3, but not v2.

Reproduction steps for the AttributeError The steps I took to reproduce were: (On a fresh Ubuntu 19.04 install with TeXLive 2019) ``` $ sudo apt install python3-pip $ sudo pip3 install pygments $ wget https://github.com/lol768/id7-pygments-style/archive/master.zip $ unzip master.zip $ rm master.zip $ cd id7-pygments-style-master/ $ sudo python3 setup.py install $ python3 ``` At the REPL: ``` import id7 ```

I think I've fixed this in the style repo. Reinstalling the package and running xelatex -shell-escape example.tex seems to work for me now! I think my previous install must've used Python2 by default.

(testing using avenir next and ubuntu mono on Travis atm, I think I should be able to make that work somehow)

Those two are probably fair enough to get working. The sans font is in a bit of a .. grey area.

PHPirates commented 5 years ago

Thanks for the fix! The little example works now for me too, so that's great.

Yeah and it's unlikely that Tectonic would work with such a complicated document anyway. I think it would be best to stay as close as possible to a local installation, so installing tex live using the install script and then cache it.

But, I still don't even manage to compile the document locally. You spoke about the \faIcons, but where are those commands coming from? I don't see them in the minted output. I still get things like

./motivation.tex:88: Undefined control sequence.
<argument> \faExclamationTriangle 

The good thing is, once it works on my machine it's likely to work on Travis as well :)

[Edit] I just realised those must be font awesome icons, I probably did something wrong there. But I installed the fontawesome package and have it in my fonts folder. Investigating... The actual error is kpathsea:make_tex: Invalid filename FontAwesome5ProSolid:script=latn;, contains ':'

lol768 commented 5 years ago

Specifically, you want the fontawesome5 package (not just ,fontawesome). Note I load it with the pro option, which then relies on the system fonts rather than those bundled with the package. Happy to provide some TTFs if you want them

lol768 commented 5 years ago

Font files: fa5.zip

I don't see this any different to publishing the compiled PDF. Obviously there are boring legal implications if you e.g. go on to use these fonts commercially without your own license :stuck_out_tongue:

PHPirates commented 5 years ago

Thanks for the fonts, though I think you want to remove the attachment again just be sure you don't have any legal implications hanging around :) I had not realised that front awesome pro is not free.

It still doesn't really work for me when I install those (same error) - but even if it did, I don't see any easy way to use non-public fonts in a Travis build? I mean, you could store a secret environment variable which gives access to some non-public storage, but I don't think this is easy. I also don't think it is a good idea to commit those fonts to a private git repository like this, just in case it ever gets public (maybe if your pro account expires or something).

lol768 commented 5 years ago

@PHPirates It's an interesting one, but as I mentioned, you'd be redistributing these font files in the same way if you used them on a web-page or inside a document. The browser/document viewer needs to render them somehow, right? If folks use your git repo/document/website to extract the fonts out and violate the license, I see that as their problem. Of course, I'm not doing a law degree :laughing:

In terms of Travis support, the package doesn't need to take the [pro] option. If it's omitted, it'll use the free fonts shipped with the package without requiring system installation. So yeah, supporting random custom fonts is probably out of scope for your work (Proxima Nova is another problem we haven't discussed) but I think that's just a consequence of me being awkward with how I put the document together :stuck_out_tongue:. I can't see many other projects doing the same.

PHPirates commented 5 years ago

Hm yeah that makes sense, hadn't thought about that. In the meantime I made the other fonts compile (I didn't really check if they actually appeared in the document, I'm not really good in recognising fonts) on Travis: https://github.com/PHPirates/travis-ci-latex-pdf#using-custom-fonts - it's pretty simple once you remember to install luaotfload when using lualatex...

I know, but you use pro icons so I still want to be able to compile those :) I should try on my Ubuntu system though, maybe it's just my Arch that doesn't want to recognise the pro font.

For reference, I think brackets-curly seems a pro icon, so:

\documentclass{article}
\usepackage[pro]{fontawesome5}
\begin{document}
    \faIcon{brackets-curly}
\end{document}