Advantages of vimserver for LaTeXMk compilation

[mvy]

I am reviewing the code for the LaTeXMk commands, to be able to port it to windows.

It seems that latex-box uses vimserver for background compilation. This requires the plugin to manage things like PIDs, program names, LaTeXMk running status and so on. Most of the commands uses "bashism" that are not always available on Linux and definitely not on windows.

I was asking me a question, I forward it to the @LaTeX-Box-Team

What are the benefits of using the current method instead of something like this : http://blog.chmd.fr/continuous-background-compilation-within-vim.html (with some adaptations obviously)

In that second method, I see the advantage of having a single command working on each systems. Furthermore, it seems to be lighter than the actual system. No need for vimserver, no need to check for LaTeXMk running status.

What do others devs think about this?

[mvy]

By the way the current method have the following issues:

37, #4, and #36

[jonasdiemer]

Not being a dev, I would find it nice to get rid of the vimserver dependency. It requires some extra work if command-line vim is used, i.e. compiling vim with server support and running it in server mode. This is confusing for a new user. Also, it seems overkill for such a simple task, many other plugins (e.g. python-mode) do background compilation in an easier way.

So to me it seems to be worth it to change to something else.

2013/1/9 Yves Stadler notifications@github.com

I am reviewing the code for the LaTeXMk commands, to be able to port it to windows.

It seems that latex-box uses vimserver for background compilation. This requires the plugin to manage things like PIDs, program names, LaTeXMk running status and so on. Most of the commands uses "bashism" that are not always available on Linux and definitely not on windows.

I was asking me a question, I forward it to the @LaTeX-Box-Teamhttps://github.com/LaTeX-Box-Team

What are the benefits of using the current method instead of something like this : http://blog.chmd.fr/continuous-background-compilation-within-vim.html(with some adaptations obviously)

In that second method, I see the advantage of having a single command working on each systems. Furthermore, it seems to be lighter than the actual system. No need for vimserver, no need to check for LaTeXMk running status.

What do others devs think about this?

— Reply to this email directly or view it on GitHubhttps://github.com/LaTeX-Box-Team/LaTeX-Box/issues/61.

[lervag]

If I understand correctly, you suggest to use something like:

augroup latexmk
    autocmd BufWritePost *.tex silent execute ":!latexmk " . expand('%') . " >out.log &"
augroup END

Correct? This does seem like a good idea. We could have an option for continuous compilation mode with an autocommand like the above and a simple mode which only runs latexmk once when <leader>ll is typed. With such an autocommand, we kind of don't need latexmk -pvc (although, that can be discussed - latexmk monitors more than just the tex-file).

[mvy]

Yes @lervag , something like this. Obviously, we should port the options set in the different existing variable of LaTeX-BoX/LaTeXMk like pdftex options and other. This should also remove the need to deal with the latexmkrc file of #36.

I am not familiar with the -pvc option of LaTeXMk, but maybe we can use it instead of the autocmd then? Not sure if this will be compatible with the error log for vim (:clist etc). If we can ensure that every run of -pvc generates a standalone log that vim can load, it should be fine.

[lervag]

Hi,

See here: http://www.phys.psu.edu/~collins/software/latexmk-jcc/latexmk-431.txt, last paragraph before the date (30 March 2012). -pvc tells latexmk to "run continuously, regularly monitoring all the source files to see if any have changed." It is very neat!

However, if latexmk is started with the -pvc option, then it will be a background process, and I think this is why the current version of LaTeXBox uses the vimserver in the first place, since LaTeXBox then needs to keep track of it and stop it when exiting (for instance).

I think the autogroup idea is good, and as far as I can see it will give a sufficient continuous solution.

Another idea could be to instead of using a vimserver, simple use shell commands to keep track of the latexmk process. I don't know if this could be portable, though. What I mean is to let <leader>ll do a similar

silent execute ":!latexmk " . g:options . expand('%') . " >out.log &"

where g:options could have -pvc. And then use something similar to

map <leader>lk :!killall latexmk

And of course also add autocommands to stop latexmk when vim is exited/stopped.

[mvy]

Or we could let the user decide with a g:latexmk_mode variable if he wants the continuous or on demand compilation for latexmk. For the first case, we run the -pvc on load and kill it on exit, for the on demand, we plug it to the <leader> commands.

I am a bit worried not to see an "output log" option for latexmk, though it should still generate the classic (pdf|la)tex log file anyway. If there is an known solution for vim quickfix with LaTeX format, we would be fine.

[lervag]

I would suggest that we open a separate branch and that we attempt to make a solution along the lines of this discussion.

[mvy]

I pushed a portable_lmk branch (exact copy of ^HEAD) for this purpose.

[mvy]

I made some pushes to the branch. Could you give me some feedback?

The functions should now call latexmk from vim (execute!) and take latexmkrc file into account.

[lervag]

Hi,

I'll try to have a look at it when I have more free time. I'm working on my PhD thesis at the moment, and I don't have very much spare time. Hopefully I will have some more time before the summer starts.

[mvy]

@lervag no problem. I knew that too :P

[jonasdiemer]

Hi, any news here? I just switched to a new machine and have to go through the hassle of building vim again manually because of the server requirement...

[jonasdiemer]

Actually, it seems like plainly "going through the hassle" does not cut it. There seems to be no version of console-vim for Mac that has a working server implementation (I tried macvim and homebrew vim). Any ideas?

[jonasdiemer]

I just tried out the branch, it seems to work on Mac OS. I tried running latexmk using <leader>ll. What's currently different is that the output of latexmk is shown during compilation. Don't know how to fix that yet (vimscript noob).

I just pushed a few minor changes to display errors in a quickfix list.

[lervag]

Hi,

I finally have some time to look at this now. I merged the master branch into this branch (before I noticed that @jonasdiemer already did this... sorry!). In addition I did a very minor update.

The branch does seem to work, however, I do not get it to work with the continuous option, that is, the '-pvc' flag for latexmk. In addition, if we start latexmk with the '-pvc' flag, we need some method to keep track of the process in order to kill it when we quit vim (or similar). This is supported with the vimserver in the master branch, but not in this branch.

I am curious: Do we really need the vimserver for the functionality that is currently in the master branch? As far as I understand, the vimserver is used to spawn background processes. Can we not do that just by prepending '&' to the command? For instance, something like this seems to work with me:

:silent!!make&

Where a test makefile contains the simple lines:

default:
        sleep 10
        date > tmp.txt

(Note: The makefile uses tabs, not spaces.)

I am also curious: Is it only the vimserver that is really a problem here?

[jonasdiemer]

Hi Karl,

regarding your second question: For me, vimserer is the only problem, as it is not (fully) available on all platforms I use (e.g. mac, some Linux/Unix servers on which I don't want to manually install vim).

Regarding your first question, I guess spawning a background process should work. Don't know how return value and outputs are handled, though. But again, there are tons of plugins that use background compilatiopn/linting without the vimserver feature...

Regards

2013/6/25 Karl Yngve Lervåg notifications@github.com

Hi,

I finally have some time to look at this now. I merged the master branch into this branch (before I noticed that @jonasdiemerhttps://github.com/jonasdiemeralready did this... sorry!). In addition I did a very minor update.

The branch does seem to work, however, I do not get it to work with the continuous option, that is, the '-pvc' flag for latexmk. In addition, if we start latexmk with the '-pvc' flag, we need some method to keep track of the process in order to kill it when we quit vim (or similar). This is supported with the vimserver in the master branch, but not in this branch.

I am curious: Do we really need the vimserver for the functionality that is currently in the master branch? As far as I understand, the vimserver is used to spawn background processes. Can we not do that just by prepending '&' to the command? For instance, something like this seems to work with me:

:silent!!make&

Where a test makefile contains the simple lines:

default: sleep 10 date > tmp.txt

(Note: The makefile uses tabs, not spaces.)

I am also curious: Is it only the vimserver that is really a problem here?

— Reply to this email directly or view it on GitHubhttps://github.com/LaTeX-Box-Team/LaTeX-Box/issues/61#issuecomment-19980004 .

[lervag]

Hi again,

2013/6/25 Jonas Diemer notifications@github.com

Hi Karl,

regarding your second question: For me, vimserer is the only problem, as it is not (fully) available on all platforms I use (e.g. mac, some Linux/Unix servers on which I don't want to manually install vim).

I see.

Regarding your first question, I guess spawning a background process should work. Don't know how return value and outputs are handled, though. But again, there are tons of plugins that use background compilatiopn/linting without the vimserver feature...

I agree, we should not need the vimserver feature. But I don't think we need to do very much to remove that dependency. I'll have a look at it one day and see if I'm right.

Regards

2013/6/25 Karl Yngve Lervåg notifications@github.com

Hi,

I finally have some time to look at this now. I merged the master branch into this branch (before I noticed that @jonasdiemer< https://github.com/jonasdiemer>already did this... sorry!). In addition I did a very minor update.

The branch does seem to work, however, I do not get it to work with the continuous option, that is, the '-pvc' flag for latexmk. In addition, if we start latexmk with the '-pvc' flag, we need some method to keep track of the process in order to kill it when we quit vim (or similar). This is supported with the vimserver in the master branch, but not in this branch.

I am curious: Do we really need the vimserver for the functionality that is currently in the master branch? As far as I understand, the vimserver is used to spawn background processes. Can we not do that just by prepending '&' to the command? For instance, something like this seems to work with me:

:silent!!make&

Where a test makefile contains the simple lines:

default: sleep 10 date > tmp.txt

(Note: The makefile uses tabs, not spaces.)

I am also curious: Is it only the vimserver that is really a problem here?

— Reply to this email directly or view it on GitHub< https://github.com/LaTeX-Box-Team/LaTeX-Box/issues/61#issuecomment-19980004>

.

— Reply to this email directly or view it on GitHubhttps://github.com/LaTeX-Box-Team/LaTeX-Box/issues/61#issuecomment-20001716 .

[lervag]

Hi,

I created a new branch where I attempted to do it the way I was thinking about. It took some time, and during the time I realize it is much the same as what @mvy already did. Except my version supports continuous mode with the latexmk flag '-pvc'.

@jonasdiemer A quick comment to your update with the log file: I don't think we need to save the output of latexmk to display the errors in the quickfix window. The errors can be parsed directly from the <basename>.log file.

As I am a full time linux user, it is a little bit difficult to check portability. In particular, I think there is one thing that might need some more work: The detection of the continuous process PID, which is needed in order to stop latexmk when wanted. At the moment, I use the utility pgrep, which is very handy. This should be available on all linux systems, I think, but it might not be available on MacOS? And on Windows I have really NO idea what is available/possible to use.

But for non-continuous mode, I think the current version should be both portable, and much simpler than the previous version.

Please test it and let me know what you think.

[jonasdiemer]

Hi,

my log file update was a quick shot, feel free to improve it. ;)

Regarding pgrep, it is available on MacOS, so no problem here. Windows is a different story, but Windows doesn't even have killall, does it? I guess we could get away with not supporting continuous compilation on Windows...

I will test your version asap.

Regads

2013/6/26 Karl Yngve Lervåg notifications@github.com

Hi,

I created a new branch where I attempted to do it the way I was thinking about. It took some time, and during the time I realize it is much the same as what @mvy https://github.com/mvy already did. Except my version supports continuous mode with the latexmk flag '-pvc'.

@jonasdiemer https://github.com/jonasdiemer A quick comment to your update with the log file: I don't think we need to save the output of latexmk to display the errors in the quickfix window. The errors can be parsed directly from the .log file.

As I am a full time linux user, it is a little bit difficult to check portability. In particular, I think there is one thing that might need some more work: The detection of the continuous process PID, which is needed in order to stop latexmk when wanted. At the moment, I use the utility pgrep, which is very handy. This should be available on all linux systems, I think, but it might not be available on MacOS? And on Windows I have really NO idea what is available/possible to use.

But for non-continuous mode, I think the current version should be both portable, and much simpler than the previous version.

Please test it and let me know what you think.

— Reply to this email directly or view it on GitHubhttps://github.com/LaTeX-Box-Team/LaTeX-Box/issues/61#issuecomment-20056359 .

[jonasdiemer]

@lervag, I just gave your version a try. After issuing <leader>ll, vim takes me to the terminal, but doesn't show anything. The build seems to lock up (wait for user interaction?), I can return to vim via q, ctrl-c and ctrl-d (some combination thereof, again a hint that latex went into "interactive mode").

Using the latest head from the portable_lmk branch, I have no problem...

[mvy]

You need -interaction=nonstopmode to prevent latex to ask for user input.

[lervag]

Hmm. That's strange. I have -interaction=nonstopmode in my .latexmkrc, so that might be it. Then I guess one should add that to the cmd as:

let cmd .= " -e '$pdflatex =~ s/ / -interaction=nonstopmode -file-line-error /'"
let cmd .= " -e '$latex =~ s/ / -interaction=nonstopmode -file-line-error /'"

@jonasdiemer: Could you try and see if this helps?

Else I wonder if I was too quick when I removed the -quiet flag? It didn't seem to do anything...

[mvy]

I removed the continuous compilation from portable_lmk because of this problem of stopping a process. It seems that the legacy version retried the process ID from $$ variable if I remember correctly. This is impossible to do on windows. The code also used some "bash-ism" that where not compatible with all terminals.

If we are gonna use process ID, we should take a look to ' tasklistandtaskkill` for windows. I'd better prefer a way to not explicitly call a kill on the program. But I have no idea for the moment.

[lervag]

I don't think my version uses any more "bash-ism"s than portable_lmk, and if pgrep is available on both linux and MacOS, then I guess we could settle with a version that works well on all platforms, but that does not support continuous compilation on windows?

At least, the above could be a solution until someone who knows windows systems well comes along and suggests a patch that allows us to stop background latexmk processes on windows.

It is easy to add a check for the continuous option, and to turn it off with a warning on windows systems, I think. And even though I agree that it is not a perfect solution to explicitly kill a program, I still think that it works well as a minimal solution. remove-vimserver:latexmk.vim is now only ~200 lines of code, which I think is getting close to a satisfyingly small amount. I would therefore propose that if we get my version to work as it should also for @jonasdiemer, then we might be close to having a solution that no longer depends on the vimserver, and that might soon be ready to be merged to the master branch?

[jonasdiemer]

@lervag, I agree. Just tried again the latest remove-vimserver, still compilation froze, I stopped it using ctrl-d. I should say that the document does contain errors, so compilation is expected to fail (but not freeze).

[jonasdiemer]

Just pushed a minor change LaTex-Box-Team/LaTeX-Box@9f397db407749205e7684d82c6c4bbd6473bb97e, which adds the -interaction=batchmode option to latexmk. This way, compilation runs fine even in case of errors.

[lervag]

Hi,

First, as response to your first comment: I didn't push an update, so the compilation should still freeze.

The reason I did not notice the error was simply that I have a .latexmkrc file that already gives the -interaction=nonstopmode flags. I would probably not have found it, though, since I did not try to compile an erroneous latex file. I have verified that your update works as it should.

I found in man latexmk that the flag -silent (or the synonym -quiet) should add the -interaction-nonstopmode or -interaction=batchmode flags to latex or pdflatex. And since this is the flag that is used in maser already, I decided to use it instead of -interaction-batchmode.

I pulled the recent updates in master into remove-vimserver, and also did some minor changes.

[lervag]

Back to topic: We now have two branches, portable_lmk and remove-vimserver. remove-vimserver is essentially the same as portable_lmk, except that it has kept support for continuous compilation. In both cases, the vimserver has been removed and the code should essentially be portable.

The question is: What do we do now?

I've read through the entire thread again now. In first issue description, @mvy sets the goal to make LatexBox work on windows. I personally don't use windows, and will not be able to do tests and check if things work. But as far as I understand, both portable_lmk and remove-vimserver should work on windows. The exception is that the continuous mode in remove-vimserver probably won't work, since it depends on the handling of PIDs with pgrep and kill.

In any case, I think remove-vimserver is a step in the right direction. There might be bugs or problems that I/we have not seen yet, but I will start to test the branch in my daily work. I think it could be merged into master in not too long.

[jonasdiemer]

Thanks, @lervag. The current 40878ffd4 of remove-vimserver works for me. I will keep using that branch to see if I have any issues.