Killing latexmk rather than disconnecting

[karlmoritz]

Hi,

I'm using atp-vim for writing latex code, and have now added latex_box in its own right for more up to date latexmk support. Loading both plugins in parallel works fine and everything works as expected.

However, I noticed that when closing vim, latexmk continues to run. I've removed atp-vim, but the issue remains. Running :LatexmkStop does not make any difference. It seems to disconnect latexmk from vim (LatexmkStatus says it has stopped), but the job remains running. "ps aux | grep latex" gives:

kmh       9083  0.0  0.0  16532   836 pts/0    S    23:15   0:00 /bin/bash -c  (vim -g --servername TEST --remote-expr '<SNR>35_LatexmkSetPID'"(\"/shared/Files/test\",$$)" ; export SUCCESSCMD='vim -g --servername TEST --remote-expr '\''<SNR>35_LatexmkCallback'\''"(\"/shared/Files/test\",0)"'        FAILURECMD='vim -g --servername TEST --remote-expr '\''<SNR>35_LatexmkCallback'\''"(\"/shared/Files/test\",1)"' ; cd '/shared/Files/test' && max_print_line=2000   latexmk -pdf -quiet  -pvc -e '$pdflatex =~ s/ / -file-line-error /' -e '$latex =~ s/ / -file-line-error /' -e '$success_cmd = $ENV{SUCCESSCMD}' -e '$failure_cmd = $ENV{FAILURECMD}' test.tex &>/dev/null) >/dev/null &
kmh       9086  2.1  0.2  43736  9224 pts/0    S    23:15   0:01 perl /usr/local/texlive/2012/bin/x86_64-linux/latexmk -pdf -quiet -pvc -e $pdflatex =~ s/ / -file-line-error / -e $latex =~ s/ / -file-line-error / -e $success_cmd = $ENV{SUCCESSCMD} -e $failure_cmd = $ENV{FAILURECMD} test.tex

My .vimrc is fairly vanilla, the only latexmk related settings are: let g:LatexBox_latexmk_preview_continuously = 1 let g:LatexBox_latexmk_async = 2

Is this an issue only I am experiencing?

[lervag]

Hi,

First: You are right, this is not supposed to happen. Could you make sure that atp-vim does not create a conflict? That is, try to reproduce your problem when atp-vim is not loaded.

If the problem persists:

1. What OS are you on?
2. What version of vim do you use?
3. Could you make a minimal working example? See below for a description of how to do this.

To make a minimal working example: Strip your vimrc-file to a minimal vimrc, see here for an example. Make sure to load the plugin and to set the plugin options you use. Then create a minimal tex-file, something like

\documentclass{minimal}
\begin{document}
Hello World
\end{document}

Now see if this reproduces the problem. If you can reproduce the problem in this manner, then you have a very strong proof that there is something wrong with LaTeX-Box.

[karlmoritz]

Hi, thanks for getting back to quickly and the suggestions for making a better issue report. I've tried to narrow this down as much as possible, but the problem still reproduces. I've pasted a minimal .vimrc to https://gist.github.com/karlmoritz/c8a4014e820ad6ec628b

I'm running Ubuntu 12.04 with VIM 7.4, Texlive 2012 and latexmk 4.33b

Loading the above minimal tex-file, "ll" will start the following jobs:

kmh       4898  0.0  0.2 181152 11796 pts/3    Sl+  10:00   0:00 vim --servername TEST test.tex
kmh       4903  0.0  0.0  16528   836 pts/3    S+   10:00   0:00 /bin/bash -c  (vim -g --servername TEST --remote-expr '<SNR>27_LatexmkSetPID'"(\"/home/kmh/test/test\",$$)" ; export SUCCESSCMD='vim -g --servername TEST --remote-expr '\''<SNR>27_LatexmkCallback'\''"(\"/home/kmh/test/test\",0)"'        FAILURECMD='vim -g --servername TEST --remote-expr '\''<SNR>27_LatexmkCallback'\''"(\"/home/kmh/test/test\",1)"' ; cd '/home/kmh/test' && max_print_line=2000   latexmk -pdf -quiet  -pvc -e '$pdflatex =~ s/ / -file-line-error /' -e '$latex =~ s/ / -file-line-error /' -e '$success_cmd = $ENV{SUCCESSCMD}' -e '$failure_cmd = $ENV{FAILURECMD}' test.tex &>/dev/null) >/dev/null &
kmh       4906  0.1  0.2  42756  8284 pts/3    S+   10:00   0:00 perl /usr/local/texlive/2012/bin/x86_64-linux/latexmk -pdf -quiet -pvc -e $pdflatex =~ s/ / -file-line-error / -e $latex =~ s/ / -file-line-error / -e $success_cmd = $ENV{SUCCESSCMD} -e $failure_cmd = $ENV{FAILURECMD} test.tex

"lk" then reports that latexmk was successfully stopped. To investigate a little further, I've modified the LatexmkStop function to also return the PID its attempting to stop. In the example above, it reports PID 4902, so if it's attempting to stop the bash job it is off by one. Similarly:

g:latexmk_running_pids {'/home/kmh/test/test': 4902}"

With async turned off, the error persists. However, when querying running pids I get this:

g:latexmk_running_pids {'/home/kmh/test/test': '54895490^@'}

with the matching part of ps aux containing:

kmh       5485  0.1  0.2 181152 11756 pts/3    Sl+  10:31   0:00 vim test.tex
kmh       5488  0.0  0.0  16524   664 pts/3    S+   10:31   0:00 /bin/bash -c cd '/home/kmh/test' && max_print_line=2000   latexmk -pdf -quiet  -pvc -e '$pdflatex =~ s/ / -file-line-error /' -e '$latex =~ s/ / -file-line-error /' -e '$success_cmd = $ENV{SUCCESSCMD}' -e '$failure_cmd = $ENV{FAILURECMD}' test.tex &>/dev/null &
kmh       5489  0.2  0.2  42756  8284 pts/3    S+   10:31   0:00 perl /usr/local/texlive/2012/bin/x86_64-linux/latexmk -pdf -quiet -pvc -e $pdflatex =~ s/ / -file-line-error / -e $latex =~ s/ / -file-line-error / -e $success_cmd = $ENV{SUCCESSCMD} -e $failure_cmd = $ENV{FAILURECMD} test.tex

The wrong PIDs stored don't seem to match any jobs (unless these jobs disappear before I have a change to list them). Stopping latexmk here, too, reports success but doesn't terminate the actual job.

Unfortunately that is as far as my VimL knowledge allowed me to investigate at this point.

[lervag]

Thanks for your input, I think we should be able to solve this with some effort. Did I understand correctly: You have now tested the same both with and without async? That is, with both of the following:

let g:LatexBox_latexmk_async = 2
let g:LatexBox_latexmk_async = 0

[Btw: Note that this option is really a switch, and that there should be no difference between = 2 and = 1, where the latter is preferred).]

It seems to be very likely that the problem here is that the recorded PIDs are wrong.

If the problem also affects the case where async is off, then I suggest we focus on this case first. This is simply because it should be easier to debug. If you look in latexmk.vim line 327, you will see how the continuous mode without async gets its PID. It uses pgrep -f "perl.*<mainfile-base-path>", which should in your case be something like pgrep -f "perl.*test". You could try these commands in your terminal after you have used <leader>ll and report the results?

In the async case, you could try to echo line 268, that is, add after line 268:

echom callsetpid

Finally, it is interesting to me to see if this problem is also reproducible with my own fork of LaTeX-Box. This fork does not support the async mode, but if I understood correctly the issue also affects the case where async is not used.

[karlmoritz]

Hey Karl, thanks for getting back to me on this. I think we're making progress here: I've added a line "echom pid" after the pid is assigned in the case without async. It outputs "43794380" when a subsequent pgrep ... only returns 4379, so it seems as if there were two perl scripts running very briefly and hence get recorded. When repeating the assignment and print command like this

                   let pid = substitute(system('pgrep -f "perl.*'
                                                    \ . mainfile . '"'),'\D','','')
                            echom "pid " . pid
                            let pid = substitute(system('pgrep -f "perl.*'
                                                    \ . mainfile . '"'),'\D','','')
                            echom "2pid " . pid
                            let pid = substitute(system('pgrep -f "perl.*'
                                                    \ . mainfile . '"'),'\D','','')
                            echom "3pid " . pid
                            let pid = substitute(system('pgrep -f "perl.*'
                                                    \ . mainfile . '"'),'\D','','')
                            echom "4pid " . pid
                            let g:latexmk_running_pids[basepath] = pid

I get different outputs, i.e. messages returns:

pid 44314432
2pid 44314434
3pid 44314436
4pid 44314438

I added a lot more assign/print steps like this, removing the substitution function and get this:

pid 49234924
2pid 49234926
3pid 49234928
4pid 4923^@4932
4pid 4923^@4936
4pid 4923^@4939
4pid 4923^@4941
4pid 4923^@4943
4pid 4923^@4945
4pid 4923^@4947
4pid 4923^@4949
...

So it seems as if the pgrep command somehow ends up either matching a system job related to itself. I modified the function by removing the substitution (\D), like this:

let pid = system('pgrep -f "perl.*'. mainfile . '"')

which now stores pid as:

g:latexmk_running_pids {'/home/kmh/test/test': '5175^@5176^@'}

This isn't very principled, but does the job, i.e. lk now kill the correct pid (5175). To fix the problem a little better, I instead added a "| head -n 1", like this:

let pid = substitute(system('pgrep -f "perl.*'
                    \ . mainfile . '" | head -n 1'),'\D','','')

Note, I also reintroduced the non-digit substitution. Not sure whether it is strictly necessary, but I assume you put it there for a reason. This now also works and seems to have fixed the bug for the non-async case. I'll fork and fix this in my branch in a minute and send you a pull request if you're happy to use this fix.

For the asynchronous case I'm a little more lost. I've added echom callsetpid as you suggested, which outputs

callsetpid '<SNR>27_LatexmkSetPID'"(\"/home/kmh/test/test\",$$)"

echoing vimsetpid gives:

vim -g --servername TEST --remote-expr '<SNR>27_LatexmkSetPID'"(\"/home/kmh/test/test\",$$)"

However, let g:latexmk_running_pids give me

g:latexmk_running_pids {'/home/kmh/test/test': 5337}

while pgrep ... gives 5341, so it seems that the callback stores the process id of some temporary job rather than of the actual latexmk instantiation. Next, I tried to look at the actual command sent to the server:

cmd vim -g --servername TEST --remote-expr '<SNR>27_LatexmkSetPID'"(\"/home/kmh/test/test\",$$)" ; export SUCCESSCMD='vim -g --servername TEST --remote-expr '\''<SNR>27_LatexmkCallback'\''"(\"/home/kmh/test/test",0)"'        FAILURECMD='vim -g --servername TEST --remote-expr '\''<SNR>27_LatexmkCallback'\''"(\"/home/kmh/test/test\",1)"' ; cd '/home/kmh/test' && max_print_line=2000   latexmk -pdf -quiet  -pvc -e '$pdflatex =~ s/ / -file-line-error /' -e '$latex =~ s/ / -file-line-error /' -e '$success_cmd = $ENV{SUCCESSCMD}' -e '$failure_cmd = $ENV{FAILURECMD}' test.tex &>/dev/null

Running this manually (without the /dev/null) while running a latexserver gives the following output:

0
Viewing pdf
For rule 'view', running '&if_source(  )' ...
sh: 1: acroread: not found
sh: 1: Syntax error: ")" unexpected

=== Watching for updated files. Use ctrl/C to stop ...

Not sure if this is an issue, though. Finally, I tried to execute the whole command as given:

! (vim -g --servername TEST --remote-expr '<SNR>27_LatexmkSetPID'"(\"/home/kmh/test/test\",$$)" ; export SUCCESSCMD='vim -g --servername TEST --remote-expr '\''<SNR>27_LatexmkCallback'\''"(\"/home/kmh/test/test",0)"'        FAILURECMD='vim -g --servername TEST --remote-expr '\''<SNR>27_LatexmkCallback'\''"(\"/home/kmh/test/test\",1)"' ; cd '/home/kmh/test' && max_print_line=2000   latexmk -pdf -quiet  -pvc -e '$pdflatex =~ s/ / -file-line-error /' -e '$latex =~ s/ / -file-line-error /' -e '$success_cmd = $ENV{SUCCESSCMD}' -e '$failure_cmd = $ENV{FAILURECMD}' test.tex ) > /dev/null &

followed by an "echo $!", which returns 5823 while the matching pgrep gives 5826, so again there is a discrepancy here. Still not sure exactly what's going on in the async case, but maybe the above us more useful to you or someone else who really understands vimL?

Cheers, Karl (also)

[lervag]

Hi Karl :),

Good work, now it remains to figure out what is wrong with the asynchronous case.

To be honest, I have no idea why it doesn't work for you. And it is very difficult to debug when I am not able to reproduce it. With me, the $$ will return the correct PID.

[karlmoritz]

Hey, I looked into this a little further but didn't make much headway. Could you elaborate a bit on what's going on in the asynchronous case? Your delete function seems to look for multiple processes associated with the main PID?

I've hacked together a fix by using pgrep together with the setpid callback function to manually set a:pid in the async case. However, the deletion function still didn't work (I think I ended up with a PID instead of a group id), so I replaced it with a simple kill command. This seems to do the trick, but I suspect I'm lacking some understanding of how the async case is supposed to work, so this probably won't be a generally applicable solution. It's on my branch if you want to have a look at it.

[lervag]

Hi,

What is happening in the async case is that we are creating function calls that are sent through the vim server. The call back function is called when latexmk is finished, in order to open/close the error window. $$ should be expanded when the command is run in a system shell, and it should expand to the previous command.

The vim server is a feature in vim that allows you to execute commands in vim remotely, e.g. with

$ vim --servername SERVERNAME --remote-expr SOME-VIM-EXPRESSION-OR-FUNCTION

I do not understand why it works with most people, but not with you.

A couple of things you can do to debug:

1. Remove silent from silent execute cmd, and also remove > /dev/null and possibly & (lines 297 and 306).
2. Try to use call system(cmd) instead of silent execute command.

[karlmoritz]

Ok, that makes sense. I played a little more with this, and if I remove the &, multiple calls of the command are made. Then $$ correctly captures the first of these calls. However, when removing the ampersand, only the last of these multiple calls survives, hence we end up storing the wrong PID. Why the long bash expression is executed multiple times I still don't know. Do you have any insight into this?

[lervag]

Ok, so I investigated a little bit. First, I removed the silent part of silent execute cmd and the & from the cmd string. Then I commented out the call to the error function in line 96 (to prevent the messages to be cleared from "compilation success" or "... failed" messages). This allowed me to capture the full command that is being issued (it is a beast!):

:! (gvim -g --servername GVIM1 --remote-expr '<SNR>149_LatexmkSetPID'"(\"/home/l
ervag/tmp/test\",$$)" ; export SUCCESSCMD='gvim -g --servername GVIM1 --remote-e
xpr '\''<SNR>149_LatexmkCallback'\''"(\"/home/lervag/tmp/test\",0)"'        FAIL
URECMD='gvim -g --servername GVIM1 --remote-expr '\''<SNR>149_LatexmkCallback'\'
'"(\"/home/lervag/tmp/test\",1)"' ; cd '/home/lervag/tmp' && max_print_line=2000
   latexmk -pdf -quiet  -pvc -e '$pdflatex =~ s/ / -file-line-error /' -e '$late
x =~ s/ / -file-line-error /' -e '$success_cmd = $ENV{SUCCESSCMD}' -e '$failure_
cmd = $ENV{FAILURECMD}' test.tex &>/dev/null) >/dev/null

You claim that the expression is executed multiple times, but this does not seem to happen with me. Are you able to reproduce what I pasted above, so that I may inspect the multiple commands? You could do the same and verify that you get a corresponding command (you could put both outputs in files and diff them, or something similar).

Again, I am not able to reproduce the problem, which makes it very difficult to debug.

Do you have any strange vimrc settings? I know you already tried with a minimal vimrc, but could you try once more without the VAM-stuff, but rather with the rtp commands I suggested in my first post see this example?

[lervag]

Oh, and I am still interested in knowing whether this issue also affects my fork. My fork does not have the async mode (I don't find it necessary), but it uses pgrep to capture the PID. It could have the same issue that you already fixed, but I have not experienced it myself. If you have the time, please let me know if it does.

[karlmoritz]

Hey, I looked at your fork just now. I'm still getting the wrong PID, but I made some progress on debugging. Just under line 104 in latemk.vim I added

echomsg system('pgrep -lf latexmk')

The output from that (cleaned up for newline) is:

6165 vim autoload/latex/latexmk.vim^@
6289 /bin/bash -c cd /home/kmh/test && max_print_line=2000 latexmk -pdf -quiet  -pvc  -e '$pdflatex =~ s/ / -file-line-error /' -e '$latex =~ s/ / -file-line-error /' test.tex &>/dev/null &^@
6290 perl /usr/local/texlive/2012/bin/x86_64-linux/latexmk -pdf -quiet -pvc -e $pdflatex =~ s/ / -file-line-error / -e $latex =~ s/ / -file-line-error / test.tex^@
6293 /bin/bash -c (pgrep -lf latexmk) >/tmp/vKDbrPs/1 2>&1

When I check the variable, in that instance the PID is set to 6291, so I assume that the pgrep in 104 also issues a "/bin/bash -c (pgrep -nf ...", which then messes up the process identification. For myself, I could obviously now fix this by removing the "-n" and applying a tail -n 2 | head -n 1, but that seems a bit strange.

Do you have any idea what could cause this behaviour? If you add the echomsg line from above, do you get a comparable output?

I think understanding/resolving this issue might help solve the problem with the original Latex-box package, too.

[lervag]

If I add the line as you suggest, the output I get is the following (with added newlines):

4470 perl

With a full ps, I get:

---> ps aux|grep latexmk
lervag    4470  0.1  0.0  39528  9388 ?        SN   13:49   0:00 perl /usr/bin/latexmk -pdf -quiet -pvc -e $pdflatex =~ s/ / -file-line-error / -e $latex =~ s/ / -file-line-error / tmp.tex
lervag    4602  0.0  0.0   7264   984 pts/10   S+   13:49   0:00 egrep -i latexmk

So there is something very strange going on. It seems like your vim does strange things. But let's start to rule out certain things, for instance, which version of pgrep do you have? I have pgrep from procps-ng 3.3.9. Do you have any strange .bashrc settings or .zshrc settings? Are you using vim or gvim (this should not matter, and indeed it works in both vim and gvim for me).

[karlmoritz]

This is very surprising: I would have expected that you would at least get the full command from the perl command, but that doesn't matter. The more important difference it seems is that the /bin/bash commands don't appear in your case. I've just tested again with an empty vimrc and only your plugin in .vim, and get the same behaviour as above.

Will look into this a little further.

pgrep is from procps 3.2.8 . I don't have a .zshrc file and the .bashrc is fairly boring

[karlmoritz]

I think I might have solved the problem.

What is your shell setting in vim? :set shell ? gives /bin/bash in my case. I'm wondering if that might be the root of the problem.

For instance, if I execute :let x = system('find / -name "*tex"'), and then look at my processes with ps aux, I get the following:

kmh       7893  0.0  0.0  16512  1204 pts/1    S+   13:45   0:00 /bin/bash -c (find / -name "*tex") >/tmp/v2S9FO7/1 2>&1
kmh       7894  8.0  0.0  18956  1232 pts/1    S+   13:45   0:00 find / -name *tex
kmh       7895  0.0  0.0  22360  1284 pts/4    R+   13:46   0:00 ps aux

Now, if I use a different shell, say :set shell=/usr/bin/ksh and repeat the command, I only capture the find but not the shell wrapping command with ps aux. ksh doesn't start any of the latex-box commands.

Finally, setting shell to /usr/bin/zsh does the trick. Now, pgrep captures the process correctly.

Could you check what shell your vim uses by default and if it's not bash, try setting it to bash? If I'm correct, this would allow you to replicate the bug.

[karlmoritz]

Just to clarify, this fix unfortunately doesn't extend to the LaTeX-Box plugin, but only to your branch. The LaTeX-Box version is still off by one, storing 5449 when it should store 5450 etc. using zsh

[lervag]

I can confirm that the issue is reproduced with shell=/usr/bin/bash (which is the path for bash on my box). Thus I can fix the issue in my fork.

However, for LaTeX-Box with async mode, it is still not as easy. Could you try to change your default shell to zsh, and then test the async mode? I think the problem is that with bash as a default shell, we get some more "spurious" processes, and so the $$ does not capture the PID of the correct command.

[karlmoritz]

Glad to hear that you can reproduce it!

Yes, LaTeX-Box is still funny - I've just reinstalled it in async mode, using zsh all the way through (instantiating vim out of a zsh shell with SHELL=/usr/bin/zsh and :set shell=/usr/bin/zsh but there still appears to be a spurious process causing an off-by-one error. Maybe it would be best to change the callback here so that $$ is replaced by a grep command, too?

In terms of fixing the issue in either system, I think it might be a good idea to replace the pgrep command with a "pgrep -lf latexmk | grep perl | tail -n 1". This would first return all commands, then only the ones that expanded latexmk to perl and of those the latest. Maybe there is a more elegant solution, but this one should work across all shells I believe.

[lervag]

I fixed it in vim-latex already with a slightly more complicated pgrep command. That is, at least it works for me with shell=/usr/bin/bash.

Yes, it could be possible to change the callback to use ps|grep or pgrep instead of $$. That would be something like

let callsetpid = shellescape(setpidfunc).'"(\"'.basepath.'\",$(pgrep ...))"'

[karlmoritz]

Hmm, I'm not sure if that approach works. I've just tried that, but wouldn't this execute the pgrep the moment that this command is defined, i.e. before the actual process is launched. I tried an alternative in my fork of LaTeX-Box, but with no immediate success. Will leave this for now.

I'm glad it works for latex-box now, and will be using that version for the time being. I agree that that's a much cleaner implementation! Thanks for fixing that just now :)

[lervag]

It does sort of work, actually. I am able to run pgrep with $(pgrep ...) instead of $$. However, it seems that the command is run before latexmk starts, and so it does not get the PID of the latexmk command.

I am sorry to say that I give up on this issue. At least we made some progress. Good luck to whoever tries to solve this!

[tisaac]

I was having the same issue (at commit c7809), but fixed this by substituting '$PPID' for '$$'. I don't know if this is a universal fix: I use zsh. Maybe the same logic is needed as in setup_vim_server() ?

[bsdbeard]

I have the exact same issues, with shell=/bin/zsh and shell=/bin/bash, the g:latexmk_running_pids variable is off by one, that is if it reports the PID is 21654, the real PID is 21655, as reported by ps ax | grep perl.

I have not found a fix that fixes this other than changing

function! s:LatexmkSetPID(basename, pid)
    let g:latexmk_running_pids[a:basename] = a:pid
endfunction

to

function! s:LatexmkSetPID(basename, pid)
    let g:latexmk_running_pids[a:basename] = a:pid + 1
endfunction

which is very unsafe, changing '$$' to '$PPID' did not help, neither did the other suggestions.

This is the output I get:

    echo g:latexmk_running_pids
    {'/home/bsdbeard/blahblahblah/': 23041}

    > ps ax | grep perl
    23042 pts/5    SN+    0:02 perl /usr/local/texlive/2013/bin/i386-linux/latexmk -pdf -quiet -auxdir=./aux -outdir=./aux -pdf -pdflatex=pdflatex -shell-escape -halt-on-error %O %S -pvc -e $pdflatex =~ s/ / -file-line-error / -e $latex =~ s/ / -file-line-error / -e $success_cmd = $ENV{SUCCESSCMD} -e $failure_cmd = $ENV{FAILURECMD} blahblah.tex
    23109 pts/2    S+     0:00 grep -Pi --color=auto perl

I'm willing to fix this, however I do not understand the logic used in the vim server code, why is the PID set by a callback (vimsetpid), isn't it easier to set the PID right after running latexmk? I think a much better way to handle all of this is to simply run latexmk with vimproc asynchronously, this would remove the need for a vim server completely. I will try doing this in my own fork, however I'm not entirely sure if it will work.

EDIT:

This problem does not occur in lervag's fork vim-latex, it correctly gets the PID and correctly kills latexmk.

[nilsbecker]

I have a similar problem but maybe not exactly the same: I am using g:LatexBox_latexmk_async=0. I can successfully kill the latexmk perl process with :LatexMkStop. However this does not work when quitting vim with :q when editing a tex file. I then get a stale perl process still running.