VimTex can't find log file

tyalie commented 1 year ago

Description

VimTex cannot find the log file when latexmk is used together with $emulate_aux=1. When using :VimtexErrors (or similar) in Vim, the message VimTeX: No log file found appears.

Steps to reproduce

Create .latexmk with:

$emulate_aux = 1
$out_dir = "gen"
$aux_dir = ".aux"

open vim and compile
try to use :VimtexErrors

Expected behavior

No response

Actual behavior

No response

Do you use a latexmkrc file?

Yes

VimtexInfo

System info:
  OS: Ubuntu 22.04.1 LTS
  Vim version: NVIM v0.7.2
  Has clientserver: true
  Servername: /tmp/nvimo5jleA/0

VimTeX project: main
  base: main.tex
  root: /project
  tex: /project/main.tex
  main parser: current file verified
  document class: IEEEtran
  packages: base
  source files:
    main.tex
    chapters/01_introduction.tex
  compiler: latexmk
    engine: -lualatex
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    build_dir: gen
    callback: 1
    continuous: 1
    executable: latexmk
    job: 
      jobid: 19
      output: /tmp/nvimo5jleA/2
      cmd: max_print_line=2000 latexmk -verbose -file-line-error -synctex=1 -interaction=nonstopmode -lualatex -outdir=gen -pvc -view=none -e '$compiling_cmd = ($compiling_cmd ? $compiling_cmd . " ; " : "") . "echo vimtex_compiler_callback_compiling"' -e '$success_cmd = ($success_cmd ? $success_cmd . " ; " : "") . "echo vimtex_compiler_callback_success"' -e '$failure_cmd = ($failure_cmd ? $failure_cmd . " ; " : "") . "echo vimtex_compiler_callback_failure"' 'main.tex'
      pid: 1460577
  viewer: General
  qf method: LaTeX logfilel

lervag commented 1 year ago

VimTex cannot find the log file when latexmk is used together with $emulate_aux=1. When using :VimtexErrors (or similar) in Vim, the message VimTeX: No log file found appears.

Create .latexmk with:
$emulate_aux = 1
$out_dir = "gen"
$aux_dir = ".aux"

Currently, only the $out_dir option of latexmk (and latex utilities in general) is supported. In fact, I was not aware of $aux_dir and of the distinction. So, for now, things should work well if you remove the $aux_dir option (in which case $emulate_aux becomes irrelevant).

I'm curious, what's the point or idea behind using separate $aux_dir and $out_dir? Why don't you just only $out_dir?

tyalie commented 1 year ago

I'm curious, what's the point or idea behind using separate $aux_dir and $out_dir? Why don't you just only $out_dir?

I'm using this option in order to separate the temporary output files in aux like the log, ... from the relevant outputs (i.e. pdf) put into gen.

That allows me to declutter my directory tree. I commit the generated output into my git and the .aux output directory is cleanly hidden from view.

Currently, only the $out_dir option of latexmk (and latex utilities in general) is supported.

I think that stems from a mistake how this plugin handles the latexmkrc. The file is a perl script that is being executed at runtime and as such the build and aux dir are relatively unpredictable as they can be influenced by external circumstances quite easily.

I for example use it to change the job name (name of output pdf) depending on whether this is a production or editing build. In the former I remove any notes, scribbles and co automatically.

For exactly that purpose latexmk has the -dir-report and -rules command line arguments. Especially the latter lists the location of all output files corresponding to the their targets. If we could parse those instead we can fully circumvent the parsing of the latexmkrc and don't need to worry about any (future) weird output bending features.

lervag commented 1 year ago

I'm curious, what's the point or idea behind using separate $aux_dir and $out_dir? Why don't you just only $out_dir?

I'm using this option in order to separate the temporary output files in aux like the log, ... from the relevant outputs (i.e. pdf) put into gen.

That allows me to declutter my directory tree. I commit the generated output into my git and the .aux output directory is cleanly hidden from view.

Ok, I agree this makes sense and I can understand that some people may want this type of workflow (even if I don't).

Currently, only the $out_dir option of latexmk (and latex utilities in general) is supported.

I think that stems from a mistake how this plugin handles the latexmkrc.

No, the reason only $out_dir is supported is because I was never fully aware of $aux_dir and noone (until now) raised an issue related to it.

So, if I were to add support for a separation of the $aux_dir from the $out_dir, I would first have to make some partly substantial updates to VimTeX. Then my initial thought would be to parse the .latexmkrc file similar to how I already do that for $out_dir. I don't see any reason why this approach should not work well?

lervag commented 1 year ago

Hmm. I notice this in the latexmk manual:

       $aux_dir [""]
              The  directory in which auxiliary files (aux, log, etc) are to be
              written by a run of *latex.  If this variable  is  not  set,  but
              $out_dir  is  set, then $aux_dir is set to $out_dir, which is the
              directory to which general output files are to be written.

              Important note:   The  effect  of  $aux_dir,  if  different  from
              $out_dir,  is achieved by giving *latex the -aux-directory.  Cur‐
              rently (Dec. 2011 and later) this only works on the  MiKTeX  ver‐
              sion of *latex.

Specifically: This option seems to be only valid with MikTeX. That's unfortunate, and it makes me skeptical of putting in the effort to implement this - at least at a general level. Perhaps we can do it with some less rigorous manner, e.g. with a callback to allow user customization, or with an option to specify the location explicitly.

lervag commented 1 year ago

Ah, sorry, I now understand the point of $emulate_aux.

lervag commented 1 year ago

Sorry about the flow of comments. I'm writing them as I'm looking into this, instead of collecting at the end. I hope you don't mind too much!

Now, with this simple latexmkrc file:

$emulate_aux = 1;
$out_dir = "out";
$aux_dir = "aux";

and a minimal LaTeX file, I get the following output after compiling with latexmk:

> tree --dirsfirst
.
├── aux
│   ├── main.aux
│   ├── main.fdb_latexmk
│   └── main.log
├── out
│   ├── main.fls
│   ├── main.pdf
│   └── main.synctex.gz
├── latexmkrc
├── main.tex
└── minivimrc

Notice specifically that the out/ directory has several files. I find in the manual under $emulate_aux:

              Aux_directory  emulation  means  that when *latex is invoked, the
              output directory provided to *latex is set to be the aux_dir. Af‐
              ter  that, any files that need to be in the output directory will
              be moved there. (These are the files with extensions  .dvi,  .ps,
              .pdf, .fls, .synctex, .synctex.gz.)

So, from the last paranthesis, it seems this is the intended behaviour. Thus, to properly support the aux_dir feature, I would need something like this:

VimTeX must be changed in the core to support both a build directory and an output directory (currently, these are considered one and the same).
- Technically, these directories should be considered as two attributes of the compiler module.
- If an output directory is specified, but not a build directory, then the build directory is the same as the output directory (similar to how latexmk works).
- We should support an environmental variable $VIMTEX_BUILD_DIRECTORY similar to the current $VIMTEX_OUTPUT_DIRECTORY.
VimTeX has a function to get any auxiliary file. I should separate this into two functions: a. b:vimtex.get_aux_file(extension) - to get an auxiliary file. Corresponds to the $aux_dir location. b. b:vimtex.get_out_file(extension) - to get an output file. Corresponds to the $out_dir location.
The latexmk module for VimTeX should parse both $out_dir and $aux_dir from the .latexmkrc file (both local and global).
The docs will require an update to accommodate these changes.

tyalie commented 1 year ago

Then my initial thought would be to parse the .latexmkrc file similar to how I already do that for $out_dir. I don't see any reason why this approach should not work well?

Eh an easy example why this doesn't work. The $jobname variable in my .latexmkrc is dynamically generated according to environment variables. With a simple parsing approach you won't be able to find the output pdf, except when you start to each "find all pdfs" but that could get messy

lervag commented 1 year ago

Eh an easy example why this doesn't work. …

Your reply is a little upsetting to me, because you respond to only a part of what I wrote, and you do so in a tone that comes off as somewhat "entitled". I believe I've written some sensible thoughts about how to proceed on this issue. Did you have any thoughts about the outline for how to proceed?

Please note that the issue is more complicated than just the communication between latexmk and VimTeX. VimTeX itself must be changed to accommodate the idea of having separated output and auxilliary directories. This must be done in a general way independent of latexmk. Only thereafter can I think about good ways to implement support for latexmk, and I believe my original first thought is a simple and useful first step. Perhaps we want or need something more complicated, but we never needed a "preprocessing" step with latexmk -dir-report or similar before, so adding it now will be overkill for more or less every existing workflow with VimTeX.

Finally, please consider this: I've already spent quite a lot of time reading your original issue, thinking about it and writing some thoughts on how to possibly act in order to improve VimTeX in a way that could resolve your issue. I've also clearly stated that what you want will require substantial changes, which means it will take time. Time is a resource I unfortunately do not have very much of. I do like very much to work on and improve VimTeX for the sake of the users and community, as it is generally a rewarding experience in which I often learn interesting things. But when it does not feel rewarding, then why would I want to spend my time on it?

tyalie commented 1 year ago

Hey I'm sorry for the miscommunication. I haven't had much time to fully read and understand all you wrote and just answered on the most pressing issue for me.

I didn't make the assumption from my side that you'll need to fix it - I'm sorry that this came across differently. Tbqh the thing that holds me back to try my own solutions is that this plugin is written in Vim Script. I'm mostly here to discuss possible solutions, even if they are never really implemented.

The thing is just that parsing doesn't work for latexmk config files. I can write you a single line .latexmkrc that is not parsable with reasonable amount of work without using any help. I'm honestly not sure if "would be overkill" is the right answer here.

I've looked into the source of VSCode's LaTeX Workspace and they go the easy route by ignoring any concept of .log files and just parsing the compiler output of latexmk (which mostly equals the log). From that one could also get the current output pdf file name.

I'm aware that this would mean quite substantial changes in the way this plugin works - so honestly if you don't wanna do it don't.

What would work for me would honestly be an option to manually configure the location of the .log file. If something like that is already implemented that would be nice to know.

lervag commented 1 year ago

Hey I'm sorry for the miscommunication. I haven't had much time to fully read and understand all you wrote and just answered on the most pressing issue for me.

I didn't make the assumption from my side that you'll need to fix it - I'm sorry that this came across differently. Tbqh the thing that holds me back to try my own solutions is that this plugin is written in Vim Script. I'm mostly here to discuss possible solutions, even if they are never really implemented.

Ok, thanks!

I'm aware that this would mean quite substantial changes in the way this plugin works - so honestly if you don't wanna do it don't.

I don't mind making changes, even substantial, if the changes clearly improve VimTeX either from a maintenance standpoint or from a user perspective.

What would work for me would honestly be an option to manually configure the location of the .log file. If something like that is already implemented that would be nice to know.

Partly, but not in the way you want. You can set an environment variable VIMTEX_OUTPUT_DIRECTORY, which will override whatever is in your VimTeX configuration in e.g. init.vim or .vimrc and the $out_dir from any latexmkrc file. Thus, you could use this to make the log file be parseable, but then it will break the viewer because it will not find the pdf file.

I believe it may be useful to align VimTeX with latexmk in the sense of defining an output directory (build_dir) that is similar to latexmk's $out_dir and an additional auxiliary directory (e.g. aux_dir) similar to latexmk's $aux_dir. These would work the same, so that the log file would be found under aux_dir and the pdf under build_dir. If we start here, it should be a first step towards allowing what you need.

I think the behaviour should also mimic latexmk: If build_dir is specified, but not aux_dir, then aux_dir should be set to build_dir.

I would need to think about how this will affect the other build tools (latexrun, arara, etc). These do not necessarily have any concept of these directories, and we need to handle that issue.

lervag commented 1 year ago

Sorry for not responding again in quite some time. I've been very busy with other things (other issues, other projects, life, and so on).

I've quickly read the thread again. I forgot to answer your question:

What would work for me would honestly be an option to manually configure the location of the .log file. If something like that is already implemented that would be nice to know.

No, this does not exist today. To me, it does not seem like an option I would want to add, because VimTeX should support different configurations in different projects. E.g., if you have a local .latexmkrc file that specifies an out directory (that is currently supported), we should respect it.

Thus, for the present issue, I again propose the following actions to add support for $aux_dir:

VimTeX must be changed in the core to support both a build directory and an output directory (currently, these are considered one and the same).
- Technically, these directories should be considered as two attributes of the compiler module.
- If an output directory is specified, but not a build directory, then the build directory is the same as the output directory (similar to how latexmk works).
- We should support an environmental variable $VIMTEX_BUILD_DIRECTORY similar to the current $VIMTEX_OUTPUT_DIRECTORY.
VimTeX has a function to get any auxiliary file. I should separate this into two functions:
1. b:vimtex.get_aux_file(extension) - to get an auxiliary file. Corresponds to the $aux_dir location.
2. b:vimtex.get_out_file(extension) - to get an output file. Corresponds to the $out_dir location.
The latexmk module for VimTeX should parse both $out_dir and $aux_dir from the .latexmkrc file (both local and global).
The docs will require an update to accommodate these changes.

This will, as you point out, not work if you use $jobname:

The $jobname variable in my .latexmkrc is dynamically generated according to environment variables.

However, this was not part of the initial request, and I believe it may be possible to handle it after applying the proposed changes.

If you do not really care about any updates, then please let me know. I will not implement these things if they are not of interest to anyone.

tyalie commented 1 year ago

I think that change would be lovely. The jobname can be happily ignored for now, as it is only a nice touch and not strictly necessary. Introducing a change to handle build and output directory separately feels more important as it would greatly help in keeping the directory tree clean.

I'm not using LaTeX locally quite so often, as most exercises are done on Overleaf. But when I do, it's bigger things like a seminar paper or bachelor thesis and I would love to be able to use VimTex to its fullest for these.

Can I offer my help or do you think that the changes are small enough, that it would be faster for you to just write them down instead of me trying to learn vimlang and understanding the project structure?

lervag commented 1 year ago

Sorry for the long delay! I've been quite busy and have prioritized to resolve minor things in the meantime. I'll try and get back to look at this. I think it should not be too hard, but if you have the time and interest, then please feel free to try and suggest a PR!

lervag commented 1 year ago

I have done some more research now. I would be happy to get some feedback on my suggested change below. First, I did a test now on how latexmk behaves with regard to the files we are currently using in VimTeX:

Options	fls	synctex.gz	pdf	blg	bcf	aux	log
	r	r	r	r	r	r	r
a	r	r	r	a	r	a, r	a, r
a, e	r	r	r	a	a	a	a
o	o	o	o	o	o	o	o
o, a	o	o	o	a	o	a, o	a, o
o, a, e	o	o	o	a	a	a	a

The table indicates where files with the specified extensions are put after compilation with the various directory options:

r: root directory
a: $aux_dir
o: $out_dir
e: $emulate_aux

I believe a good approach is to change the current b:vimtex.get_aux_file(extension) into b:vimtex.compiler.get_generated_file(extension). This makes more sense, because the compiler knows where it puts the files. I would also want to remove the b:vimtex.out() method as it only calls the same function behind the scene.

The generic template compiler would work more or less the same as the current b:vimtex.get_aux_file(extension). I.e., it would look inside $VIMTEX_OUTPUT_DIRECTORY first, then inside the project root directory.
For the latexmk compiler, we would implement this by mapping the input extension to the specified correct path depending on which option is used (ref. the table above). We will align the option paths with the same as the latexmk options.
For the other compilers, I will check what makes most sense for each of them.

One possibility would be to keep the current b:vimtex.get_aux_file and just make it call the new compiler-specific function. But I'm not sure if there's a good reason for that.

lervag commented 1 year ago

So, essentially, I will still apply most of the changes I suggested previously. After I'm finished, users will have the following options for latexmk:

    let g:vimtex_compiler_latexmk = {
        \ 'aux_dir' : '',
        \ 'out_dir' : '',
        \ 'callback' : 1,
        \ 'continuous' : 1,
        \ 'executable' : 'latexmk',
        \ 'hooks' : [],
        \ 'options' : [
        \   '-verbose',
        \   '-emulate-aux-dir',
        \   '-file-line-error',
        \   '-synctex=1',
        \   '-interaction=nonstopmode',
        \ ],
        \}

The main changes would be:

Rename: build_dir to out_dir (align VimTeX options for latexmk with the latexmk options)
Added: aux_dir
Added default option: -emulate-aux-dir (this seems safe; not sure, though)

I guess it is still useful to have the environment variables to allow flexibility here, but I'm not sure. But if we use latexmk, it would suffice to use a local latexmkrc file, so perhaps the environment variables are not necessary?

tyalie commented 1 year ago

I've looked over the suggested changes and they seem to solve the issue quite well. The table you provided also matches my experience. I cannot rly give feedback about the implementation details of get_aux_file as I barely have an overview over the code, but I think your expertise can be trusted here without any doubts. ^^

Regarding env variable: How would vimtex even use these?

lervag commented 1 year ago

I've looked over the suggested changes and they seem to solve the issue quite well.

👍🏻

Regarding env variable: How would vimtex even use these?

It's actually documented under :help VIMTEX_OUTPUT_DIRECTORY. But I don't think you should worry about it, as you will probably not need it.

lervag commented 1 year ago

I've started to work seriously on this now. I believe the docs should be ready; please check by either pulling the branch from #2677 or by reading here.

lervag commented 1 year ago

I've pushed a first draft of this now. It would be very nice if you have the time to test it on your end. Let me know how it works for you.

Notice that VimTeX should respect your .latexmkrc file(s) here.

tyalie commented 1 year ago

Hi. I've right now a lot on my plate which will ease up over the next month. Would it be okay for you if I test this more extensively and look into the documentation in a month?

lervag commented 1 year ago

Yes, that's fine. Take your time. I'll not merge this until it's been tested, though.

lervag commented 1 year ago

Friendly nudge here; still no rush. :)

lervag commented 1 year ago

I'll close this issue - let's have the continued discussion in #2677. When that PR is merged you can of course feel free to open a new follow-up issue if there were more things that are not provided in the PR.

lervag / vimtex