Nvim-cmp's keyword_pattern and Vimtex's Omnifunc

[micangl]

Description

Vimtex support in nvim-cmp is an issue which has been extensively discussed. These are just some of the related issues:

• https://github.com/lervag/vimtex/issues/2215
• https://github.com/hrsh7th/nvim-cmp/issues/833
• https://github.com/hrsh7th/nvim-cmp/issues/1170
• https://github.com/hrsh7th/nvim-cmp/issues/652
• https://github.com/hrsh7th/cmp-omni/issues/1
• https://github.com/hrsh7th/nvim-cmp/issues/258

The problem explained in those issues dealt mainly with the Vimtex-provided neocomplete pattern (https://github.com/lervag/vimtex/blob/master/autoload/vimtex/re.vim#L35), why it didn't work, and why cite-completion wasn't triggered automatically (i.e. the completion menu appeared) as soon as a \cite{ statement was typed.

I may have found some workarounds and/or hints towards a possible solution. I'll explain them in order:

1. Cite-completion isn't triggered automatically as soon as a \cite{ statement is issued. Differently from other engines, with nvim-cmp the user first has to type a character to have the menu pop up and show bilbiographic references. This is obviously annoying, as the user may not always remember, at the top of his head, exactly which reference to insert (or the key name). A workaround which I don't think was ever discussed is to use a trigger_character in nvim-cmp's configuration, as such:

   { name = "omni", trigger_characters = { "{" } },

2. Correctness of the Vimtex-provided Neocomplete pattern. When the discussions in the previously linked issues were held, the validity of said pattern was questioned and, if I recall correctly, noone ever managed to make it work with nvim-cmp. The pattern, though, is indeed correct. This can be tested by opening the following file in Neovim and sourcing (:so):

   vim.g.regex =
            [=[\%(]=]
         .. [=[\v\\%(]=]
         .. [=[%(\a*cite|Cite)\a*\*?%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|%(\a*cites|Cites)%(\s*\([^)]*\)){0,2}]=]
         ..    [=[%(%(\s*\[[^]]*\]){0,2}\s*\{[^}]*\})*]=]
         ..    [=[%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|bibentry\s*\{[^}]*]=]
         .. [=[|%(text|block)cquote\*?%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|%(for|hy)\w*cquote\*?\{[^}]*}%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|defbibentryset\{[^}]*}\{[^}]*]=]
         .. [=[|\a*ref%(\s*\{[^}]*|range\s*\{[^,}]*%(}\{)?)]=]
         .. [=[|hyperref\s*\[[^]]*]=]
         .. [=[|includegraphics\*?%(\s*\[[^]]*\]){0,2}\s*\{[^}]*]=]
         .. [=[|%(include%(only)?|input|subfile)\s*\{[^}]*]=]
         .. [=[|([cpdr]?(gls|Gls|GLS)|acr|Acr|ACR)\a*\s*\{[^}]*]=]
         .. [=[|(ac|Ac|AC)\s*\{[^}]*]=]
         .. [=[|includepdf%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|includestandalone%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|%(usepackage|RequirePackage|PassOptionsToPackage)%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|documentclass%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|begin%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|end%(\s*\[[^]]*\])?\s*\{[^}]*]=]
         .. [=[|\a*]=]
         .. [=[)]=]
   local regex_1 = vim.regex(vim.g.regex)
   local match_str = [=[noise \cite{]=]
   
   vim.print(regex_1:match_str(match_str))

3. And now the hardest issue: Making that pattern work with nvim-cmp. I've only partially achieved this goal, and I write this issue with the hope that someone more expert than me will be able to help.

   First thing, nvim-cmp modifies the keyword_pattern provided by the user: https://github.com/hrsh7th/nvim-cmp/blob/5dce1b778b85c717f6614e3f4da45e9f19f54435/lua/cmp/context.lua#L77

       return pattern.offset([[\%(]] .. keyword_pattern .. [[\)\m$]], self.cursor_before_line) or self.cursor.col

   This is necessary to ensure that only the last-typed text will be matched, as discussed in https://github.com/hrsh7th/nvim-cmp/issues/1273. The problem is that such a modification assumes to be in magic mode, whilst the keyword pattern we would be using set \v very-magic mode. This can be fixed by either adding a \m at the end of the pattern we pass to nvim-cmp, as such

    .. [=[|\a*]=]
    .. [=[)]=]
    .. [=[\m]=]

   or by correcting nvim-cmp's handling. I still haven't made a PR, but it should be like this:

       return pattern.offset([[\%(]] .. keyword_pattern .. [[\m\)$]], self.cursor_before_line) or self.cursor.col

   After making such an addition, nvim-cmp still doesn't match. It took me multiple hours of debugging but, by modifying cmp/source.lua

   +  local offset = ctx:get_offset(self:get_keyword_pattern()) + 1
   -  local offset = ctx:get_offset(self:get_keyword_pattern())

   nvim-cmp start finally to match correctly and show the completion menu. The issue I've found is that the completion menu, with such a modification, doesn't appear after typing a \cite{ statement. I've tried to troubleshoot this, but I'm not a programmer and the code in that nvim-cmp's function is just too complicated for me.

If someone who has the expertise would like to help, I'd be happy to contribute where I can. I honestly hope that we will, finally, be able to solve this stubborn issue.

Finally, sorry for bearing with me 'till the end of this message.

VimtexInfo

System info:
  OS: Linux 6.4.12-arch1-1
  Vim version: NVIM v0.9.1
  Has clientserver: true
  Servername: /run/user/1000/nvim.896581.0

VimTeX project: a
  base: a.tex
  root: /home/mike
  tex: /home/mike/a.tex
  main parser: current file verified
  document class: article
  compiler: latexmk
    engine: -pdf
    options:
      -verbose
      -file-line-error
      -synctex=1
      -interaction=nonstopmode
    callback: 1
    continuous: 1
    executable: latexmk
  viewer: Zathura
    xwin id: 0
  qf method: LaTeX logfile

[micangl]

@hrsh7th I think some of the points I discussed in this issue could interest you. Feedback from both ends will be beneficial.

[lervag]

@micangl Sorry for not responding sooner! I've been busy with a lot of things, including playing a computer game - which I seldom do. As I don't have much spare time, this means the time for working with VimTeX has been at a minimum for a few weeks.

I'll try to step it up and I'll begin reading your issue post in full now.

[lervag]

I may have found some workarounds and/or hints towards a possible solution. I'll explain them in order:

Thanks; I appreciate your effort and I recognize that you've worked quite hard on this!

Vimtex support in nvim-cmp is an issue …

My initial thought is that I'm using nvim-cmp and I feel like I have very good VimTeX support. The following is my personal configuration (the relevant snippets):

• The start of my lazy.nvim specification of nvim-cmp: https://github.com/lervag/dotnvim/blob/b1dc12bc2d4ecf8f5b5e5c467ccbdd88f5303986/lua/lervag/init/packages.lua#L569
  • My specification is quite lengthy, but most of it is adjustment of formatting and similar and is not so relevant.
• Where I specify my sources for .tex buffers: https://github.com/lervag/dotnvim/blob/b1dc12bc2d4ecf8f5b5e5c467ccbdd88f5303986/lua/lervag/init/packages.lua#L738-L745

For me, this "just works"™. I don't specify any keyword_pattern, and it does not really seem necessary. So, could we first establish that there really is an issue?

Still, I'll give some initial thoughts to your investigations:

1. Cite-completion isn't triggered automatically as soon as a \cite{ statement is issued.

Ah, yes, that's a valid point. Completion triggers for me after typing two letters (due to my settings).

A workaround which I don't think was ever discussed is to use a trigger_character in nvim-cmp's configuration

Cool, thanks! That seems useful!

2. Correctness of the Vimtex-provided Neocomplete pattern. When the discussions in the previously linked issues were held, the validity of said pattern was questioned and, if I recall correctly, noone ever managed to make it work with nvim-cmp. The pattern, though, is indeed correct.

Working with VimTeX has made me quite adept at writing regexes. ;)

3. And now the hardest issue: Making that pattern work with nvim-cmp.

I would ask: do we need that? Could we first establish the current real pain points?

Let's say we recognize some real pain points: I believe a possible better way forward would be to write a full source for nvim-cmp. I believe we could use the existing completion function and possibly the existing regexes for that. I remember that I did attempt this once, but I didnt immediately figure things out and I realized that things worked well enough that I just stopped.

The problem is that such a modification assumes to be in magic mode, whilst the keyword pattern we would be using set \v very-magic mode. This can be fixed by either …

or by adjusting the regex (make a new one) that is magic mode. It shouldn't be too hard.

After making such an addition, nvim-cmp still doesn't match. It took me multiple hours of debugging …

<3

nvim-cmp start finally to match correctly and show the completion menu. …

It would not be surprising to me if such a proposed changed implies other changes that may not be consistent - but I wouldn't know as I've not studied this code.

[micangl]

The main issue, currently, is that one has to add the trigger_characters to make the completion menu appear automatically with \cite{. I've started testing things out by modifying the cmp-omni source.

It would also be very nice if said source would configure formatting for the completion menu, like the solution discussed in issue #2215:

local cmp = require("cmp")
cmp.setup.buffer({
  formatting = {
    format = function(entry, vim_item)
      vim_item.menu = ({
        omni = (vim.inspect(vim_item.menu):gsub('%"', "")),
        luasnip = "[LuaSnip]",
        buffer = "[Buffer]",
      })[entry.source.name]
      return vim_ite
    end,
  },
})

I've tried the pattern with vanilla nvim-cmp and it didn't work, as expected. But it does match without the leading backslashes (which is in accord with that offset issue I've talked about). But as you pointed out, it's just not worth the headache to try making the pattern work with cmp.

The custom source would be nice, though, to set the trigger_characters and formatting. What do you think?

[lervag]

The main issue, currently, is that one has to add the trigger_characters to make the completion menu appear automatically with \cite{.

Ok, but then it also seems as adding the addition config you suggest more or less fixes this, right?

I've started testing things out by modifying the cmp-omni source.

It would also be very nice if said source would configure formatting for the completion menu, like the solution discussed in issue #2215:

Yes, one of the reasons I have a lot of config for nvim-cmp is precisely to improve the formatting of e.g. VimTeX completion with nvim-cmp.

The custom source would be nice, though, to set the trigger_characters and formatting. What do you think?

Well, yes, I still agree. And I would be happy to help figure out these things. Though, I don't have much time. If someone else (you?) would take some initiative, then I could be a "passenger" and would likely be less of a "blocker".

It would be fine by me to consider a PR for this, although I may end up to suggest to add it as an extension for nvim-cmp instead of a part of VimTeX itself.

[micangl]

Yes, what I meant was that we should just create a nvim-cmp extension (plugin). I'm working on it.

[lervag]

Great! Let me know if/when you need input/advice/help!

[micangl]

Well, the hope of creating a custom source for cmp has already faltered. Unfortunately, there's no way for a source to set a format function in a non-destructive way (i.e. without overwriting the user's implementation).

[lervag]

Unfortunately, there's no way for a source to set a format function in a non-destructive way (i.e. without overwriting the user's implementation).

Ah, no, you are probably right. However, I think the conclusion may still be wrong. That is, I don't think we want a source to have a custom formatter, instead, we want it to provide the metadata in a way that is compatible with the nvim-cmp standard format.

That is, the lsp completer looks like this on my end:

I believe we can have a VimTeX completer for nvim-cmp that translates information for the completion items from the omnicomplete function into a data model that fits with nvim-cmp.

[micangl]

The custom format function which I planned to implement simply removed double quotes from the completion items. Adding it, gets you this result (no quotes surrounding the book titles:

I believe we can have a VimTeX completer for nvim-cmp that translates information for the completion items from the omnicomplete function into a data model that fits with nvim-cmp.

What would you like to achieve, exactly?

[lervag]

On my end, without changing the formatter option, I get a completion menu that looks like this:

I've made some adjustments which basically just changes Text to a symbol (but it does more for other completion types):

What would you like to achieve, exactly?

I'm not very familiar with the spec for nvim-cmp, but I believe it could be possible to leverage it to get a better experience. E.g., we could move the full info into a preview window.

The complete function should call a callback function with one or more completion items according to this spec:

https://github.com/hrsh7th/nvim-cmp/blob/5dce1b778b85c717f6614e3f4da45e9f19f54435/lua/cmp/types/lsp.lua#L264-L284

There may be some better docs for this that I have not found yet, but I think this may point to a direction of what I want to achieve.

[micangl]

On my end, without changing the formatter option, I get a completion menu that looks like this:

The citations section of the menu looks strange; mine, even without the format function, doesn't present such an issue. Might it be a problem with the .bib file?

I've added an option to display the full Info in the documentation window:

I think that the result is quite lame, especially considering the huge amount a .bib file may provide:

@article{knuth:1984,
  title={Literate Programming},
  author={Donald E. Knuth},
  journal={The Computer Journal},
  volume={27},
  number={2},
  pages={97--111},
  year={1984},
  publisher={Oxford University Press}
}

I've noticed that the info_fmt variable can only accept a small set of parameters; maybe we should add support for the other Bibtex entries? Having all of the bibliographic data in the documentation window would really be game-changing, I think.

I've made some adjustments which basically just changes Text to a symbol (but it does more for other completion types):

Should I add an option to set this too?

Another game-changing option is the ability to fuzzy match, not only against the citation keywords, but also against the data contained in the menu field (like author, publication date, and title):

P.S. I still haven't uploaded the repository on Github, but I will soon (I need to make the codebase presentable).

[micangl]

Another big problem has arised: cmp-omni is unlicensed. I could open an issue, but @hrsh7th, the author, seems to have been inactive with regard to nvim-cmp for some weeks. The only contact information he provides is his twitter account, but I don't use it.

[lervag]

Could you explain why it is an issue that cmp-omni is unlicensed?

[micangl]

Because the cmp source I was working on for Vimtex is based on cmp-omni.

[lervag]

Ok, I see. There are a lot of other sources that could be suitable as a reference, though. I bet some of them have a license..?

[micangl]

Not really, since Vimtex still uses omnicompletion. Cmp-omni is the source for omnifuncs in nvim-cmp. I've commented under the last of @hrsh7th tweets but he hasn't answered.

[lervag]

IMHO, the only thing about omnicompletion that is relevant here is how to fetch the completion candidates. Basically, this is all documented under :help omnifunc.

My idea for how to create a custom source would be to:

1. Get a skeleton source up and going that can provide some pre-defined completion candidates for some of the relevant context (e.g. \cite{).
2. Use vim.fn["vimtex#complete#omnifunc"] to get the candidates (this is an omnifunc that is created to follow the spec explained in the Vim docs).

[micangl]

Yeah, we could do that. For a skeleton source we could choose anything, such as cmp-latex-symbols or cmp-buffer. For what concerns the rest, the documentation is pretty good, so it shouldn't be too difficult to set up the completion.

I'll still wait a little though, in case I hear from hrsh7th.

[micangl]

He updated the license! Here's my repository: https://github.com/micangl/cmp-vimtex. Wait to update the documentation, though: it isn't ready yet for public use.

Now it's ready (also added to nvim-cmp's sources page).

[micangl]

I've noticed that the info_fmt variable can only accept a small set of parameters; maybe we should add support for the other Bibtex entries? Having all of the bibliographic data in the documentation window would really be game-changing, I think.

What do you think about this?

[lervag]

He updated the license! Here's my repository: https://github.com/micangl/cmp-vimtex. ~Wait to update the documentation, though: it isn't ready yet for public use.~

Now it's ready (also added to nvim-cmp's sources page).

Cool thanks! You're fast! (Sorry about not being quicker in my replies. Busy life!)

I'll look at your repo. Feel free to announce it on reddit (r/neovim) - it may help you get more users on board, which will help in discovering issues and thus improve things if needed.

I've noticed that the info_fmt variable can only accept a small set of parameters; maybe we should add support for the other Bibtex entries? Having all of the bibliographic data in the documentation window would really be game-changing, I think.

What do you think about this?

Ah, sorry, I overlooked that comment. I'll read it and respond asap.

[lervag]

I've added an option to display the full Info in the documentation window: …

Cool, that's a very nice proof of concept! I believe you can have the preview formatted by using Markdown (e.g. *Title* or similar).

I think that the result is quite lame, especially considering the huge amount a .bib file may provide: …

Yes: The VimTeX completion functions were never developed with a preview window in mind (the concepts barely existed at the time and I've never personally had much use of it).

I've noticed that the info_fmt variable can only accept a small set of parameters; maybe we should add support for the other Bibtex entries? Having all of the bibliographic data in the documentation window would really be game-changing, I think.

Perhaps. Notice that process of populating the completion candidate list with info can be somewhat slow. Fetching more data may lead to a significant slow down.

One possibility would be to fetch the wanted extra data on demand. I think that could be fast enough and possibly more practical. I do this for the context menu.

Pseudo vimscript code that will get a dictionary with the bibtex info for a specified "KEY":

" Get all bibtex entries into a list. The pushd is to ensure we are at the root
" directory when we are locating bib files.
call vimtex#paths#pushd(b:vimtex.root)
let l:entries = []
for l:file in vimtex#bib#files()
  let l:entries += vimtex#parser#bib(l:file, {'backend': 'vim'})
endfor
call vimtex#paths#popd()

" Get the desired entry
call filter(l:entries, {_, x -> x.key ==# "KEY"})
let l:entry = get(l:entries, 0, {})

It should not be too hard to do something like the above in Lua. Notice, though, that for very large projects or projects with very large bibfiles, this could be slow. It could be of interest to implement a caching mechanism for the vimtex#parser#bib function, which should make things much faster.

I've made some adjustments which basically just changes Text to a symbol (but it does more for other completion types):

Should I add an option to set this too?

Yes, or at least, I would prefer not to have "Text" as the indicated type. A symbol for TeX or VimTeX or something like that would be more correct IMHO.

Another game-changing option is the ability to fuzzy match, not only against the citation keywords, but also against the data contained in the menu field (like author, publication date, and title): …

I personally don't like fuzzy matching that much. With VimTeX and the omnicompletion, you can actually match against the menu field with regexes, but I think that has become an esoteric thing.

Still, I know a lot of people like fuzzy matching, so I agree it is nice to have!

[micangl]

I'll look at your repo. Feel free to announce it on reddit (r/neovim) - it may help you get more users on board, which will help in discovering issues and thus improve things if needed.

That would surely help! But I don't use it; if you wish, feel absolutely free to make a post though, I'd appreciate it!

Cool, that's a very nice proof of concept! I believe you can have the preview formatted by using Markdown (e.g. Title or similar).

I hadn't though of this! It will definitely increase readability. It's a shame that Github's markdown flavor doesn't support colors, as having the specifiers highlighted would be even better.

Parsing the bibtex files with that snippet is indeed slow, so a cache will be absolutely essential. Sorry if I'm dragging this on, it's my first plugin (and open source project, for the matter). I hope I'm not abusing your willingness to help too much. Here are my questions:

• Does the paths variable contains the path to bibtex files which should, ideally, be available in all .tex files? Or does it contain files specific only to the current one? This is because of the fact that, if user were to open multiple latex files (part of separate projects), it would be imperative, for reasons of efficiency, to have a common cache for the bibliographic information common to all projects.

• I have though of two ways to handle the parsing:

  1. When a .tex file is opened, user your code snippet to parse the necessary bibtex files and cache them. This would have to be done asynchronously, since large files could slow down the startup (I think so, at least). Drawback: if the user were to modify one of the bibtex files while those had already been cached, said modifications wouldn't be available.
  2. The bibliographic information is fetched only when the user selects an item in the completion menu (hence causing the documentation window to open and creating the need for entry-specific information). This is absolutely inefficient unless retrieving only one item from the bibtex files is way faster than parsing everything on the opening of the buffer (which I don't think is the case).
  • Would there be a way to, in an intelligent way, re-parse only the modified files, i.e. of knowing which ones were actually modified?

[micangl]

Okay, I've made some progress, in the meantime. Currently, I'm parsing the Bibtex files on startup (with an autocmd for BufWinEnter). Unfortunately, even after using Neovim's integrated vim.schedule (async execution), the parsing of very large files still slows down the editor (and significantly).

I've explored the possibility of creating a separate thread, but I can't pass or receive lua tables, which makes the functionality useless for this application.

Do you have any idea on how I could proceed? (At this point, I'm starting to think that I'll have to do the parsing in lua, which doesn't seem that fun...).

[micangl]

@lervag After two weeks, I finally managed to implement the asynchronous parser. I've had to port it to lua, though. The last thing I need to do, before pushing the changes, is updating the licenses. Here's the problem: I'm using code from hrsh7th's cmp-omni and cmp-buffer, and the parser is just a rewrite in lua of your vimscript one.

Everything, luckily, is licensed under the MIT License. So, how should I specify the copyright (i.e. what should I write in license file)?

• The base of the plugin is cmp-omni, which I modified.
• The asynchronous handling code (a single file) has been taken from cmp-buffer.
• The bibtex parser is a lua rewrite of your vimscript one.

[lervag]

I'm very sorry for not having had the time to reply. Life has been very busy lately. I really appreciate that you are working on this, and I would love to be able to help more. I will try to make more time to follow up more closely in the coming week. I still don't have the time to fully read your previous comment and give a detailed reply, but I'll make a quick remark to your latest update.

The last thing I need to do, before pushing the changes, is updating the licenses. Here's the problem: I'm using code from hrsh7th's cmp-omni and cmp-buffer, and the parser is just a rewrite in lua of your vimscript one.

Everything, luckily, is licensed under the MIT License. So, how should I specify the copyright (i.e. what should I write in license file)?

My suggestion would be to publish your work under the MIT license as well and add an acknowledgement in your README file where you refer to the source you've based your work on. Similar to what you've written here.

By the way: would it make sense to adopt the Lua code for the parser into VimTeX? Then it could be made the default choice (if it is fast enough) for neovim users.

[lervag]

I'll look at your repo. Feel free to announce it on reddit (r/neovim) - it may help you get more users on board, which will help in discovering issues and thus improve things if needed.

That would surely help! But I don't use it; if you wish, feel absolutely free to make a post though, I'd appreciate it!

When things are ready, I will test it personally, then write a section in the docs and post on Reddit. Let's keep the current issue open until I do, if you don't mind.

Parsing the bibtex files with that snippet is indeed slow, so a cache will be absolutely essential.

Is this still true with your new Lua parser?

Sorry if I'm dragging this on, it's my first plugin (and open source project, for the matter). I hope I'm not abusing your willingness to help too much.

No, not at all! I'm sorry for not having made time to follow up from my end!

Does the paths variable contains the path to bibtex files which should, ideally, be available in all .tex files? Or does it contain files specific only to the current one?

The current one.

Would there be a way to, in an intelligent way, re-parse only the modified files, i.e. of knowing which ones were actually modified?

That's a hard question. I don't have the code in my "brain buffer". I can investigate and figure out a good answer if this is still relevant?

Okay, I've made some progress, in the meantime. Currently, I'm parsing the Bibtex files on startup (with an autocmd for BufWinEnter). Unfortunately, even after using Neovim's integrated vim.schedule (async execution), the parsing of very large files still slows down the editor (and significantly).

Yes; it is hard to make this stuff fully asynchronuous I think without calling out to Lua. From Lua, I think it may be possible to start a separate thread with coroutines.

I've explored the possibility of creating a separate thread, but I can't pass or receive lua tables, which makes the functionality useless for this application.

Do you have any idea on how I could proceed? (At this point, I'm starting to think that I'll have to do the parsing in lua, which doesn't seem that fun...).

Ah, yes; you're clearly way ahead of me already. And if I understood correctly, you already did this.

You are still working with this repo, right? I didn't see the parser code, is it in a separate repo?

Would you say it is a good time for me to test things?

[micangl]

Is this still true with your new Lua parser?

For large files, like the one Vimtex uses for tests, it still may not be instantaneous. So, the parsing is started asynchronously on a BufWinEnter event, and the results are simply stored in a lua table. Since tables, in lua, are just assocative arrays implemented through a hash map, they are already sufficient as a simple 'cache'.

That's a hard question. I don't have the code in my "brain buffer". I can investigate and figure out a good answer if this is still relevant?

I investigated a little and there doesn't seem to be a way, with Neovim's api, to check the last-modified timestamp of a file. But it should be easy to just call a terminal program (like stat).

You are still working with this repo, right? I didn't see the parser code, is it in a separate repo?

Currently, I was keeping the changes on my local repo. I'll update it today. Also, the configuring options have changed as you'll be able to see in the README.

About the asynchronous execution of the parsing, it was so messy to figure out... Luckily, hrsth7th, for his cmp-buffer, had already created a lua implementation of javascript's start_timer, which allows me to run the parsing without the slightest hint of lagging.

Also, there are more features that, in the future, I'd like to implement:

• Providing advanced configuration options to the user, allowing granular control over things such as the maximum size of parsed files, when a reparse should be triggered.
• Sorting bibliographic fields in the documentation window.
• Shortcuts to search for currently selected item in the completion menu (valid for \cite{ completion) on search engines, databases (Google Scholar, IEEExplore, etc.). Also, the possibility to open the currently selected entry in the bibtex file to modify it on the fly.

This last feature, I think, could be very useful to check things on the fly when writing a paper.

Thanks a lot for your support!

[lervag]

Is this still true with your new Lua parser?

For large files, like the one Vimtex uses for tests, it still may not be instantaneous.

Cool, but I think this also holds for the other parsing methods. I have a test for this (test-completion-bibtex-speed) that would be cool to use for benchmarking your parser.

So, the parsing is started asynchronously on a BufWinEnter event, and the results are simply stored in a lua table. Since tables, in lua, are just assocative arrays implemented through a hash map, they are already sufficient as a simple 'cache'.

Nice. Notice, when I talk about caching, what I mean is to store the result of parsing a single file. If the file was not changed, then instead of parsing it, we can just load the stored result.

That's a hard question. I don't have the code in my "brain buffer". I can investigate and figure out a good answer if this is still relevant?

I investigated a little and there doesn't seem to be a way, with Neovim's api, to check the last-modified timestamp of a file. But it should be easy to just call a terminal program (like stat).

Oh, yes, there are apis for this. Since Neovim's Lua APIs allow direct access to the builtin functions, you can just use them. E.g. :help getftime(). See e.g. here:

https://github.com/lervag/vimtex/blob/cbb20643b7bfe721902dac5760bf0d4889fb5f7e/autoload/vimtex/cache.vim#L224-L227

You are still working with this repo, right? I didn't see the parser code, is it in a separate repo?

Currently, I was keeping the changes on my local repo. I'll update it today. Also, the configuring options have changed as you'll be able to see in the README.

About the asynchronous execution of the parsing, it was so messy to figure out... Luckily, hrsth7th, for his cmp-buffer, had already created a lua implementation of javascript's start_timer, which allows me to run the parsing without the slightest hint of lagging.

Great, I look forward to having the time to test this! Hopefully within this week.

• Shortcuts to search for currently selected item in the completion menu (valid for \cite{ completion) on search engines, databases (Google Scholar, IEEExplore, etc.). Also, the possibility to open the currently selected entry in the bibtex file to modify it on the fly.

Nice. Notice that the context menu already provides some of this (see :help vimtex-context-citation). But if you find a way to make things even more user friendly, I'm all for it!

One last thing: You didn't respond to one of my questions:

By the way: would it make sense to adopt the Lua code for the parser into VimTeX? Then it could be made the default choice (if it is fast enough) for neovim users.

[micangl]

Nice. Notice, when I talk about caching, what I mean is to store the result of parsing a single file. If the file was not changed, then instead of parsing it, we can just load the stored result.

If I understand correctly the point you are raising, the parsing is not done after every BufWinEnter event, but only one time. The results then get stored in the lua table and, when another event takes place, the autocmd's callback first checks if the file was already parsed (if this is the case, then, the file doesn't get parsed again, as it would be useless and inefficient).

Oh, yes, there are apis for this. Since Neovim's Lua APIs allow direct access to the builtin functions, you can just use them. E.g. :help getftime(). See e.g. here:

Cool, this criteria will definitely be added for determining if a file should be reparsed.

Nice. Notice that the context menu already provides some of this (see :help vimtex-context-citation). But if you find a way to make things even more user friendly, I'm all for it!

The use-case I thought of is the following: you're inserting a citation and you can't remember which is the correct one, as there may be many with slightly different titles/keys/etc. So, you select one in the completion menu and, with a shortcut, open the bibtex file(for modification), or the Doi, or the pdf (exactly as the context-menu does), or you can search for it on a specific research database or through a standard search engine (maybe even opening Zotero could be implemented). The key difference, here, is that the databases/engines should also be specifiable by the user (a feature which could also be implemented in Vimtex), and that this operations would be done on-the-fly, while you're choosing the citation and not after you've chosen it.

This, I think, could turn out to be very useful for people who write papers and have to deal with a lot of literature.

One last thing: You didn't respond to one of my questions:

By the way: would it make sense to adopt the Lua code for the parser into VimTeX? Then it could be made the default choice (if it is fast enough) for neovim users.

Sorry, my mistake. I don't really see why it couldn't be. The delay for opening the context-menu up, when dealing with very big bibtex files, is consistent. Lua will probably speed it up, as testing will might indicate. The "main" parser function has been adapted to be called asynchronously, but it could easily be reverted to just being a simple translation of the vimscript code into lua.

[lervag]

Nice. Notice that the context menu already provides some of this (see :help vimtex-context-citation). But if you find a way to make things even more user friendly, I'm all for it!

The use-case I thought of is the following: you're inserting a citation and you can't remember which is the correct one, as there may be many with slightly different titles/keys/etc. So, you select one in the completion menu and, with a shortcut, open the bibtex file(for modification), or the Doi, or the pdf (exactly as the context-menu does), or you can search for it on a specific research database or through a standard search engine (maybe even opening Zotero could be implemented). The key difference, here, is that the databases/engines should also be specifiable by the user (a feature which could also be implemented in Vimtex), and that this operations would be done on-the-fly, while you're choosing the citation and not after you've chosen it.

This, I think, could turn out to be very useful for people who write papers and have to deal with a lot of literature.

Ok. This is not a feature I personally want or need as I can't really say I have this problem very often. But people are different and I guess someone would like this very much. And I think it does work out nicely as a feature from a VimTeX compatible completion plugin!

By the way: would it make sense to adopt the Lua code for the parser into VimTeX? Then it could be made the default choice (if it is fast enough) for neovim users.

I don't really see why it couldn't be. The delay for opening the context-menu up, when dealing with very big bibtex files, is consistent. Lua will probably speed it up, as testing will might indicate. The "main" parser function has been adapted to be called asynchronously, but it could easily be reverted to just being a simple translation of the vimscript code into lua.

I think the asynchronous part could just as well be moved one step up, right?

[lervag]

I've started to test things now. My first two comments:

• To enable the plugin, I need to run .setup(). I think that should be unnecessary unless I actually want to change any options. And in the meantime, this should be made clear in the README, i.e. that running .setup is important.
• I don't get any info on the citation completion, nor any preview window. Not sure why, perhaps I have some different options enabled?

[micangl]

Ok. This is not a feature I personally want or need as I can't really say I have this problem very often. But people are different and I guess someone would like this very much. And I think it does work out nicely as a feature from a VimTeX compatible completion plugin!

A nice thing to do, in the future, could be to add the online databases search functionality to Vimtex, so that it could be used in the context menu, allowing cmp-vimtex to call this functions directly from Vimtex, so as not to have redundant code. But this is just a thought as, currently, I don't have a lot of time available.

I think the asynchronous part could just as well be moved one step up, right?

The problem with Lua, in such a context, is that it's not really asynchronous. Lua provides coroutines, which give the semblance of asynchronousness by allowing software to schedule execution of a function on the main event-loop. If the scheduled functions are small there is no harm in this, but, if they take some time to complete (like for parsing), the main event-loop is occupied with this task, effectively freezing Neovim. The problem is solved by parsing a small chunk of the file, and then scheduling execution of the same function to continue. But, if you want, I can provide you with a literal translation of your integrated parser; the function I need to change is only the main one.

[lervag]

I've mostly converted your lua parser to a synchronous version now, see here: https://github.com/lervag/vimtex/blob/feat/bib-parser-lua/lua/vimtex/bibparser.lua

I'll test it some more and see how it compares to the other methods in terms of speed when I have time to continue this.

[micangl]

I don't get any info on the citation completion, nor any preview window. Not sure why, perhaps I have some different options enabled?

I've tested it in a VM with the following minimal configuration:

• init.vim

  filetype plugin on
  let g:vimtex_view_method = 'zathura'
  let g:vimtex_quickfix_enabled = 0
  
  call plug#begin()
  Plug 'hrsh7th/nvim-cmp'
  Plug 'micangl/cmp-vimtex'
  
  Plug 'L3MON4D3/LuaSnip'
  Plug 'saadparwaiz1/cmp_luasnip'
  
  Plug 'lervag/vimtex'
  call plug#end()
  
  set completeopt=menu,preview,menuone,noselect
  lua <<EOF
  local cmp = require'cmp'
  
  cmp.setup({
    snippet = {
      expand = function(args)
        require('luasnip').lsp_expand(args.body)
      end,
    },
    mapping = cmp.mapping.preset.insert({
      ['<C-b>'] = cmp.mapping.scroll_docs(-4),
      ['<C-f>'] = cmp.mapping.scroll_docs(4),
      ["<C-n>"] = cmp.mapping.select_next_item { behavior = cmp.SelectBehavior.Insert },
      ['<C-p>'] = cmp.mapping.select_prev_item { behavior = cmp.SelectBehavior.Insert },
      ['<C-Space>'] = cmp.mapping.complete(),
      ['<C-e>'] = cmp.mapping.abort(),
      ['<C-y>'] = cmp.mapping.confirm({ select = true }), -- Accept currently selected item. Set `select` to `false` to only confirm explicitly selected items.
    }),
    sources = cmp.config.sources({
    })
  })
  
  require('cmp_vimtex').setup()
  EOF

• ftplugin/tex.vim

  let maplocalleader = ","

• after/ftplugin/tex.lua

  local cmp = require'cmp'
  
  cmp.setup.buffer {
    sources = {
        {
          name = 'vimtex',
        },
    },
  }

To enable the plugin, I need to run .setup(). I think that should be unnecessary unless I actually want to change any options. And in the meantime, this should be made clear in the README, i.e. that running .setup is important.

I agree.

M.setup = function(options)
    require('cmp').register_source('vimtex', require('cmp_vimtex.source').new(options))
end

As you can see, the setup function registers the source with nvim-cmp, whilst creating an "instance" of the source with the user-provided config. Earlier, the source was registered in the after/plugin directory of the plugin (without user configuration). To address the issue you raised, I could check if the source has already been registered (if not register it with the default config), inside a file in said directory.

EDIT: I've added this feature.

[lervag]

Thanks for the quick and detailed response! I'll check things again later when I get the time.

I've continued work on the lua parser in VimTeX now and it should work. Performance wise it seems very good. On the speed test I have running, it seems to be faster than the previous fastest parser (which is, fascinatingly, to run bibtex and then source the output). I think I will end up using this as the default backend for neovim users in not too long. :)

[lervag]

The speed test is also running in the ci pipeline, so you can see the results here:

https://github.com/lervag/vimtex/actions/runs/6606130297/job/17942017427?pr=2816#step:4:102

[ThSGM]

@micangl @lervag I just want to add that your work on the recent cmp plugin has been a huge quality of life upgrade for me, and has made a huge difference to some of the issues I've had in using autocompletion with vimtex.

Part of my issue is not being able to articulate all the little issues I've had in the past (things like autocompletion not triggering on '{' and also the lack of fuzzy finding on bibliography data (this has been asked multiple times by people on Reddit and otherwise).

Please continue the development of the cmp plugin and also @lervag I hope you also work to integrate cmp-vimtex.

Thank you!!

[micangl]

@ThSGM Thanks a lot! If you feel like there are some features missing, don't hesitate to open an issue.

I've also checked your nvim-cmp config, lervag, and I can't understand why you're missing information from the completion menu. Maybe something unexpected is happening in the formatter function, even thought it doesn't appear to be the case, in my opinion.

[lervag]

@ThSGM

I just want to add that your work on the recent cmp plugin has been a huge quality of life upgrade for me, and has made a huge difference to some of the issues I've had in using autocompletion with vimtex.

Credit goes to @micangl here, I've just helped on conceptual stuff really. But I'm glad to hear feedback from other users; I can see how this gives a noticable improvement to many users.

Please continue the development of the cmp plugin and also @lervag I hope you also work to integrate cmp-vimtex.

I will integrate it in the sense that I will refer to it in the documentation. Perhaps @ejmastnak may be interested in updating his tutorial as well, at least when cmp-vimtex has reached some more maturity?

@micangl

I've also checked your nvim-cmp config, lervag, and I can't understand why you're missing information from the completion menu. Maybe something unexpected is happening in the formatter function, even thought it doesn't appear to be the case, in my opinion.

Thanks; I'll look into this.

Now, I want to bring your attention to #2816 - a PR where I add support for the Lua backend to VimTeX. I've adjusted your implementation somewhat and included it, and as I reported earlier it seems to work quite well. I would be very glad to get some feedback on it before I merge. In particular, I've removed the license header from the source file as I instead use a global license file and a simple header for all the source files.

Before I merge I also want to check if I can further improve the parser somewhat. I'm not quite happy with the code, it doesn't feel quite right.

[lervag]

Regarding my problem with getting the preview window: it seems to be some strange issue with plugin loading. I use lazy.nvim, and I loaded cmp-vimtex as a dependency for lazy.nvim. I tested with nvim main.tex, and the result was that the autocommand on BufWinEnter was not triggered.

I've opened an issue for that: https://github.com/micangl/cmp-vimtex/issues/1.

[lervag]

Thanks to some useful comments by @clason, I've managed to further improve the performance of the Lua bib parser. It is now significantly faster than the bibtex parser (which was usually the fastest one). On my end it is often near twice as fast. 🥳

[lervag]

The main improvements come from using the Lua pattern matching instead of regexes. Regular expressions are powerful, but Lua patterns are powerful enough in most cases and much faster.

[lervag]

If you want to check on your side, go to the VimTeX folder, then to test/test-completion-bibtex-speed and run make. I get these results now:

make -C test-completion-bibtex-speed
Backend: BIBTEX
Time elapsed (1st run):  0.73314
Time elapsed (2nd run):  0.02925
Time elapsed (3rd run):  0.03039
Number of candidates: 1610

Backend: VIM
Time elapsed (1st run):  2.95427
Time elapsed (2nd run):  0.03498
Time elapsed (3rd run):  0.03220
Number of candidates: 1612

Backend: LUA
Time elapsed (1st run):  0.46517
Time elapsed (2nd run):  0.03001
Time elapsed (3rd run):  0.03017
Number of candidates: 1612

(The 2nd and 3rd runs are just for checking that the caching of completion items work as expected.)

[micangl]

Thanks to some useful comments by @clason, I've managed to further improve the performance of the Lua bib parser. It is now significantly faster than the bibtex parser (which was usually the fastest one). On my end it is often near twice as fast. 🥳

Keeps getting better and better! Thanks @clason!

[lervag]

I realize it is much easier to follow up on the remainings tasks from this thread by opening a new issue for that, so I just did: #2818. I'll therefore close this issue now.