Completion of citation keys from both bib files and Zotero

[jalvesaq]

I implemented omnicompletion of citation keys from bib files for Rnoweb, and from both bib files and Zotero for RMarkdown. The implementation is in the brach bibcompl. In addition of citation key completion, you can open the file attached to a reference in Zotero by doing CTRL-] in normal mode with the cursor over the citation key. It also works if the reference in the bib file has the field "file".

The new features are documented at the section "Completion of non R code" of Nvim-R documentation.

The code works for me, but it would be great if other people reported issues before I merge the branch.

I used code from many sources, but mostly from citation.vim (many thanks to @rafaqz for having written it!).

[rafaqz]

Cool! Out of interest, why you didn't just point people to citation.vim, what is the need for duplicating all that work? Personally it means I'm running both now!

[jalvesaq]

I already have nvim-completion-manager + ncm-R installed, and didn't want to use Unite. My preference was for omnicompletion. I recently developed a pandoc filter to format references in the ABNT style (the only one used in Brasil) because it seems that we can't get parenthetical citations different from textual citations with the CSL specification as required by ABNT. So, I had already learned how to deal with PyBTeX. I have also found a comment of someone explaining how to use Emacs for both completing citations and formatting the references directly from Zotero in such a way that it was never necessary to export the references to bib files. The biggest advantage was to keep all the references in one place with no need of synching zotero data and bib files (Judd, 2016). I wanted the same functionality in Vim because I'm tentatively moving from LaTeX to Markdown.

Perhaps we could find a way of having two Python classes (PyBTeX and Zotero) that could be used as base classes by other Python modules that would do regular omnicompletion (as Nvim-R does), and be used as sources for Unite, nvim-completion-manager, etc...

[rafaqz]

Ok that makes sense. I definitely went strongly with the unite angle.

I was thinking about a shared lib too. Some of that code was taken from "libzotero" that was actually just a subfolder of gnotero, but never actually formalised into a library. It could be a pip package. But that could be more work than just forgetting about it, it looks like you pretty heavily altered things. Maybe we can revisit next time zotero dramatically changes their database schema.

Thanks in the readme wouldn't go astray for that much code contrib!

[rafaqz]

Also your no bibtex method sounds interesting, bibtex is a bit of a pain. I haven't set it up yet but this could be useful in a zotoero/markdown/pandoc workflow: https://github.com/egh/zotxt

[jalvesaq]

I've tried zotxt, but it didn't work. Anyway, it depends on Zotero being running.

it looks like you pretty heavily altered things

I adapted the code to Nvim-R needs. The idea is that slow things such as parsing zotero.sqlite or parsing a big bib file should happen only once, at the startup of the Python application. From then on, Nvim-R requests information to the Python application which is running asynchronously as a job. The zotero.sqlite and bib files are parsed again only if their modification times increase.

[rafaqz]

Citation.vim actually does that too... But anyway, glad you found the code useful.

[jalvesaq]

I didn't merge the branch yet. I may try to use citation.vim and let you know if I have any difficulty.

[jalvesaq]

@rafaqz Trying to run citation.vim, I have put the following in my init.vim:

let g:citation_vim_mode="zotero"
let g:citation_vim_cache_path = '/home/aquino/.cache/citation_vim'
let g:citation_vim_key_format="{author}{date}{title}"
nmap <leader>u [unite]
nnoremap [unite] <nop>
nnoremap <silent>,c        :<C-u>Unite -buffer-name=citation-start-insert -default-action=append      citation/key<cr>
nnoremap <silent>[unite]co :<C-u>Unite -input=<C-R><C-W> -default-action=start -force-immediately citation/file<cr>

But when I press ,c I get:

Citation.vim error:
Traceback (most recent call last):
  File "/home/aquino/.cache/vim-plug/citation.vim/python/citation_vim/citation.py", line 24, in connect
    return Builder(ContextLoader().context).build_source()
  File "/home/aquino/.cache/vim-plug/citation.vim/autoload/unite/sources/../../../python/citation_vim/builder.py", line 30, in build_source
    return self.get_sub_source()
  File "/home/aquino/.cache/vim-plug/citation.vim/autoload/unite/sources/../../../python/citation_vim/builder.py", line 34, in get_sub_source
    for item in self.get_items():
  File "/home/aquino/.cache/vim-plug/citation.vim/autoload/unite/sources/../../../python/citation_vim/builder.py", line 89, in get_items
    items = parser.load()
  File "/home/aquino/.cache/vim-plug/citation.vim/autoload/unite/sources/../../../python/citation_vim/zotero/parser.py", line 34, in load
    zot_data = zotero.load()
  File "/home/aquino/.cache/vim-plug/citation.vim/autoload/unite/sources/../../../python/citation_vim/zotero/data.py", line 146, in load
    self.get_item_detail()
  File "/home/aquino/.cache/vim-plug/citation.vim/autoload/unite/sources/../../../python/citation_vim/zotero/data.py", line 171, in get_item_detail
    self.get_authors()
  File "/home/aquino/.cache/vim-plug/citation.vim/autoload/unite/sources/../../../python/citation_vim/zotero/data.py", line 250, in get_authors
    self.cur.execute(self.author_query_v4)
sqlite3.OperationalError: no such table: creatorData
[unite.vim] function unite#helper#call_unite[33]..unite#start[1]..unite#start#standard[50]..unite#candidates#_recache[74]..<SNR>121_recache_candidates_loop[42]
..<SNR>121_get_source_candidates[40]..338, line 3
[unite.vim] Vim(return):E712: Argument of map() must be a List or Dictionary
[unite.vim] Error occurred in gather_candidates!
[unite.vim] Source name is citation/key

[rafaqz]

Wow that's an sqlite error. Are you on Zotero 5? 4 is no longer supported as there were some major schema changes between versions. But that's been the only change in database tables I am aware of. creatorData is the main author table so not having it is pretty weird.

[jalvesaq]

Yes, it's version 5. I downloaded Zotero for the first time a few days ago and imported three bib files to develop the omni completion.

[rafaqz]

Ok sorry that's actually the v4 version! I just realised I hadn't actually bumped the default version! that's awful. Most people have it in their config from the changover period so I haven't had any issues.

let g:citation_vim_zotero_version=5

[rafaqz]

I just bumped that on master so you should need that setting if you pull again now. But I'm going to overhaul all the docs and cruft code from that changeover period, citation.vim could do with a little more attention.

I was also thinking that it would be very easy to fit a nvim completion manager fontend, nearly all the work is in python and it already returns an array with all the keys for completion, it should be a very minor edit.

[jalvesaq]

Thanks! I inserted some citation keys. Now, how should I call pandoc to convert the document to pdf?

[rafaqz]

I don't do any of that in citation.vim, I try to follow unix philosphy so citation.vim just inserts text citations (hah and a lot of other things) but it doesn't actually do anything actively in vim, you have to add your own key commands.

The vim_pandoc plugin, command line pandoc command or your scriipts etc deal with the rest.

I personally use an oldschool make file:

https://gist.github.com/rafaqz/efdb079829bc02975400afc0f6e5efef

And then I have this script to watch my md with entr

#!/bin/bash
# Watch files for changes. Build, view and word-count 

in_ext=${1:-md}
out_ext=${2:-pdf}
for in_file in *.$in_ext; do
  out_file=${in_file%%.*}.$out_ext
done 

makeit="make EXT=$in_ext $out_ext"
eval $makeit
rifle $out_file
echo $in_file | entr sh -c "reset && eval $makeit && sleep 1 && wordcount $out_file"

But vim_pandoc probably is the sanest option, although it does too many other things as well. There could be a vim plugin that just compiled pandoc on save... but I personally tweak my config so much the makefile is a better option.

[jalvesaq]

I work in a social sciences department, and I need something easier to be able to convince a few students to switch from LibreOffice and RStudio to Neovim and Nvim-R. I'm doing the following:

• Run zotcite continuously. On startup, it creates a ZoteroEntries object with all the information from zotero.squlite. Then, Nvim-R asks for citation keys completion (omni completion) and for the path of attachments associated with the citation keys (CTRL] over a citation key). It also sets what collections are used by each Rmd file.

  • To build either a PDF or an HTML, the zotref filter runs ZoteroEntries just once. It gets all citation keys from the document (stdin), requests the references to the ZoteroEntries object, and adds them to the YAML header of a dummy markdown document. Finally, it extracts the references from the dummy document in the json format and adds them to the original document and pass it to stdout. Because the the same library is used to get citation keys and to get the references, all citations are found. The output is processed by pandoc-citeproc.

[rafaqz]

That sounds like a good workflow, especially the lack of bibtex export. Mine is admittedly for the hardcore users only. What are zotcite and zotref?

It sounds like the compilation side you describe could be a separate plugin that would also work with any citation method, however they got into a document.

Edit: ok I understand why you prefer one library for both ends of the process. Better-bibtex fills that standardising role using citation.vim/pandoc separately.

[rafaqz]

The ZoteroEntries object is basically like the cache I'm using, which is a list of Items. It writes to disk but mostly its actually hitting ram, and just retrieving the list of Item objects. It also contains the zotero id for each item, which is the best unique id if you don't have better-bibtex installed.

[jalvesaq]

zotcite and zotref are python scripts: https://github.com/jalvesaq/Nvim-R/blob/bibcompl/R/zotcite https://github.com/jalvesaq/Nvim-R/blob/bibcompl/R/zotref

The environment variables used by them are set by Nvim-R, and the use of zotref is documented at https://github.com/jalvesaq/Nvim-R/blob/bibcompl/doc/Nvim-R.txt#L1855

It sounds like the compilation side you describe could be a separate plugin that would also work with any citation method, however they got into a document.

Yes, it could be the same plugin or a different one, as long the citation keys match.

[jalvesaq]

I didn't install better-bibtex because I found it strange that it has 20MB of what seems to be Japanese data.

[rafaqz]

Really? hah I never really looked. The whole ecosystem is pretty messy, I guess that's why we are having this conversation at all.

I had to write a stop-gap for better bibtex that mimics the formatting during the period when there wasn't a zotero 5 version, but was very happy when they caught up. Currently using better-bibtex is the best method for guaranteed key matches, as otherwise you can have duplictates. You also kind of need a solution that works inside zotoero to lock the key even when you make changes to its fields. I'm not sure how to get around that.

[jalvesaq]

You also kind of need a solution that works inside zotoero to lock the key even when you make changes to its fields. I'm not sure how to get around that.

Does Zotero change the itemIds of references when the user makes changes in the fields? I changed a few fields and their itemIds remained the same.

[rafaqz]

No they stay the same too. But you would have to maintain a link between a human readable key you use in the document and the zotero key, and then updating that key might be an issue.

Unless of course you just use the zotero id as your key, which is what citation.vim will do given no other options. But its not a practical solution for me as you cant read what the key is!

Maybe appending it to a human readable key is an option? You could then hide it with vim conceal as its pretty ugly to have all through your document, not sure why I never thought of that before...

[rafaqz]

I just pushed a change that lets you do this:

let g:citation_vim_key_format="{author}{date}{title}##{zotero_key}##"

and use the zotero key however you like.

[jalvesaq]

You could then hide it with vim conceal

Great idea!

[jalvesaq]

Vim documentation for syn-region:

The decision to start a region is only based on a matching start pattern. There is no check for a matching end pattern.

If the user-defined citation key is the first part of the whole citation key, the start pattern would have to be "@" which I think is too simple to avoid highlighting wrong regions. It seems safer to have the Zotero key first, like this:

A @zotero_key#citation_key# and another @zotero_key#citation_key#.

which could be highlighted with:

set conceallevel=2
syntax match citationKey /\S*/ contained
syntax region citationRegion matchgroup=Comment start=/@\w*#/ end=/#/ contains=citationKey concealends keepend
hi def link citationKey Identifier

[rafaqz]

let g:citation_vim_key_format="{zotero_key}#{author}{date}{title}#"

note there are also [@bracketed] and @unbracketed forms of the keys.

[jalvesaq]

note there are also [@bracketed] and @unbracketed forms of the keys.

I will look for a solution for this later today, and I will add a note to Nvim-R documentation about citation.vim as an alternative way of inserting citation keys.

Perhaps I should release the zotref filter and the omnicompletion function using it as a separate plugin to not restrict its use to those writing RMarkdown code.

[rafaqz]

I would probably use it and recommend it if it meant no more bibtex

[jalvesaq]

I would probably use it and recommend it if it meant no more bibtex

Existing bib files could be imported into Zotero.

If the syntax highlighting of citation keys is done by vim-pandoc-syntax, this should work:

set conceallevel=2
syntax match citationKey /\S*/ contained
syntax region citationRegion matchgroup=Comment start=/@\w*#/ end=/#/ contains=citationKey containedin=pandocPCite concealends keepend
hi def link citationKey Identifier

[jalvesaq]

I moved the Zotero related code to https://github.com/jalvesaq/zotcite and we can continue this discussion at https://github.com/jalvesaq/zotcite/issues/1