Completion source for \usepackage{ based on tlmgr list --only-installed

[kiryph]

Would you consider adding a cached omnicompletion source based on tlmgr list --only-installed triggered when the user has entered \usepackage{ or \RequirePackage{?

[lervag]

Hmm. tlmgr is not available for me.

> locate tlmgr
/usr/share/texmf-dist/scripts/texlive/tlmgr.pl
/usr/share/texmf-dist/scripts/texlive/tlmgrgui.pl

I'm on Arch Linux. Do you by chance know how to get it to work?

[clason]

That's the one. In vanilla texlive, it's symlinked to $TL_ROOT/bin/$ARCH/tlmgr and hence available from $PATH. (In Arch, it might be hidden to force people to use texlive-local-manager.)

In my installation, the first two lines are

#!/usr/bin/env perl
# $Id: tlmgr.pl 41476 2016-06-18 00:45:25Z preining $

[lervag]

@clason I don't, actually. I'm assuming that the package maintainer updates packages once in a while and pushes them to the Arch repositories.

If I try to run the script, I get the following error:

> /usr/share/texmf-dist/scripts/texlive/tlmgrgui.pl
Can't locate TeXLive/TLUtils.pm in @INC (you may need to install the TeXLive::TLUtils module) (@INC contains: /usr/lib/perl5/site_perl /usr/share/perl5/site_perl /usr/lib/perl5/vendor_perl /usr/share/perl5/vendor_perl /usr/lib/perl5/core_perl /usr/share/perl5/core_perl .) at /usr/share/texmf-dist/scripts/texlive/tlmgrgui.pl line 41.
BEGIN failed--compilation aborted at /usr/share/texmf-dist/scripts/texlive/tlmgrgui.pl line 41.

[lervag]

texlive-local-manager is also not available...

[clason]

Hmm, that seems to be a quirk of the Arch packages; they indeed do not include any tlmgr script. (I always recommend to install vanilla texlive and add $TL_ROOT/bin/$ARCH to $PATH. Updating packages at will is very useful when you run into bugs, which tend to be fixed quickly for the larger packages. Of course, you also run into bugs earlier...)

Can you install https://aur.archlinux.org/packages/texlive-localmanager-git/ ?

[lervag]

You might be right about it being better to install vanilla texlive, but at the same time: I do not have any issues with my texlive installation. So it seems unnecessary to me, at least for my normal usage.

What is the difference between tlmgr and tllocalmgr?

[clason]

Of course, whatever works for you. (I personally find it very useful to update packages, especially more modern ones such as biblatex, pgfplots or fonts, which I make heavy use of.)

tllocalmgr seems to be an Arch specific script that I don't know anything about; presumably, it manages the texlive installation in a user-specific manner and doesn't require root access.

[clason]

Anyway, that is by the by -- the important thing is to be able to run the script even if you never need to.

I think there are some hard-coded relative include links in tlmgr that prevent it from working unless it is called from $TL_ROOT/bin/$ARCH. Can you try creating a symlink to the tlmgr.pl script in the same directory where the texlive binaries (pdflatex, kpsewhere etc.) are?

EDIT That won't work, either, since Arch has repackaged the binaries differently. You need to create the directory /usr/share/bin/$ARCH, link the perl script there, and call it with the full path (or link further to, e.g., /usr/local/bin).

[kiryph]

On Debian tlmgr is contained in texlive-base (https://packages.debian.org/sid/all/texlive-base/filelist) and on Ubuntu the situation is similar as expected (http://packages.ubuntu.com/yakkety/all/texlive-base/filelist). In both cases the main texlive installation is managed by the distribution package manager (apt). However, tlmgr is run in usermode which allows non-privileged user to install (updated) packages into their home directories:

USER MODE
       "tlmgr" provides a restricted way, called ``user mode'', to manage arbitrary texmf trees in the same way as the main installation.  For example, this allows people without write permissions on the
       installation location to update/install packages into a tree of their own.

       "tlmgr" is switched into user mode with the command line option "--usermode".  It does not switch automatically, nor is there any configuration file setting for it.  Thus, this option has to be explicitly
       given every time user mode is to be activated.

       This mode of "tlmgr" works on a user tree, by default the value of the "TEXMFHOME" variable.  This can be overridden with the command line option "--usertree".  In the following when we speak of the user
       tree we mean either "TEXMFHOME" or the one given on the command line.

       Not all actions are allowed in user mode; "tlmgr" will warn you and not carry out any problematic actions.  Currently not supported (and probably will never be) is the "platform" action.  The "gui" action
       is currently not supported, but may be in a future release.

       Some "tlmgr" actions don't need any write permissions and thus work the same in user mode and normal mode.  Currently these are: "check", "help", "list", "print-platform", "search", "show", "version".

       On the other hand, most of the actions dealing with package management do need write permissions, and thus behave differently in user mode, as described below: "install", "update", "remove", "option",
       "paper", "generate", "backup", "restore", "uninstall", "symlinks".

       Before using "tlmgr" in user mode, you have to set up the user tree with the "init-usertree" action.  This creates usertree"/web2c" and usertree"/tlpkg/tlpobj", and a minimal
       usertree"/tlpkg/texlive.tlpdb".  At that point, you can tell "tlmgr" to do the (supported) actions by adding the "--usermode" command line option.

       In user mode the file usertree"/tlpkg/texlive.tlpdb" contains only the packages that have been installed into the user tree using "tlmgr", plus additional options from the ``virtual'' package
       "00texlive.installation" (similar to the main installation's "texlive.tlpdb").

       All actions on packages in user mode can only be carried out on packages that are known as "relocatable".  This excludes all packages containing executables and a few other core packages.  Of the 2500 or
       so packages currently in TeX Live the vast majority are relocatable and can be installed into a user tree.

       Description of changes of actions in user mode:

   User mode install
       In user mode, the "install" action checks that the package and all dependencies are all either relocated or already installed in the system installation.  If this is the case, it unpacks all containers to
       be installed into the user tree (to repeat, that's either "TEXMFHOME" or the value of "--usertree") and add the respective packages to the user tree's "texlive.tlpdb" (creating it if need be).

       Currently installing a collection in user mode installs all dependent packages, but in contrast to normal mode, does not install dependent collections.  For example, in normal mode "tlmgr install
       collection-context" would install "collection-basic" and other collections, while in user mode, only the packages mentioned in "collection-context" are installed.

       If a package shipping map files is installed in user mode, a backup of the user's "updmap.cfg" in "USERTREE/web2c/" is made, and then this file regenerated from the list of installed packages.

   User mode backup, restore, remove, update
       In user mode, these actions check that all packages to be acted on are installed in the user tree before proceeding; otherwise, they behave just as in normal mode.

   User mode generate, option, paper
       In user mode, these actions operate only on the user tree's configuration files and/or "texlive.tlpdb".  creates configuration files in user tree

Unfortunately, on OSX with a single user following command does not do what I would expect:

❯ tlmgr list --only-installed --usermode
cannot setup TLPDB in /Users/kiryph/Library/texmf at /Library/TeX/texbin/tlmgr line 5753.

I would expect that I still see the list of installed packages. The files are readable.

I have also checked http://tex.stackexchange.com/questions/55437/how-do-i-update-my-tex-distribution.

Miktex does also not contain tlmgr but has mpm --installed (mpm = miktex package manager).

Ok, this is more complicated than I have expected. In particular, that the --usermode does not work is not good.

If it would work out of the box for Windows Miktex/Texlive, OSX MacTex, Debian and Ubuntu, I would expect that this covers the majority of users. Ok, lervag on Arch should also have a working solution :-).

[kiryph]

The error in usermode cannot setup TLPDB can be fixed with

$ tlmgr init-usertree

However, tlmgr list --only-installed --usermode returns an empty list. I guess this simply means it would only list packages installed with usermode.

[clason]

I think --usermode only works if used consistently (i.e., you have to use that already at install time). On the other hand, list should work without root privileges even without --usermode. (I can't test this on vanilla texlive since all my installations are user writable; on a Ubuntu 16.04 machine I have access to, tlmgr list doesn't work with any combination of --usermode and sudo...)

So I agree, any feature that relies on the specific installation method would not be a good fit for a vim plugin...

[lervag]

@kiryph Yes, I'd like to have a working solution. It becomes too difficult to implement if I can't work with the command itself. And preferably, I would like not to install texlive manually just for this feature. It's a bit surprising that this is not generally available. However, perhaps there is a different way of listing installed packages?

[clason]

If it's cached, how about a brute force solution using system tools:

sh -c "find / -name '*.sty' | grep -oE '/([^/]+)/[^/]+\.sty$' | cut -f2 -d'/' | sort | uniq" 2>/dev/null

(from http://tex.stackexchange.com/a/341399/16855)

And similar for .cls for \documentclass{.

[lervag]

Yes, but I'd prefer a smarter solution that does not search through the entire system. It should be possible to use kpsewhich or similar...?

[kiryph]

There is even a separate question for documentclasses pointing in the same direction http://tex.stackexchange.com/q/115028/how-do-i-list-the-available-classes-in-my-installation

• documentclasses

  $ grep "\.cls" `kpsewhich --var-value TEXMFDIST`/ls-R

• usepackage

  $ grep "\.sty" `kpsewhich --var-value TEXMFDIST`/ls-R

I guess this is even better. The list of names returned by tlmgr does not reflect all possible package names. And they are faster than the perl script tlmgr. Caching would still be sensible.

[lervag]

Ok, now we're getting somewhere. Thanks, @kiryph, for looking into this! This should work on Linux and OSX, and so I can work with this and implement a simple solution when I get the time.

It remains to find a good way to make this work on Windows, but we could do that later (I don't really care about windows, to be honest :p).

[clason]

The problem is that there can be user-installed packages in TEXMFHOME which this approach would miss. But the ls-R approach is nice and fast, so that might be an acceptable trade-off!

[kiryph]

@clason This should be fixable by

grep "\.sty" `kpsewhich --var-value TEXMFDIST`/ls-R `kpsewhich --var-value TEXMFHOME`/ls-R

grep accepts several file names.

[lervag]

I'm curious: The syntax with /ls-R is unknown to me. Could you explain it?

[lervag]

Sorry, I'm stupid. My fault! It's of course just a file...

[clason]

ls-R is exactly what it sounds like: A file (created by mtexlsr) that contains all file names in the specific tree.

It's not generated by default for $TEXMFHOME, but asking the user to do that themselves by calling (not tested)

cd kpsewhich -var-value TEXMFHOME && ls -LAR > ./ >ls-R

 texhash `kpsewhich -var-value TEXMFHOME`

would not be too much to ask if they want autocomplete of user-installed packages or classes.

(There's also TEXMF which contains all paths known to kpathsea, but that might be overkill.)

[lervag]

However, I notice there are even more locations where packages may exist. On my system, I have both TEXMFDIST, TEXMFHOME, as well as a location /usr/local/share/texmf. The latter one I don't quite know.

From here I see I can find the list of directories with kpsewhich --var-value TEXINPUTS. I could use this list of paths, get all ls-R files that are available under these paths, then use the combined entries as the candidate source. What do you think?

[lervag]

E.g.:

files = vimtex#util#kpsewhich('TEXINPUTS', '--var-value')
files = split(files,...)
files = map(files, 'v:val . ''/ls-R''')
files = filter(files, 'filereadable(v:val)')

candidates = []
for f in files
  call add(candidates, readfile(f))
endfor
...

Consider the above pseudo code, but I think it is pretty close to something that should work.

[clason]

TEXMF might be easier to parse. On my system:

> kpsewhich --var-value TEXINPUTS
.:{/home/clason/.texlive2016/texmf-config,/home/clason/.texlive2016/texmf-var,/home/clason/.texmf,!!/usr/local/texlive/2016/texmf-config,!!/usr/local/texlive/2016/texmf-var,!!/usr/local/texlive/texmf-local,!!/usr/local/texlive/2016/texmf-dist}/tex/{kpsewhich,generic,}//

> kpsewhich --var-value TEXMF
{/home/clason/.texlive2016/texmf-config,/home/clason/.texlive2016/texmf-var,/home/clason/.texmf,!!/usr/local/texlive/2016/texmf-config,!!/usr/local/texlive/2016/texmf-var,!!/usr/local/texlive/texmf-local,!!/usr/local/texlive/2016/texmf-dist}

Grepping as early as possible is important, though, since ls-R contains all files (and subdirectories) in the corresponding texmf-tree...

Other than that, sounds good!

[lervag]

Great, and thanks for suggesting TEXMF, it is clearly easier to parse! :)

[kiryph]

Another solution to get the list of ls-R files is $ kpsewhich -all ls-R which returns a neat list

/usr/local/texlive/2016/texmf-config/ls-R
/usr/local/texlive/2016/texmf-var/ls-R
/usr/local/share/texmf/ls-R
/usr/local/texlive/texmf-local/ls-R
/usr/local/texlive/2016/texmf-dist/ls-R

No parsing necessary.

[lervag]

Haha, now you're making it too easy :)

[clason]

Then I don't mind remarking that this misses the ls-R in TEXMFHOME :)

[lervag]

Ah, then it really was too easy. But parsing a comma separated list is not really very difficult.

[clason]

Or take the list from kpsewhich and just append TEXMFHOME/ls-R (if it exists).

(By default, TEXMFHOME is searched directly since it's assumed to be small. It's possible to make kpathsea look for ls-R in TEXMFHOME as well, but that requires editing config files which I wouldn't recommend.)

[kiryph]

For example, following works:

❯ kpsewhich -all MinionPro.sty
/Users/kiryph/Library/texmf/tex/latex/MinionPro/MinionPro.sty
/usr/local/texlive/texmf-local/tex/latex/MinionPro/MinionPro.sty

@clason thanks for your details about this.

[clason]

@kiryph Yes. Just in case it wasn't clear (otherwise, sorry for the noise): The first line is found by (roughly) find /Users/kiryph/Library/texmf -name MinionPro.sty, the second by grep MinionPro.sty /usr/local/texlive/texmf-local/ls-R. This is controlled by the !! in front of the corresponding TEXMF tree in texmf.cnf (if it is there, ls-R is used, otherwise not).

[lervag]

I'll try to work on this later this evening. I opened a branch: feature/complete-usepackage.

[kiryph]

The documentation of kpathsea contains following bits of information (http://tug.org/texinfohtml/kpathsea.html#ls_002dR):

Kpathsea looks for ls-R files along the TEXMFDBS path, so that should presumably match the list of hierarchies.

Looking into the variable TEXMFDBS with kpsewhich --var-value TEXMFDBS misses TEXMFHOME. In other words, kpsewhich does not look for ls-R files in TEXMFHOME by intention. As @clason said, one would have to modify a texmf.cnf file. In my case that would be vim +105 /usr/local/texlive/2016/texmf-dist/web2c/texmf.cnf

and change

TEXMFDBS = {!!$TEXMFSYSCONFIG,!!$TEXMFSYSVAR,!!$TEXMFLOCAL,!!$TEXMFDIST}

to

TEXMFDBS = {!!$TEXMFSYSCONFIG,!!$TEXMFSYSVAR,!!$TEXMFLOCAL,!!$TEXMFDIST,!!$TEXMFHOME}

So I guess a manual addition of TEXMFHOME/ls-R would document that vimtex takes this ls-R file additionally into account in contrast to kpsewhich.

Anyhow, these are details which most user do not care about as long as vimtex finds also packages installed into home directories.

[clason]

@kiryph Exactly (with the addition that the ls-R file is not created automatically but has to be generated by hand). This is fine if the documentation explicitly states this; something like

Only system packages (i.e., those installed in TEXMFDIST, TEXMFLOCAL) are completed by default. If you also want completion for packages installed in your user tree (TEXMFHOME), you need to first run

 mktexlsr `kpsewhich -var-value TEXMFHOME`

This needs to be done every time a package that should be completed is added (or removed) in TEXMFHOME.

should do.

(Since mktexlsr with argument doesn't care what directory it hashes, in principle one could add completion for arbitrary directories via a vim variable containing directories with ls-R files in them. I don't know how useful this would be, though, if LaTeX itself won't find the files...)

[kiryph]

I think, I would also add to the documentation the possibility to extend TEXMFDBS by TEXMFHOME and leave it to the user how he wants to handle this. Running tlmgr --update or mktexlsr without any parameter can offer some convenience.

I think your comment in parenthesis could be an addition for a future feature addition together with running mktexlsr asyncronously from vim. This would generate and update the lsr files also for a user defined list of directories. Consider project-specific texmf directories defined in the environment variable $TEXINPUTS (possibly defined in latexmkrc file). see this question and the two answers http://tex.stackexchange.com/q/50697/how-can-i-keep-a-clean-document-folder-with-custom-cls-sty-bst-files-in-a-s.

Following variables and function could be added

let g:vimtex_additional_lsr_directories += vimtex#get_texinputs()
let g:vimtex_auto_update_mktexlsr = 1

vimtex#get_current_texinputs() gets $TEXINPUTS as environment variable and from latexmkrc.

However, right now I would keep the new addition for vimtex small and leave this for the future. User will see if they need this quite often (out-of-date ls-r files and missing packages due to recent local additions).

[clason]

I would be VERY wary of suggesting to tinker with the texlive infrastructure to someone who doesn't understand the possible side effects. (For example, if you extend TEXMFDBS, TEXMFHOME is never searched directly, so you need to run mktexlsr every time you add a file for it to be visible to tex and friends, in contrast to the default setting. I can imagine this leading to a lot of confusion, especially if people assume the default behavior, e.g., on tex.SE.)

Although mktexlsr is pretty fast on a small tree (and a fast disk), automatically running it every time on completion (or even file load) seems quite wasteful.

But I agree, something for another time :)

[kiryph]

I see your reason not to suggest adding it to TEXMFDBS. I wasn't aware of the fact that if a ls-r file exists, no regular search is performed. I was actually thinking the other way, that it would be more user friendly. But now I agree with you that it can irritate user and asking questions on tex.SE without this piece of information would lead to confusion. You have convinced me.

I didn't have in mind running mktexlsr always for completion. Only when filetype tex is detected and later on when compilation is triggered and of course nowadays only in the background (vim8).

Anyhow, this is not important right now. The basic functionality for the main packages and provide one way to add packages in the home directory.

[lervag]

I'm curious if I've understood correctly. Did we land on using the output of kpsewhich -all ls-R and append $TEXMFHOME/ls-R if it exists?

As we also noted, the $TEXMFHOME/ls-R typically does not exist. Perhaps we could also just do the globbing manually for this directory?

In any case, I've pushed something that seems to work pretty well, at least for me. I would be happy if you could test. Let me know if you need pointers for how to pull the branch.

[lervag]

I'd prefer if we could continue the discussion at #711. I'm therefore closing this issue.