[Discussion] Do not load all available modules

[naquad]

Situation: LGI and IUP installed. IUP loads GTK2, LGI loads GTK3. BOOM! SEGFAULT! GTK2 and GTK3 can not be loaded into the same address space. CDIUPLUACAIRO - can not be loaded w/o IUPLUA, CDLUAIUP, IMLUA, ... in other case - segfault.

My point is: one can not load all libs and hope they'll somehow work. Plus modules can have side effects.

I'm thinking about the following solution:

1. drag in LPeg and Lua parse in LPeg (see http://lua-users.org/wiki/LpegRecipes) or some another Lua parser written in Lua (http://lua-users.org/wiki/LuaGrammar) for example piece of LuaMinify (https://github.com/stravant/LuaMinify) to find all require() calls
2. fork process trying to require module and get all its vars + tracing globals

Even if it'll crash we won't care because VIM itself won't crash. Forking will require us to add some sort of caching for stuff to avoid dozens of forks per line typed (better take more RAM then hang for seconds). There'll also have to be a way to clear cache (in theory its just one command).

It sounds bad, ugly and hard to do, but I don't see much options here. Even if we'll drop part 1 (thats with parser) stuff will be much more stable with moving require()s to separate process.

What do you think about this?

P.S. I hope to get to implementation of this somewhere on weekend.

[xolox]

Hey Daniel,

Thanks for the feedback! This is a complex topic so here's a heads up for the wall of text below ;-)

I'm assuming we are talking about omni completion here? The feature is disabled by default precisely because of the can of worms you just opened: potential side effects like rm -Rf / ;-), loading of incompatible shared objects, etc.

A lot of reasoning/thought went into how the Lua file type plug-in for Vim works right now. Literally years :-). Of course that's no to say that it is perfect (far from it). Anyway, I like KISS as a design principle and the things you propose are far from KISS :-P. I understand why of course; I struggled with the same issues you bring up here for quite a while before I published the file type plug-in with its current feature set.

My thoughts on the issue:

• Static analysis is never going to get us anywhere because:

  1. Lua is too dynamic in nature.
  2. The Lua file type plug-in cannot (or: should not want to) do static analysis of compiled modules (in fact it's near impossible) even though it is really useful to get completion for these modules.

  Do you disagree with this assessment? Of course it is true that for completion, an approximation of the truth is already very useful, I just worry that Lua is so dynamic in nature that we'll never come up with a decent list of completion candidates. Also we would have to simulate all of Lua's module search mechanisms → far from ideal.

• If static analysis is out, all that's left is runtime analysis (the current approach). This simply cannot be made 'safe' (idempotent, no side effects, robust), amongst other reasons because of the variety of systems on which I want the file type plug-in to work, e.g. Windows vs UNIX (portability excludes effective sandboxing).

I can see problems with moving the runtime analysis to a separate process:

• In issue #4 it was reported that importing the awful module will hang the process indefinitely on a futex() system call. I've seen similar issues before, which is one of the reasons why omni completion has to be explicitly enabled :-). So per your suggestion and this issue, the handling of the external process would have to involve timeout handling and forceful killing of the subprocess after the timeout to avoid these kinds of issues. That's far from trivial to implement in a portable fashion... :-(
• There will be a overhead in performance (can be compensated) and complexity (more important) when we add subprocess handling in a portable fashion so that it works on e.g. Windows and UNIX alike.

Despite all of the above, I am of course interested in improving the plug-in and the dynamic completion system.

One simple change that would improve the usability of the plug-in given how it currently works would be to implement a black list of modules that should never be imported for dynamic completion. This would allow to exclude e.g. awful while preserving the usefulness of the current completion system. Of course this doesn't (yet) deal with modules that 'conflict' with each other (like you reported above).

So no clear answer from me, just more questions :-). Sorry about that. I'm interested to read your thoughts on this.

[xolox]

I will go ahead and close this issue now because it's been more than a year and I just implemented the previously mentioned black list (see pull request #21 for details).