Trying to upgrade vis lexers to scintillua master but it doesn't seem to be valid Lua anymore (or …???)

[mcepl]

While trying to update lexers in vis to https://github.com/orbitalquark/scintillua/releases/tag/scintillua_6.1 I have it on a strange issue, where I don’t understand. It seems that all lexers have something similar to this change:

-- local lexer = require('lexer')
-- local lex = lexer.new('ansi_c')
++ local lexer = lexer
++ local lex = lexer.new(...)

I really don’t understand the new code. Is local lexer = lexer even valid Lua? And what does lexer.new(...) actually mean (varargs are suppposed to be used in definition of a function not when calling it, right)?

Is there some Textpad magic involved which is not available for us poor Lua users?

[orbitalquark]

Yes, local lexer = lexer is a Lua idiom for making a global value a local one. Accessing locals is much more performant than globals.

lexer.new(...) means to call lexer.new() with any arguments passed to the current chunk (i.e. lexer) being called: https://github.com/orbitalquark/scintillua/blob/2181945f83beef745044ad6ab05f41395485ea3a/lexers/lexer.lua#L1616

[mcepl]

I have completely failed in resyncing vis with Scintillua in the first run, so I hoped to do just lexers first and lexers/lexer.lua afterwards, but apparently it couldn’t be done so, and I have to really dive into debugging what we did wrong. Oh well.

[orbitalquark]

Ah, I see. You should try the other way around: update lexer.lua first and then the lexers. The old lexers are now legacy, but should still be backwards-compatible.

[mcepl]

Thank you, I will try that.

[mcepl]

OK, complete breakage: when I start vis with this branch https://github.com/mcepl/vis/tree/scintillua-6.1 (and no ~/.config/vis/visrc.lua) I get this lovely message on the startup:

/usr/share/vis/lexer.lua:1616: no file '/usr/bin/lua/text.lua'
no file '/usr/bin/lua/text/init.lua/text.lua'
no file '/home/matej/.config/vis/text.lua'
no file '/home/matej/.config/vis/text/init.lua/text.lua'
no file '/etc/vis/text.lua'
no file '/etc/vis/text/init.lua/text.lua'
no file '/usr/share/vis/text.lua'
no file '/usr/share/vis/text/init.lua/text.lua'
no file '/usr/share/lua/5.4/text.lua'
no file '/usr/share/lua/5.4/text/init.lua/text.lua'
no file '/usr/lib64/lua/5.4/text.lua'
no file '/usr/lib64/lua/5.4/text/init.lua/text.lua'
no file '/text.lua'

The problem is that we had to modify lexers/lexer.lua to search in the right places. These lexer files are either in /usr/share/vis/lexers/ or in ~/.config/vis/lexers/ (in this order). Moreover, something seems to break even file type detection … this is a C (aka ansi_c) file I was trying to open.

Cc: @ninewise, @qiu-x, @jpe90, @Nomarian, @deepcube, @martanne

[mcepl]

https://gist.github.com/mcepl/cab5ab8b31fa6d9f74b0d47352e05026 is a diff between /usr/share/vis/lexers/lexer.lua (working with vis) and ~/repos/tmp/scintillua/lexers/lexer.lua (not working from 2181945 here).

[orbitalquark]

When using Scintillua as a standalone Lua library, Scintillua looks in package.path by default. If you're okay with modifying lexer.lua, then changing this line should be enough: https://github.com/orbitalquark/scintillua/blob/2181945f83beef745044ad6ab05f41395485ea3a/lexers/lexer.lua#L1542. (Set scintillua.lexers to whatever path(s) your lexers are at. Just use directory paths separated by ;, no ?.lua postfix or whatever.)

[mcepl]

When using Scintillua as a standalone Lua library, Scintillua looks in package.path by default. If you're okay with modifying lexer.lua, then changing this line should be enough:

https://github.com/orbitalquark/scintillua/blob/2181945f83beef745044ad6ab05f41395485ea3a/lexers/lexer.lua#L1542 . (Set scintillua.lexers to whatever path(s) your lexers are at. Just use directory paths separated by ;, no ?.lua postfix or whatever.)

You mean something like:

M.property = setmetatable({['scintillua.lexers'] = package.path:gsub('/%?%.lua', '/lexers')}, {

Hmm, that removed the error (the value seems sensible … package.path = /usr/bin/lua/lexers;/usr/bin/lua/?/init.lua;/home/matej/.config/vis/lexers;/home/matej/.config/vis/?/init.lua;/etc/vis/lexers;/etc/vis/?/init.lua;/usr/share/vis/lexers;…), but highlighting still didn’t happen.

[mcepl]

If I didn’t change anything in your files, and just add to the ~/.config/vis/visrc.lua:

package.path = package.path .. "/usr/share/vis/lexers/?.lua;/home/matej/.config/vis/lexers/?.lua"

I get an error message:

/usr/share/vis/lexers/json.lua:6: attempt to index a nil value (global 'lpeg')

[orbitalquark]

LPeg also needs to be on your path.

[mcepl]

LPeg also needs to be on your path.

Well, it is there, at least I think:

stitny~$ rpm -ql lua54-lpeg
/usr/lib64/lua/5.4
/usr/lib64/lua/5.4/lpeg.so
/usr/share/lua/5.4
/usr/share/lua/5.4/re.lua
stitny~$ 

And besides, on the very same computer vis with the previous version of lexers/* files works.

[orbitalquark]

I think I see the problem. Sorry about that. This should fix it: https://github.com/orbitalquark/scintillua/commit/61900949800f9acc1238b45f4683250bd5c60755

[mcepl]

Better, but unfortunately now I hit our own code: https://git.sr.ht/~mcepl/vis/tree/devel/item/lua/vis.lua#L268 gives me:

/usr/share/vis/vis.lua:268: bad argument #1 to 'for iterator' (table expected, got nil)

How do you translate lexer._TOKENSTYLES to the current codebase?

Cc: @ninewise @Nomarian @qiu-x @jpe90 @deepcube @martanne @orbitalquark

[mcepl]

Second effort (see https://lists.sr.ht/~martanne/devel/patches/43512 and https://lists.sr.ht/~martanne/devel/%3CCAEbpLMBpYoA_RUyjoPMBs%3DU5Xx_59pamNfqgPRD4G4xdFX9iYA%40mail.gmail.com%3E) seems to lead to good progress. The set of patches rebasing on the scintillua_6.2 tag has been prepared.