search

jgm / skylighting

A Haskell syntax highlighting library with tokenizers derived from KDE syntax highlighting descriptions
195 stars 63 forks source link

How does `tokenize` separate `SourceLine`s? #182

Open Eisverygoodletter opened 11 months ago

Eisverygoodletter commented 11 months ago

The function tokenize outputs an array of SourceLines. Is it guaranteed that if i provide it with Text with some multiline code, every line's tokens will be mapped to one SourceLine without any formatting/line changing behaviour?

For example, if I had this single line of oddly written piece of C code:

int do_something() { call_a_function(); int some_number = 3; return 123; }

Would very token (e.g. int, do_something, (), {, call_a_function ...) will end up on the same SourceLine with every token in sequence?

jgm commented 11 months ago

Yes -- why don't you simply try it?