improved splitting: store end marker in registry

[emmanueltouzery]

fixes #54

this is what we were talking about... storing the next marker in the index so that we know where to cut the preview. works in my tests! fixes the case of rust, doesn't break any of the other docs I tested.

[emmanueltouzery]

and yes, it's more complicated than what I hoped for, because the main bit where we process doc entries is processing them as an unordered list, so you can't just get "the next one", so i had to introduce a new list.

[emmanueltouzery]

another thing to keep in mind... the change might cause a regression for users, if they don't refetch their docs.

[emmanueltouzery]

Hmm. I see now your suggestion at the time highlighted another possible implementation of this issue 😬 I wish I read that more carefully before doing this. Well, maybe check what I did and it you still think the approach you suggested was better, maybe changing this to that approach is not too bad. In any case this is at least a proof of concept.

[luckasRanarison]

Thanks again for this PR!

another thing to keep in mind... the change might cause a regression for users, if they don't refetch their docs.

This will probably be the last breaking change that'll be made to the index file and rebuilding docs shouldn't be a big deal, nvim-treesitter litteraly recompile every parser after each update...

The splitting issue is fixed and it's really promising, but as you've noticed it can be improved :)

The optimal way would be ordering the entries when transpiling, so storing the next pattern in the index file could be avoided. If the entries are ordered, we could instead add a reference to the next entry at runtime and achieve the same result. The next entry will not be necessarily be the next section but that'll do the trick

Do you still want to work on it?

[emmanueltouzery]

I understand. I'll try to whip up something today. And we'll see how far I can get.