Is it possible to turn vlf into minor-mode?

[tuhdo]

vlf provides a very nice way to divide and conquer large files. It would be much nicer if only it becomes a minor mode to be used with other major modes for extra features of the major modes, while having superb performance on large file.

Only the core feature of vlf, which allows Emacs to read a file chunk by chunk and later save back would be far enough.

Thanks.

[m00natic]

This has been crossing my mind lately too. I'll see what I can do ;-)

[m00natic]

If you feel adventurous, have a go at the minor-mode branch. I'll do it myself in the next few days and if no major regression pops out, should go in master towards end of week.

Beside VLF becoming minor mode, I made several refinements to be more cautious in case of modifications. Also other little improvements and fixes. I'm not sure C-c C-v is the best prefix we can get but all in all VLF integration now seems more natural.

[tuhdo]

Very nice!!!

Despite being in alpha status, I can already use it to view large file conveniently.

After playing with it for a while, here is a list of things that I think could be improved:

• Make a global minor mode to make any opened file using regular functions (find-file, ffap....) use vlf by default.
• There should be a variable to keep track of global line number between chunks and display the current line appropriately on mode line. I think this would help the developers of various linum plugins to adapt to vlf.
• There should be a variable to keep track of global percentage position. Both of this and the above features make experience with vlf more transparent and users wouldn't see the differences between vlf/non-vlf mode, except the performance difference :) . Doing this, you can actually replace normal incremental search (C-s) with vlf search.
• Context should be preserve between the next and previous chunks. For example, I'm editing a C file in vlf mode. At the end of current chunk is in the middle of a long C comment. Jump to the next chunk, the highlighting will be messed up because the current chunk loses the comment opening character "/*"

Thanks for the improvement. Please keep it up!

Edit: I think the most important issue to fix is the syntax highlighting between chunks. This makes vlfi functional with the major modes.

Edit2: If you want to release at the end of this week, you should do it. I don't think there's an easy fix for the highlighting problem. You can improve vlf with my other suggestions to make it better though.

[tuhdo]

Also, I think that C-c v is fine for prefix already, i.e. C-c v s for search... We don't need the extra control.

[m00natic]

Thank you for the comprehensive feedback and good ideas!

• Global minor mode - when did VLF become that useful :D I like this and will do it. But I'll first have to address the unreliable saving of changes which would be fatal for extensive use.
• Global current line information is doable with some idle timers, I think. It would optionally scan back the file and run some hook when ready. Jumping to consecutive batches may have some optimization.
• Global percentage position is already indirectly present through vlf-start-pos, vlf-file-size and vlf-batch-size. This is actually continuously reflected in the buffer name and shows current batch number / maximum available batch number. At some point even percentage was present, but I think the raw numbers are more useful "I'm currently on chunk 4225 of 8500...let's jump 1000 forward".
• Isearch integration is definitely on the TODO. Also Ediff.
• Context - this seems more complex. Like if you have huge comment block that spawns multiple batches, vlf would have to scan back a lot to detect this and somehow integrate with individual language or whatever modes. VLF would gain a lot once threads are available to Elisp and maybe something about context would be more doable then.

I'll later create tickets so things will be more tractable.

[m00natic]

[ 5 декември 2013, 05:58 +0200, четвъртък ] tuhdo:

Also, I think that C-c v is fine for prefix already, i.e. C-c v s for search... We don't need the extra control.

I'm afraid this is against Emacs guidelines where C-c + single letter is reserved for users. Maybe "C-x v" would be possible if VLF becomes Emacs built-in.

[tuhdo]

Context - this seems more complex. Like if you have huge comment block that spawns multiple batches, vlf would have to scan back a lot to detect this and somehow integrate with individual language or whatever modes. VLF would gain a lot once threads are available to Elisp and maybe something about context would be more doable then.

Actually this can easily occur without a comment or a string spans multiple batches. For example, the last line or few lines of a batch is in a middle of a comment. Moving to the next chunk will cut off the information from the previous chunk, and the highlighting is messed up. I think you can perform a regex search in previous batch to find the opening character of a pair and add the content from the point of that character until the second batch is filled up. The should be a set of opening pairs for VLF to search backward:

• Opening of C Comment:/*
• Double quote: "
• Single quote: '

Once it finds theses characters, it loads the next batch from this point. The default set can also be customized for more opening characters.

User can also manually specify a point where the next batch should be loaded from that point. At least doing it this way, a user has the ability to control the batch content and able to prevent the broken highlighting problem. This is pretty practical, because I think user will usually specify a big enough batch size, so a comment that spans multiple batches hardly happens. Currently I set my batch size to 500,000 bytes. You should set the default size of VLF bigger as well (like 100,000), so the situation like comment spans multiple batches won't happen.

It is also much easier to implement compare to the automatic solution.

[m00natic]

I understand what you observe (me too). Unfortunately such automatic scanning would be mode dependent. Even one-line comments should be taken into account. Org-mode headers, tables, Lisp sources (finding the top-most opening paren), different comment symbols for different languages... Imagine the comment block is additional 10MB (OK this is pathological). I'll defer this for now.

The idea of manually specifying where to start next batch is cool and easy hack and gives control to the user. Coming soon ;-)

[m00natic]

I have now pushed the minor-mode branch in master. Added a command to display batch starting from point C-c C-v n. Also opened tickets for things we discussed here. Enjoy!