Typing lags, CPU gets my laptop fan spinning, on larger notes when typing quickly

[dajoker]

openSUSE 15.1 x86_64, KDE tomboy-ng 0.30.2, non-Qt

I have noticed this for a while and figure it may be worth reporting finally. Today's note, which is exhibiting this symptom anytime I type in it more than a couple lines, is around 37157 bytes of text (when copied into another text editor), and the actual .note file is 38309 bytes, so not enormous, but quite a bit of text. If I go and continue typing more data as I usually do, the cursor will stop advancing, letters not showing up, until I pause for a half second or so, and at the same time my laptop's fan lets me know my system is now working really hard doing something, though I have no idea what other than tomboy-ng-related.

If I stop typing for a bit, it calms down again. If I copy/paste a large block of text, no problems overall. If I work in smaller notes, the problem is NOT as bad, but I can duplicate this easily by putting several dozen KiB of text into a note and then trying to type in it, and it gets worse as the note grows in size. Watching "top" I see one CPU core hit 100% and stay there while typing.

Steps to duplicate: Create a note with 50 KiB text within (just to make it happen more). Type in the note as fast as possible while watching the CPU.

Expected results: Not much, other than data being typed

Actual results: A core is very busy processing the text for some reason

Able to duplicate: always on a larger note

I presume this has something to do with tomboy-ng trying to watch for titles of other notes, or URLs, or something, but it's a bit annoying to not be able to see if I have typos while typing unless I pause to let the system catch up.

[davidbannon]

This is not a lot I can do I am afraid. As you say, its just a scaling issue. I have several 50k note in my normal repository, I don't experience this problem on my working laptop but sure do on my smaller testing system. Its an Atom powered unit and really struggles with larger notes. Truth is quite a lot is going on every time you press a key.

I expect your problem is more cpu than memory. It might be interesting to see how much an improvement happens if you turn off the auto functions, sync and snapshot probably won't make a lot of difference but turning off the auto linking (to other notes, web addresses) probably will.

I am guessing you have a lot of notes too ?

Davo

[dajoker]

Yes, I have 4497 notes currently.

Turning off the two "show" options (internal links and external links) did not make a difference. After doing so (without restarting tomboy-ng entirely) I copied the contents of yesterday's big note to a new note, and now the links (internal and external) are all gone. I then tried typing and on the second character it hung, so I kept typing, finished my paragraph, and then paused for a couple seconds, and everything then showed up. As a result, it appears disabling those options does not help.

I do not have auto-snapshots or auto-sync enabled.

[davidbannon]

Hmm, I am surprised turning off the links did not help. It does on my underpowered test system. That says its a problem related to the size of the note rather than the number of notes you have. Are there other apps running at the time that is consuming resources ? On my normal working laptop, I have a 8 core processor, so each individual core is not terribly fast. If I use the keyboard's auto repeat, highest I can get is 40%-50%.

Lots of bullet points may generate more cpu demand, the way I do it is quite ugly in kcontrols.

I am considering, very long term, replacing kcontrols with richmemo and expect it will be faster but thats only guess work right now. I don't have much to offer you at this stage, sorry.

Davo

[dajoker]

I did not restart tomboy-ng to see if that would possibly impact things; should I need to do that? I, too, was surprised that it was so busy without the links being checked. I still have Lazarus on my test system and I could potentially try to duplicate things there to see if I can cause the same symptom, but perhaps with more troubleshooting tools specific to the language. Otherwise, I'm decent with OS troubleshooting tools (strace, etc.) so if you have things I can watch I'd be happy to help.

This laptop is my primary system with its 5000 notes of varying sizes (49 MiB total size for ~/.local/share/tomboy-ng). 'lscpu' shows eight (8) processors, i7, 2.7 Ghz, 16 GiB RAM (6+ GiB free), 600 GiB SSD, so it's not a slouch, but it's also not even close to being new.

[davidbannon]

Sorry, did not get back to you here. I am convinced that TRichMemo is not yet even near ready for this sort of use and TK has very kindly changed KControls license so its acceptable to Debian. So, it looks like its going to be KControls for some time if not indefinitely.

I am just a little bit wondering if it would be worthwhile me sending you a version of my RichMemo based tomboy-ng ? Its got lots of problems and must not be allowed to save but I could force that in the test binary.

Hmm, 'save' ? I wonder if its the auto save that causing you a problem ? I experimented with running the save in a separate thread but decided it was not worth the complication but that was on my system, maybe I needed to check a bit wider ......

Again, a test version that does not save might reveal whats happening.

[dajoker]

What is/are the difference(s) between TRichMemo and KControls, other than licensing? What other options, if any, are there?

I appreciate the offer for another build, and I'm willing to try one if you'd like, but I personally do not feel that save is necessarily the issue. The notes that I have are plenty, and the problem only shows up when they get bigger (near the end of the day, but sometimes sooner if I paste a lot into them), but my understanding is that you do not save on every keystroke, and that understanding is because the asterisk in the title bar only goes away after a few seconds of inactivity. With that in mind, there is no way a save every dozen seconds is causing this problem. If, for some reason, it is saving with every keystroke, and that asterisk means something else, then yes let's fix that, but I find that very unlikely.

Per-keystroke things seem more like your earlier idea around URL detection, though that too is not instantaneous, or spell checking (though that is not an inline thing at all with tomboy-ng, sadly), or maybe other-note-detection, or whatever.

I am not sure I mentioned before, but when this gets bad the text actually stops showing up until I pause for a fraction of a second, and then it all shows up at once. If I get going again at my usual rate (push 100 wpm) the text will freeze again until I pause for a part of a second, then all show up again. I think this also points away from saving and toward other text analysis, but I have no idea what that could be other than the ideas already presented by each of us.

[dajoker]

Here's another tidbit; it seems to have more to do with many lines than much data, or so I think. e.g. if I put 8k lines in a note, I can cause the problem terribly, but if I put 8k characters in a line it is not as bad. If I put in eight of those 8k lines (so now the note size should be (but I did not check) much larger than the 8k-line note) I can still operate reasonably. What cares about many lines, other than perhaps UI rendering due to break/paragraph/div tags or something?

[davidbannon]

What is/are the difference(s) between TRichMemo and KControls, other than licensing?

KControls is, substantially an all Pascal implementation, as such it works, more or less on any platform supported by Lazarus, and thats a lot ! RichMemo on the other hand uses, directly, the underlying OS Widgets, so it makes direct calls to GTK2 or Gt5 widgets. So, in practice for each 'thing' to work, someone needs to implement it on GTK2, GTK3, QT5, Win32 and Cocoa (ie MacOS). A lot of them are not done. I have sent a few in but the manager of that project is somewhat slow to act. Sigh.
No license issues on either now. I would expect RichMemo to be much faster because its does a whole lot less. KControls does absolutely heaps of things we don't need.

I appreciate the offer for another build, and I'm willing to try one if you'd like, but I personally do not feel that save is necessarily the issue.

Yes, I save nominally every ten seconds or so but try to find a quiet time to do so.

Per-keystroke things seem more like your earlier idea around URL detection, though that too is not instantaneous, or spell checking (though that is not an inline thing at all with tomboy-ng, sadly), or maybe other-note-detection, or whatever.

What worries me is I am not duplicating your issue here. On a much slower system, turning off in line links and not using bullets makes a huge difference, you are not seeing that difference. I would really like to know why not. I suspect that would be an eye opener....

...when this gets bad the text actually stops showing up until I pause for a fraction of a second, and then it all shows up at once.

Yes, thats pretty much what I would expect if the app is struggling to accept characters that are banking up in the keyboard buffer. Now, that struggling can be due to processing being done in my code or in KMemo's code. 'My code', active on each key press, is where I look around to see if any new links need to be dealt with, massage the bullet display, check for any control characters and, possibly, saving to disk. All of that is, of necessity, single threaded (saving does not have to same thread, its just I found no real benefit in giving it its own).

Hmm, time I did a critical audit of that part of the code. So easy to keep hooking into that loop.

... my usual rate (push 100 wpm)

??&% ?

Davo

[davidbannon]

Here's another tidbit; it seems to have more to do with many lines than much data, or so I think. e.g. if I put 8k lines in a note,

OK, thats interesting. And unexpected. I will need to consider ....

Davo

[dajoker]

Another little tidbit from my test rig (fired up just for this fun testing).

I created a file to cause problems, 100 lines of 10000 characters each, so basically 1 MiB in raw data that's just repeating long strings over and over, so no chance of a match on any title or anything.

Now whenever I press a key it locks up for a while, probably twenty (20) seconds or more, though I'm not timing it. If I watch strace output of the tomboy-ng process at the time, there is nothing at all with the filesystem that entire time, right up until near the end when I see a temporary file written and then immediately renamed (to overwrite the real note), and since it's just a 1 MiB file that's all very quick. The point is I see no file I/O at all until the end, so I do not think this is a problem with the I/O part necessarily (though writing a completely new note and then renaming with every save is a bit intense, but very safe avoiding corruption I suppose).

[davidbannon]

Hmm, I have done a somewhat similar test, I made a note of somewhat shorter note of about 10K lines and about half a meg. About half the lines are empty lines, a test inspired by your message of 9 days ago.

I can generate a 6 to 8 second delay when typing my sort of fast, a lot slower than you !

I hope I can now use that to embed some debugging statements that will let me isolate the problem into a particular area. But have not progressed that far yet !

Davo

[salamander2]

I think you should see if other people can duplicate this problem too.

[davidbannon]

Just released version 0.34, it has a couple of tweaks to speed up saving to disk. Just possible that this may impact on the issues discussed here. Otherwise, not much I can do I am afraid.

In tidying up mode so closing this note !

Davo