A Tomboy user with 10,000+ notes can create and rename notes without 5-20 second losses of access

[mlncn]

I have more than 15,000 notes (taking up 75 MB in the ~/.local/share/tomboy folder) and since at least the 10,000 mark (but i think well before) i have experienced frustrating degradation of my experience due to performance issues, and i can get used to almost anything.

• Starting Tomboy can take five minutes or more before it loads and can be used. This i can live with.
• Creating a new note can take 5-15 seconds for the blank note to appear (more if Tomboy is competing for resources)
• Naming or renaming a note can take 7-20 seconds after clicking away from the title to be able to type or take any other further action with Tomboy (again, longer if Tomboy is competing for resources)
• Intermittently, Tomboy will be inaccessible for several seconds (but not greyed out; i'd classify being greyed out/unresponsive as a separate issue that only happens when my computer is over taxed)

Making sure that a heavy user can continue to use Tomboy smoothly while creating, naming, and renaming notes is something that i can and will put $5,000 toward fixing.

Preventative note to anyone who wants to blame Mono: I tried GNote a while back and the performance was much worse.

Note: I don't have the newest computer but it's a dual core Intel Core i5-2540M @ 2.60GHz x 2 with 16GB RAM and a solid state drive. I don't expect throwing hardware at this to fix it; it's a scaling issue.

Additional note: I do not have many add-ins enabled and have tried with all add-ins disabled; no appreciable difference in performance.

[mlncn]

Really excited to see this project active! When i was last looking into this over a year ago i guess, i didn't even see a good place to beg for help / offer a bounty. I'm hoping there are other people out there who use Tomboy for tens of thousands of notes (or would go back to it if it were more performant).

I forgot to mention that classifying a note into a notebook can also disable access to use Tomboy for several seconds, and that this is important for me to fix also.

I am not a desktop application developer and have not made much progress in understanding Tomboy code. But as far as i understand, all the notes are loaded for cross-linking and search purposes. I think two changes would address the speed issues and make Tomboy as fast as anyone could want:

1. Create an index for note titles and another index for search terms.
2. Make going through note text looking for text matching note titles to link a background process.

I'm sure i'm missing some critical nuance, but is this an enhancement i can help make happen by throwing money around?

Thank you!

[kernelOfTruth]

Yeah, GNote is horrible in that regard, I'm currently using it with jemalloc preloaded which seems to make performance a bit better, it however still takes quite some time to open up a new note (up to 30 seconds or 2 minutes), my tomboy folder is also quite big

Usually with tomboy + mono - upgrading and using the latest mono versions tends to improve performance and make things better

There is a site on the web where folks can gather around and back it (or what its name is) to have a feature implemented or problem fixed,

can't seem to find it right now ...

edit:

seems it was bountysource: https://www.bountysource.com/

https://www.bountysource.com/teams/tomboy-notes/issues

[davidbannon]

Ben, great move and I hope it works (not me, don't do Mono). But some discussion.

I reckon the issue is the flat file approach. Iterating through 15,000 files in one directory is a slow process and one that taxes most file systems. Have worked with HPC systems with similar problems, science user drop lots of small files and their access and backup and restore times suffer.

A database backend would make a lot of sense, it would be sensibly sqlite for the average user with the option for a extreme (and Ben, you are extreme) user to plug it into mysql or whatever.

I assume Mono has database interfaces that can easily switch between backends ?

Ben, one other suggestion, there may be some benefit in using a different filesystem, I think its possible some are more suited that sort of numbers than other. But been out of that space for 5 or 6 years now and don't know authoritatively. Assume you are using ext4 ?

Davo

On 02/07/17 01:07, Benjamin Melançon wrote:

I have more than 15,000 notes (taking up 75 MB in the ~/.local/share/tomboy folder) and since at least the 10,000 mark (but i think well before) i have experienced frustrating degradation of my experience due to performance issues, and i can get used to almost anything.

• Starting Tomboy can take five minutes or more before it loads and can be used. This i can live with.
• Creating a new note can take 5-15 seconds for the blank note to appear (more if Tomboy is competing for resources)
• Naming or renaming a note can take 7-20 seconds after clicking away from the title to be able to type or take any other further action with Tomboy (again, longer if Tomboy is competing for resources)
• Intermittently, Tomboy will be inaccessible for several seconds (but not greyed out; i'd classify being greyed out/unresponsive as a separate issue that only happens when my computer is over taxed)

Making sure that a heavy user can continue to use Tomboy smoothly while creating, naming, and renaming notes is something that i can and will put $5,000 toward fixing.

Preventative note to anyone who wants to blame Mono: I tried GNote a while back and the performance was much worse.

Note: I don't have the newest computer but it's a dual core Intel Core i5-2540M @ 2.60GHz x 2 with 16GB RAM and a solid state drive. I don't expect throwing hardware at this to fix it; it's a scaling issue.

Additional note: I do not have many add-ins enabled and have tried with all add-ins disabled; no appreciable difference in performance.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/tomboy-notes/tomboy/issues/54, or mute the thread https://github.com/notifications/unsubscribe-auth/AF__MA6b-DrUZQATo7ZMHxE-obCVZ6ryks5sJmDHgaJpZM4OLVoh.

[davidbannon]

One other suggestion Ben, try mounting the disk where Tomboys notes are with noatime. Edit /etc/fstab, find the line and it will probably have "defaults" under options, make that "defaults,noatime". Remount or reboot. That will tell the file system to not keep a record of when each file was last looked at. I assume most of your notes are only occasionally written to but are read during those peak times you mention. You won't miss atime, bet you have never used it. Last changed time is still recorded .... David

[alex-ter]

This number of notes is indeed quite impressive and I'm afraid Tomboy is simply not architected to handle such amounts effectively. However $5K is an impressive number as well :-) so I think your idea of starting a bounty is a step in the right direction.

As for viability - I think (and this is bugging me for a while, after I started reviving the project and looked more attentively at the code and dependencies), it may be easier to rewrite everything from scratch maybe in a modular fashion with tomboy-library at the base as previous maintainers started to work out. Not only performance-wise, but from the evolution standpoint - Gtk2 is now slowly vanishing, there are no builds for Windows for years, Gtk3 C# bindings are still not there, same as Win version, not to mention work required to port Tomboy to Gtk3. On OSX the dependendency library is similarly stopped being supported. All in all this looks like a need for a major overhaul and a respective team of contributors (or a lot of time for one :)

[davidbannon]

On 06/07/17 04:53, Alex Tereschenko wrote:

...... it may be easier to rewrite everything from scratch..... Gtk2 ..... Gtk3

OK, we'll rewrite it in pascal. Lazarus claims "write once, compile anywhere". :-) For something I'm doing, seems it might just be true-ish ??

David

[alex-ter]

On the language level everything is fine - C# was a great choice for the goal at hand. However besides that you need gui as well as "middleware" framework and that's where problems begin in our case, rooted in a lack of love for a while + natural framework evolution we now have a situation, where our dependencies are no longer that widespread and evolving in a different direction than before, which calls for rethinking the approach and ways into the future. At least that's how it looks to me.

[mlncn]

@davidbannon Thanks! I have "rw,errors=remount-ro" for the options column for the disk with Tomboy (which has everything, not just Tomboy, so i'm a little hesitant to change anything). And yes, the file system is ext4.

@alex-ter et. al. I'm happy to help support a major overhaul but want to make sure it helps extreme users like me. The architectural change of using indexes—even if still using flat files not databases—would i think address my needs.

[davidbannon]

Hi Ben, understand your concern but I do think you may get some benefit with just about zero risk. atime records the time a file was accessed, that is read. Its quite unnecessary, a file system feature you don't make use of. Windows users don't get the option for example. Eat my own dog food ? OK, I just set that option on my laptop, its my main computer, use it for everything from reading the news to dev work :

dbannon@Aspire:~$ touch blar
dbannon@Aspire:~$ ls -l blar
-rw-rw-r-- 1 dbannon dbannon 0 Jul 24 10:08 blar
dbannon@Aspire:~$ grep home /etc/fstab
UUID=82a2a17b-cc32-.......a /home   ext4    defaults,noatime   0   2
dbannon@Aspire:~$

I doubt it will give me any benefit as I don't have any directories with that quantity of files but it does no harm either. For you, the difference should be measurable, not huge but worthwhile. David

[alex-ter]

@mlncn, besides what @davidbannon is suggesting, there are those crowdfunding/sponsorship sites these days like BountySource and similar, please feel free to start a bounty there and post the link in this thread, that I think would be the best approach from the money-based prospective.

[davidbannon]

Sorry Ben, this is really to Alex and the "old hands'. Alex, month ago I made a throw away crack about rewriting in Pascal and Lazarus. Well, have to say I think its actually worth considering. Please see http://www.bannons.id.au/tomboy there I have made a start in doing just that. Its a basic editor so far, does most of the the markup, reads and writes Tomboy note files. Links to existing notes (and easy to disable). Would be easy to move the notes into a database. Does not do sync and a whole lot of other things. But its less than a thousand lines of code and the IDE generated more than half of that ! Builds cleanly on Windows and (gtk)Linux and needs no additional libraries ! I have not tried on a Mac yet. The key point is its someone else's problem to ensure it continues to build and run on other platforms. On a clean install of Debian xfce, Gnome3, Ubuntu Mate and Win10, all I needed do was to copy over one binary to have it work.

Anyway, thought it would be polite to run this past you before posting on the more widely read mailing list. Your thoughts please ?

David

[alex-ter]

Hi David,

Ah, Delphi (well, not exactly, but very close), that brings some nice memories... :)

Thanks for asking. I would separate several things here:

1. The performance problem described here is not within the GUI part of the Tomboy, rather in its foundational architectural decisions. So just swapping the language/framework per se won't help;
2. I personally treat the fact that notes are stored as plain files as Tomboy's advantage, as I 'm confident I'll be able to access/backup/mess with them with Tomboy or not, so it's IMHO more future-proof. Using DB has its advantages and disadvantages, but the performance part of it is not actually clear cut, depending on specific data access layer implementations both plain files and DB may be speed daemons or slow down to a crawl. It just happens that Tomboy currently creates objects with full note content for all notes it knows about for faster and simpler access/searchig logic and it wasn't tested/intended for thousands of them, as it does this synchronously on startup. So I'd rather keep the file-based approach and just make the data access layer smarter, including the "thousands of notes" as one of the basic requirements;
3. Speaking or rewriting, I'd rather take the approach similar to the one maintainers took in the past with Tomboy library - separate the data access logic into a "headless" very cross platform, well-tested, library, so that GUI is just a thing "on top" and can be built with anything - either a single cross-platform framework like Lazarus/wxWidgets/QT/whatever or some native GUI implementations per platform, should that be needed. I'm not sure I'd completely agree with namely Tomboy library approach of putting things like Sync into the library, but then I haven't really analyzed its architecture for any length of time yet.

So all in all, I'd say if we want to rewrite Tomboy (and I'm almost at the verge of starting doing just that, seeing the current state of frameworks we depend on), I'd start with either reviving or writing Tomboy library anew and then GUI would be a module on top.

[davidbannon]

Thanks Alex. Perhaps I should have created a new ticket. What I am suggesting is not specifically to address Ben's 5K notes performance issue. Its about getting Tomboy back to being a practical cross platform application again. While Mono and GTK for Windows is technically a great solution, neither are being widely supported anymore and GTK on windows is rapidly becoming impractical. Most messages to the mailing list are from Windows users struggling with the software stack. We need to act before we lose them in the same way we have already lost the Mac people.

I suggest that Lazarus provides the "GUI Module on top" that you mention. Other people are in charge of ensuring it is cross platform. Tomboy as a product will never have a community big enough to manage all those layers. You are the only one pointing vi at the code. Its too much Alex !

I have little doubt someone in the Lazarus community would, for example, pick up Ben's bounty. It would be an easy implementation in a well modularized build. But like you, I'd prefer to stay with text files for my notes.

yep, its Delphi ! I played in that space 17 years ago. I only recently 'discovered' Lazarus when someone asked me to make some changes to a program I wrote back then ! To my surprise, it was easy and I could do it all on Linux, only testing on Windows (its target platform) at the end.

Lazarus can link to external C type libraries, so a Library would work fine. But I am not sure just what would need be moved to a library ? Obviously sync but most other functions seem to be too attached to the OS to be suitable. To work well, I'd see the library totally platform independent.

David

[alex-ter]

I see, so if this is general future direction discussion then yes, it would be better held in a separate record.

You see, the caveat here is that it's easy to create a very basic functionality, like your demonstrator, but implementing things like synchronization, i18n, modular plugin system (for things like storage, sync, maybe formatting/UI) - that would be much harder and without something like a Library I was talking about, the resulting program will again be too coupled to the UI.

Overarching that is the fact that I personally am not sure I can afford to dive into a project like this, which would take much more time investment than I have and than I originally anticipated for Tomboy to take me.

However that by no means should stop the discussion about Tomboy future - after all there may be other people interested :) So if you could start a new "issue" for that, optionally copying the relevant pieces from this one (or I can copy that later), that could help I think.