[BUG] With 8k+ tiddlers, in a NodeJS wiki everything is Slow

[linonetwo]

Describe the bug

I have imported 8k+ google calendar events into my nodejs wiki, they are mostly metadata, no long texts. Then each click in my wiki becomes slow, saving is slow, and starting the wiki is slow too.

• Each click takes 1.2s to calculate
• Starting wiki takes 10s+
• Sync adaptor saving a tiddler takes 1.2s

Expected behavior

Be as fast as a empty wiki when we have 10 - 100k tiddlers, in nodejs wiki.

To Reproduce

I will prepare a github repo later, but now I'm debugging in my own wiki...

Screenshots

1. It is core/modules/wiki-bulkops.js that being so slow on client side, when clicking bottons

2. Starting TBD, what happened on nodejs side?
3. Saving TBD, what happened on nodejs side?

TiddlyWiki Configuration

• Version 5.2.5
• Saving mechanism [Node.js]
• Plugins installed [lots of]

Desktop (please complete the following information):

TidGi (darwin) Version v0.7.13.

• Electron Version: 22.0.0
• Node Version: 16.17.1
• Chromium Version: 108.0.5359.62

Additional context

I'll do some metrics tomorrow, and come up with a plan. I must fix this, because I want to use tw as my diary tool to replace the google calendar.

[pmario]

• Plugins installed [lots of]

Did you try to create a new server and import your data there. Is it still slow. If not you may have to look at your plugins.

[linonetwo]

Can we remove the toLowerCase() here?

[Jermolene]

Hi @linonetwo performance at that scale is definitely a problem, and we welcome investigations to identify and optimise performance hotspots.

One thing I'd love to experiment with is to use the new SQLLite JavaScript release as the basis for the wiki store object. It has a synchronous API which is needed for the TW store.

[Jermolene]

Can we remove the toLowerCase() here?

It is certainly possible that there is a more performant way of making a case-insensitive sort, and again would welcome investigations.

[linonetwo]

@pmario You are right, some of friends in QQ group report lagging is caused by locator plugin and kin filter plugin. I will try this in a clean environment.

[Jermolene]

I should have added that there are three usages of wiki.getTiddlers() in the core, and it looks like the usage in wiki.countTiddlers() doesn't need to be sorted at all, but in the other two usages the sorting is visible to users, and so cannot be altered without changing the semantics.

[linonetwo]

SQLLite JavaScript release

One thing I like about nodejs wiki is its file based storage, which is nice for git diff, so I don't really want to use sqlite to solve this problem.

[Jermolene]

One thing I like about nodejs wiki is its file based storage, which is nice for git diff, so I don't really want to use sqlite to solve this problem.

We wouldn't necessarily need to use SQLLite as our persistence format; we could still save to the same file formats that we do today.

[linonetwo]

I get it, you mean use sqlite as memory cache to speedup querying?

I can see the sync adaptor is slow because get all tiddlers are slow. I'm going to add some server-timing tomorrow, and maybe PR it as a metric to core.

[pmario]

... is caused by locator plugin and kin filter plugin.

I don't know locator, but IMO kin filter should only be slow, if it is shown. Similar to the Recent tab. So if Open tab is shown by default it may be faster.

[linonetwo]

[all[shadows+tiddlers]tag[$:/tags/StartupAction]!has[draft.of]](12942times)

Don't know where this filter is constructed, but it takes long time during GET /

[linonetwo]

It is kin-filter that being slow, but why will it be called during starting?

flamegraph.html.zip

---

I know, it is core/modules/utils/filesystem.js's generateTiddlerFilepath using filters from FileSystemPath.tid, in which I add a kin-filter.

[linonetwo]

My script to generate the flame graph prifiling:

0x -o ./tiddlywiki.js "+plugins/tiddlywiki/filesystem" "+plugins/tiddlywiki/tiddlyweb"  /Users/linonetwo/Desktop/wiki/wiki --listen anon-username= port=5212 host=0.0.0.0 root-tiddler=$:/core/save/lazy-images
🔥  Profilingsyncer-server-filesystem: Dispatching 'save' task: $:/StoryList
Serving on http://0.0.0.0:5212
(press ctrl-C to exit)

using https://www.npmjs.com/package/0x

[saqimtiaz]

Are there any startup actions in the wiki that use the kin filter?

Edit: Nevermind, just saw that it is used in the FileSystemPaths.

[linonetwo]

After removed the kin-filter, I added up the filter consumed times, its 1004ms in my wiki, so other code takes 1.5S, I need to add more perf timers.

---

There are still something to optimize on server side. But it seems not very urgent now. I'm going to fix client side first.

without kin-filter flamegraph.html.zip

[linonetwo]

try https://github.com/Jermolene/TiddlyWiki5/discussions/6550

[pmario]

There are still something to optimize on server side. But it seems not very urgent now. I'm going to fix client side first.

@linonetwo ... Did you find the problem? It's not clear to me.

Do we need to keep this issue open -- or do you create a new one when you have more data

[linonetwo]

On server side, it is mostly because of FileSystemPaths.

On client side it is not clear, I will find time for this, so keep this open for a while.

[yaisog]

@linonetwo: Depending on what you need kin for, you might want to look at https://talk.tiddlywiki.org/t/recursive-filter-operators-to-show-all-tiddlers-beneath-a-tag-and-all-tags-above-a-tiddler/3814 I made these as a replacement for some of kin's functionality for large wikis, where kin becomes slow.

[linonetwo]

This issue is workarounded by using yaisog/taggingtree-filter and linonetwo/in-tagtree-of plugins.

And boot speed can be further increased by #7329