FND / tiddlywebplugins.bfw

Barely Functioning Wiki, emphasizing micro-content
0 stars 2 forks source link

loading long lists of tiddlers takes ages #54

Open FND opened 10 years ago

FND commented 10 years ago

loading /bags/tiddlers/bookmarks for a bag with ~1200 tiddlers takes several minutes, with occasional CPU spikes from Git

in fact, even loading a single tiddler (page) takes unusually long:

$ time curl http://localhost:8080/bags/meta/tiddlers/index
real    0m0.089s
user    0m0.003s
sys 0m0.003s

$ time curl http://localhost:8080/bags/bookmarks/tiddlers/index
real    0m0.691s
user    0m0.003s
sys 0m0.000s

seems like an issue with gitstore - IIRC it queries Git for the latest revision rather than just reading from the filesystem (which might have uncommitted changes)

ticketing here to make sure it actually gets looked at

FND commented 10 years ago

the store code looks fairly innocent

best to discuss profiling with @cdent before chasing ghosts

cdent commented 10 years ago

Just for a quick test I would see what happens if you stub out the call to run. That fork may be expensive.

FND commented 10 years ago

You might be right: The offending line is https://github.com/FND/tiddlywebplugins.gitstore/blob/379033f4d5704f3f14c5cb3d0c6422dcfaefe2b1/tiddlywebplugins/gitstore/__init__.py#L81-82 (this affects tiddlers listings because tiddler_get is being called)

However, it's not the forking that's slow, but Git itself:

$ time git log -n1 --format=%H -- bags/bookmarks/tiddlers/helloworld
e4de29934938cc5cec481207aaec3b5ee29d2cd0

real    0m0.592s
user    0m0.537s
sys 0m0.057s

That kinda sucks...

FND commented 10 years ago

After a fair amount of consideration:

FND commented 10 years ago

actually though, due to tagdex, modifying the store directly can lead to inconsistencies - so we might additionally store the last known commit, check whether there were any new commits and recalculate the caches/indexes for everything after that