search

tiddly-gittly / HugeWikiExample

Example wiki to reproduce performance issue with 10k+ tiddlers
https://tiddly-gittly.github.io/HugeWikiExample/
MIT License
2 stars 0 forks source link

16MB PDF is embedded into the wiki?! The wiki is 240MByte in size #2

Closed pmario closed 1 year ago

pmario commented 1 year ago

@linonetwo ... I did just run the dev server of this wiki using a completely new install from node.js, vscode, git, npm.

image

The TW server does have a filer server option. BUT it needs a files directory on the same level as tiddlers

From my point of view, with the huge files loaded from a files directory the whole wiki should be sent over the wire in a view seconds ...

pmario commented 1 year ago

There is a very big problem with embedded files in TW. They need double the size, because they are base64 encoded

I did move the big files out of the way and started TW like this. It starts up in about 5 seconds. Down from 30+

cd /path/to/HugeWikiExample/
tiddlywiki.cmd +plugins/tiddlywiki/filesystem +plugins/tiddlywiki/tiddlyweb demo --build index
npm run dev

seems to rebuild the whole thing everytime I run it. ... I want to build once and then only run it for testing. How do you do that?

pmario commented 1 year ago

I did a bit more testing ... and the whole tag-pill macro rendering seems to be a huge problem. With an init and the CDDA tiddler open. While rendering the links the parser is started 20000 times. That should not happen. we should hit the cache 20000 times.

linonetwo commented 1 year ago

Hi, you can run npm i then npm run dev-html.

Or pnpm i then pnpm dev-html.

  • It should use _cannoical_uri and lazy-load the thing

That is true, I forget this. In my personal wiki, I use lazy-image that also exclude the PDF. Maybe we can just remove the PDF from this repo. I was trying to reproduce the situation in my personal wiki.

linonetwo commented 1 year ago

pnpm dev is good for developing plugin, it auto reload to reveal changes.

linonetwo commented 1 year ago

I don't use files directory, I usually use lazy-images setup. So I don't need to look at the folder, just drag everything into wiki, like how others use Notion or Obsidian, I think this make it more easy to use.

pmario commented 1 year ago

https://github.com/tiddly-gittly/HugeWikiExample/tree/master/demo/tiddlers/Files .. This directory with the huge files is a problem!

Drag & drop import into TW using huge files is a problem. .. IMO it does not matter if it is easier to use, when the whole thing is completely unusable.

linonetwo commented 1 year ago

So I always keep eye on lazy loading related things like https://github.com/Jermolene/TiddlyWiki5/pull/7014

Drag thing into wiki will add metadata to them, while files folder may have no?

Anyway, I will comeup with some solution on this in the future.

pmario commented 1 year ago

drag & drop import may make a x.pdf and x.pdf.meta but it does not automatically use _cannonical_uri ... So your files are embedded. ... And since I did download 236MByte it did not work.

So if you want those big files, you can create a "files" directory that is at the same level as the "tiddlers" directory. .. But files have to be stored there manually and not with TW.

Everything below tiddlers/ eg: tiddlers/files will be automatically embedded into TW and therefore cause problems.

I don't know if Saq's file-upload-plugin works with node. I think there are some posts somewhere at talk.tiddlywiki.org

linonetwo commented 1 year ago

I will have a try, auto add _cannonical_uri and meta file all together is what I want.

For this repo, I already remove the PDF (or maybe I should enable lazy-image, but I'm going to sleep, GN)