search

techknowlogick / shiori

[Archived] Shiori is a simple bookmark manager. Upstream is now being supported again
https://github.com/go-shiori/shiori
MIT License
78 stars 9 forks source link

http: superfluous response.WriteHeader call - Performance on database with 10000 bookmarks #144

Open m040601 opened 5 years ago

m040601 commented 5 years ago

I'm running your shiori nightly, march 10, testing with big databases. I started with 500, 1000 bookmarks. I'm reaching now the 10000 bookmarks level. The import process, or printing them on the console is not the problem.It takes some effort but shiori bravely manages it in the end. (see also my comments on, https://github.com/RadhiFadlillah/shiori/issues/125 , ADDITIONAL ) for more)

With the browser it's another story. It is where it gets sluggish. Not the browser performance itself. Firefox doesnt' freeze, keeps feeling snappy, and the cpu usage for the browser doesnt spike. But:

When logging in the first time:

Sometimes you get to to see the bookmarks listed:

It is when you either:

My (non developer) guess. It cant be sqlite slowing things down. I tried opening and querying the shiori.db sqlite database with other cli tools (visidata, sqlite_web, etc...) and it responded well.

Linux 5.0.0-arch1-1-ARCH #1 SMP PREEMPT x86_64 GNU/Linux Athlon dual core, 8 Giga RAM, ssd disk Sqlite version 3.27.2 Dont have golang installed (using the precompiled shiori)

m040601 commented 5 years ago

In addition I noticed that this does not happen with the original Radhi shiori v1.0. There seems to be some introduced change in your shiori v6.1 and v6.2 causing this.

I tested this with a 20000 bookmarks file. The file was previously checked and cleaned for illegal characters etc.. The import suceeds on the command line. There is no without breaking or any serious error. But the slugishness and error messages in the browser is there.

The original Radhi shiori v1.0, does not have this problem.

m040601 commented 5 years ago

Also, it is very much worth having a peak at the Buku bookmarks manager, https://github.com/jarun/Buku . Although it is written in python, it also uses an sqlite database.I had no issues with the speed and response of the sqlite database with the same 20000 bookmarks file. Same for the browser user interface, bukuserver.

One interesting example. When importing, they automatically add a tag with the "domain" part of the url. Very usefull. Example, you're importing lots of articles that belong to wikipedia or some big news site. You end up with them tagged "en.wikipedia.org" and "nytimes.com" (just like, https://news.ycombinator.com/best or reddit)

Of course Buku is diferent from shiori. Its focus is on heavy bookmarks management (slicing and dicing, querying/filtering tags etc), not readability and caching. Although they are different projects they operate on the same web "currency". Your precious internet links. It would be awesome if one could be sure to easly import/export, pipe the 2 together, have compatible database schema etc., without any major hurdles or incompatiblities. One more big pipe to rule them all ...

m040601 commented 5 years ago

I have now tested with the latest release v7. Both problem are still there:

I've tryied importing different files with 10,100 or 10000 or 20000 bookmarks. I also have the impression that for large numbers of bookmarks, the performance of v7 is worse than v6.

m040601 commented 5 years ago

Another anomaly in the user interface is the "existing tags" popup window. It gets shrunken and squeezed. It doest maximize to display all available tags. Maybe it's width is normal, but it's height is barely 3 lines. So that you have to scroll a lot to see all tags.

https://imgur.com/a/wsZCtFZ

This already happened in v 0.6.2. This does not happen in the original Radhi/shiori