Keepnote´s startup is very slow

[GoogleCodeExporter]

Why keepnote´s "indexing notebook" procedure is so slow?

Original issue reported on code.google.com by billath...@gmail.com on 8 Jun 2011 at 10:30

[GoogleCodeExporter]

How long does it take for you?  How many notes are in your notebook?  Also 
which operating system do you use?

The indexing should only happen when you upgrade keepnote or if you tamper with 
files outside of keepnote.  I am also testing a new version of keepnote 0.7.3:
http://keepnote.org/keepnote/index.shtml#download-test

You can see if indexing is working faster for you now.  I have working to 
improve this.  

Original comment by matt.rasmus@gmail.com on 13 Jun 2011 at 1:38

• Added labels: ui

[GoogleCodeExporter]

The "Loading note" window takes too time on the screen... too time to load a 
collection of notes. 

Original comment by billath...@gmail.com on 17 Jun 2011 at 2:24

[GoogleCodeExporter]

How many notes do you have?  How long does openning a notebook take? (1-3 
seconds or >20 seconds)

Original comment by matt.rasmus@gmail.com on 18 Jun 2011 at 4:44

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

0.7.3 is still much slower than 0.7.1.  20 secs to open 200 notes. 

Original comment by rich.tr...@gmail.com on 19 Jun 2011 at 5:35

[GoogleCodeExporter]

rich,  What operating system are you on?  Are you using any network mounted 
hard drives?  Are there any errors in the error log that correspond to opening 
the notebook?

Original comment by matt.rasmus@gmail.com on 19 Jun 2011 at 5:37

[GoogleCodeExporter]

... more ... 0.7.1 opens immediately but even with 0.7.1, re-indexing 200 
notes, 430K of text still takes 5 minutes.  Seems too long. 

Original comment by rich.tr...@gmail.com on 19 Jun 2011 at 5:44

[GoogleCodeExporter]

Are you using Windows?  or Linux?

Original comment by matt.rasmus@gmail.com on 19 Jun 2011 at 7:43

[GoogleCodeExporter]

Windows 7

Original comment by rich.tr...@gmail.com on 20 Jun 2011 at 2:12

[GoogleCodeExporter]

Hmm... the only difference should be that I check mtimes on the files within 
the notebook now in order to detect tampering outside of the keepnote 
application.  I wonder if reading file mtimes is slow on windows compared to 
linux.  I will look into this idea and let you know what I find.

Original comment by matt.rasmus@gmail.com on 21 Jun 2011 at 2:43

[GoogleCodeExporter]

Issue 469 has been merged into this issue.

Original comment by matt.rasmus@gmail.com on 21 Jun 2011 at 2:46

[GoogleCodeExporter]

For me it takes 30 seconds to open a 50 notes notebook on Windows 7 x64.

Original comment by jorgemor...@gmail.com on 29 Jun 2011 at 5:11

[GoogleCodeExporter]

Issue 499 has been merged into this issue.

Original comment by matt.rasmus@gmail.com on 4 Jul 2011 at 1:47

[GoogleCodeExporter]

Ok.  I've tried to reproduce this slow down on my windows machine (which is 
XP), but I haven't been able to observe it yet.  So perhaps its a windows 7 
specific issue.  I have uploaded a new testing version of keepnote 0.7.4 that 
disables my check for mtimes (modification times) of notebook files.  This is 
my best guess as to what has changed between 0.7 and 0.7.3 that could cause a 
slow start time.  Please let me know if the new testing version is fast again.  
Thanks!
http://keepnote.org/keepnote/#download-test

Original comment by matt.rasmus@gmail.com on 4 Jul 2011 at 1:51

• Added labels: notebook

[GoogleCodeExporter]

I'm on Win7 x64. NB has 200 notes, 500K.
It still takes 3+ minutes to re-index.  Start-up time is about 15-20 secs, 
where 071 was essentially no time. So I'd say 073 and 074 are about the same 
for me. 

Thanks for trying to resolve this.

Original comment by rich.tr...@gmail.com on 4 Jul 2011 at 2:33

[GoogleCodeExporter]

Hmm...

I just tried uploading another testing version of 0.7.4 to see if its the 
actually dialog GUI process thats slowing everything down (note: I upload 
testing version 'in-place', so you'll need to download it again from the same 
link).  If you have time please try it and let me know if this improves loading:
http://keepnote.org/keepnote/#download-test

Here are some of my thoughts about what might be going on:

Reindexing should only happen when you update between versions of keepnote or 
when you do "Tools > Update Notebook Index".  Reindexing will always be an 
"expensive" thing and should rarely need to be done.  The "Tools > Update 
Notebook Index" is only needed in rare cases where someone corrupted their 
index and needs to rebuild it manually, or if you tamper with the notebook 
files outside of keepnote and you want to give keepnote a chance to see what 
changed.

However, opening a notebook any other time should not take long and I can't 
think of what would be different between 0.7.1 and 0.7.4 (with mtime disabled) 
that could cause a slowdown on windows 7.

To give you an idea, I have ~2,500 notes and 400Mb of data.  The "Load 
notebook..." window appears for less than a second for me most of the time.  

Here are some more ideas I have in case the testing version does improve speed:

Are you using "Open Default Notebook" option?  Perhaps the program loading time 
is the slow part and actual notebook opening is fast.  Try disabling default 
notebook opening to see which step is the actual slow step.

Are there any usually errors that occur in the error log "Help > Show Error 
Log".  Maybe a misbehaving extension or other errors.

Also opening a notebook from a filesystem that hasn't been read from in a while 
will be slower since none of the files will be in cache.  Network mounted 
notebooks could also be slow (but not much different between 0.7.1 and 0.7.4).

Are you using lots of tabs?  Try using one tab and see if the notebook loading 
is much faster, this would give me ideas on what to optimize.

Another idea is that inter-process communication (IPC) is somehow slower in 
0.7.4.  You can run keepnote with the command line option 

  keepnote --newproc

to disable IPC.  You can add such command line options by editing the 
properties of the shortcut or by running keepnote from the command line.

Thanks,
Matt

Original comment by matt.rasmus@gmail.com on 4 Jul 2011 at 3:16

[GoogleCodeExporter]

Hi Matt,

I just tested the latest 0.7.4 Version. Before updating I had a notebook 
loadtime of 24 seconds total. With the latest update program load time is about 
2 seconds and 1 more second for the notebook file. Seems like you hit it.

Thx
Nikolaus

Original comment by n...@bomhard.de on 4 Jul 2011 at 3:32

[GoogleCodeExporter]

Whatever you changed, 074 has 071 speed for me.  Great!
-------------------------------------------------------

FYI:  I have a few NB.  I made shortcuts for each one and put the appropriate 
NB path on each shortcut.  All are on local SATA disks. Default NB is off for 
me. 

PS: I only mention re-indexing time, because It seems so long.  3 Min for a 
small NB.  I don't re-index on any regular basis.

This is the end of my error log from last run
==============================================
KeepNote 0.7.4: Mon Jul 04 11:37:13 2011
enabling extension 'backup_tar'
enabling extension 'new_file'
enabling extension 'editor_insert_date'
enabling extension 'command_basics'
enabling extension 'export_html'
enabling extension 'python_prompt'
enabling extension 'keepnote'
node e3c0e8f7-aa14-4f9b-9722-bd343d55ca20 not foundnode 
e3c0e8f7-aa14-4f9b-9722-bd343d55ca20 not foundTraceback (most recent call last):
  File "keepnote\gui\tabbed_viewer.pyc", line 638, in <lambda>
  File "keepnote\gui\tabbed_viewer.pyc", line 627, in highlight
AttributeError: 'gtk.EventBox' object has no attribute 'get_state'
Traceback (most recent call last):
  File "keepnote\gui\tabbed_viewer.pyc", line 641, in <lambda>
  File "keepnote\gui\tabbed_viewer.pyc", line 627, in highlight
AttributeError: 'gtk.EventBox' object has no attribute 'get_state'

Original comment by rich.tr...@gmail.com on 4 Jul 2011 at 3:39

[GoogleCodeExporter]

Ok. Cool.  I have uploaded yet another new version keepnote 0.7.4 (in place 
again, so you'll need to re-download).  
http://keepnote.org/keepnote/#download-test

I have re-enabled the mtime checks (my first change) since I think it was my 
second change (threading issues with the open notebook dialog box) that 
probably made the difference.

Just let me know if it slows down again or is still fast.

Thanks again for your patience and feedback,
Matt

Original comment by matt.rasmus@gmail.com on 4 Jul 2011 at 4:14

[GoogleCodeExporter]

The latest 0.7.4 is as fast as the previous one (3 secs total startup). Must 
have been the threading issues.

Thanks for your great work
Nikolaus

Original comment by n...@bomhard.de on 4 Jul 2011 at 4:49

[GoogleCodeExporter]

Hi Matt, FWIW the changes you made have the same effect on Win XP: startup 
times are down to acceptable values again (few seconds)

Are you sure that you re-enabled the mtime checks in the last version? I 
downloaded two versions today, both showing the mtime debug modifications (at 
least in the tar.gz file) The only difference between them is the time-stamp of 
the PKG-INFO file. The executables I downloaded with them are considerably 
different.

Thanks,
Ton.

Original comment by epposan@gmail.com on 4 Jul 2011 at 6:07

[GoogleCodeExporter]

Grr... I think you're right Ton.  I'm out of town at the moment so it's tougher 
for me to do these uploads correctly.  

But, I have uploaded again the testing version 0.7.4 and this time mtime checks 
should be re-enabled.  Let's really see if loading times are still fast.
http://keepnote.org/keepnote/#download-test

Thanks again everyone,
Matt

Original comment by matt.rasmus@gmail.com on 4 Jul 2011 at 6:17

[GoogleCodeExporter]

Much better (the tar.gz reflects the intentions now :-)

load time is still excellent: < 2 secs
reindexing time for the same notebook: 5 mins 30 secs

Thanks,
Ton.

Original comment by epposan@gmail.com on 4 Jul 2011 at 7:07

[GoogleCodeExporter]

Great!

My guess is that indexing might be slowed down by GUI threading issues as well. 
 I'll look into that in a future release.

Matt

Original comment by matt.rasmus@gmail.com on 4 Jul 2011 at 7:10

• Changed state: Started

[GoogleCodeExporter]

That would be cool. I was wondering for some time/versions already what was 
keeping Keepnote busy: my cpu is usually idling around 98% during loading 
and/or reindexing

Ton.

Original comment by epposan@gmail.com on 4 Jul 2011 at 7:21

[GoogleCodeExporter]

Matt,

the latest 0.7.4 with mtime checks enabled is very slow again on my machine.

Nikolaus

Original comment by n...@bomhard.de on 5 Jul 2011 at 7:18

[GoogleCodeExporter]

Nikolaus,

Is it slow every time or just the first time with the mtimes re-enabled?

Matt

Original comment by matt.rasmus@gmail.com on 5 Jul 2011 at 10:36

[GoogleCodeExporter]

Matt,

when I installed the latest version it was slow on every startup. So i 
reinstalled the prvious version. To answer your question I reinstalled the 
latest version and it was slow, but only one time. Then it started up very 
quickly, a little later it was slow again but not that slow. Since that until 
now (with a reboot and suspend in between) it is fast on every startup. Maybe I 
should completely uninstall to grant a consistent behaviour and answer.

Let me know what you want me to test.

Nikolaus

Original comment by n...@bomhard.de on 5 Jul 2011 at 8:21

[GoogleCodeExporter]

Hi Nikolaus,

Thanks for doing this testing for me.  It is very helpful.

What I would like to know is if using the very latest testing version of 
keepnote 0.7.4, whether notebook loading is fast.  Note, that it might be slow 
on the first load of the notebook when using the latest version of 0.7.4 with 
mtime checking.

When keepnote has the mtime checking feature enabled (which is the version 
available right now on the website), keepnote reindexes any note that has a 
newer modification time (mtime) than when it last indexed it.  Ideally, 
checking mtimes is fast and doing reindexing is slow (unavoidably).  By 
checking mtimes, I can be more confident that the index is up to date without 
spending the time reindexing all the notes all the time (like every read).  
However, the question I have is whether my assumption of fast checking of 
mtimes is really true on windows 7.

When you run keepnote 0.7.4 with mtimes enabled on a notebook that was last 
used by keepnote with mtimes disabled, you might experience a slow first load.  
This is because keepnote thinks every note is too "new" and needs to be 
reindexed (a relatively slow operation).  Now if this is the case, then you 
should see messages in the error log saying that notes are being reindexed.  If 
no messages appear in the error and notebook loading is still slow, then that 
means checking mtimes is slow on windows 7.

So let me know whether loading a notebook is always slow (or only the first 
load) and send me your error log.

Thanks,
Matt

Original comment by matt.rasmus@gmail.com on 5 Jul 2011 at 9:25

[GoogleCodeExporter]

Is someone able to confirm that the latest testing version of keepnote 0.7.4 is 
still fast for loading notebooks on windows 7 (or other windows OS)?
http://keepnote.org/keepnote/#download-test

Specifically, is it fast to open a notebook the second time or later (the first 
time opening a notebook with 0.7.4 might be slow).

Original comment by matt.rasmus@gmail.com on 7 Jul 2011 at 12:48

[GoogleCodeExporter]

074 Speed still good for me - Thx

Original comment by rich.tr...@gmail.com on 7 Jul 2011 at 1:21

[GoogleCodeExporter]

Still fine on XP SP3.

It seems to always re-indexe the top-level node at start-up, even when nothing 
was changed. I guess this is the result of the auto-save activity.

Off topic, but one GtkWarning is appearing in my error-log.txt file 
consistently: C:\Program Files\Keepnote\keepnote.exe:270: GtkWarning: 
gtk_widget_size_allocate(): attempt to allocate widget with width -19 and 
height 10

Thanks, Ton.

Original comment by epposan@gmail.com on 7 Jul 2011 at 2:11

[GoogleCodeExporter]

Search all notes (Ctrl+K) doesn't seem to do anything (no dialogue appears)
I'm using 0.7.4 0n ubuntu 10.04

Apologies if this is a known issue

Original comment by sysyphus...@googlemail.com on 25 Aug 2011 at 11:46

[GoogleCodeExporter]

Does your cursor move to the search box in the upper right?  From there you can 
do full text search.  Does this FAQ apply to your situation?
http://keepnote.org/faq.shtml#no_search_box

Original comment by matt.rasmus@gmail.com on 31 Aug 2011 at 8:44

• Changed state: Done

[GoogleCodeExporter]

thanks Matt
sorry to waste your time

Original comment by sysyphus...@googlemail.com on 1 Sep 2011 at 6:03

[GoogleCodeExporter]

no problem.  Thanks for submitting a report.

Original comment by matt.rasmus@gmail.com on 1 Sep 2011 at 6:13

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

For me, 0.7.7 was a step backward as far as speed of opening a notebook.  Not 
as bad as the 0.7.2 problem.  But I dropped back to 0.7.5 for now. 

Original comment by rich.tr...@gmail.com on 20 Jan 2012 at 8:33

[GoogleCodeExporter]

I have a guess why this might be.  I added automatic VACUUM commands to the 
sqlite database on startup.  I will make a version without this and make VACUUM 
commands an optional manual step in "Tools > Compact Notebook Index".  I will 
post here again when I have a new version for you to test.  Thanks for the 
report.

Original comment by matt.rasmus@gmail.com on 21 Jan 2012 at 4:19

[GoogleCodeExporter]

Matt, does keepnote-0.7.8 have a change for #39 to check?

Original comment by rich.tr...@gmail.com on 12 Mar 2012 at 12:59

[GoogleCodeExporter]

0.7.8 still seems slow to open on Win7

Original comment by rich.tr...@gmail.com on 31 Mar 2012 at 7:28