Framerate slows down over time (20-30 minutes) while parsing.

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run the parser for 20-30 minutes

What is the expected output? What do you see instead?
Final Fantasy XI slows to a crawl. When parsing was stopped, framerate 
immediately went from 1 (bad) back to 30 (to which the game is locked).

What version of the product are you using? On what operating system?
1.1.5 on Windows XP, 2 GHz AMD Athlon64 (3200), 1GB RAM.

Please provide any additional information below.
This felt like a side-effect of using SQL to manage the database. If 
possible I was hoping for an option to adjust the frequency with which the 
parser writes back new records to disk. Perhaps it could cache 100, 1000, 
10k, 100k entries and then write everything to the database at once, or 
even disable writing the entries until parsing is stopped. Even if this 
means that stats displayed during parsing are inaccurate or unavailable, 
that seems better than only being able to run the parser for about 20 mins 
max.

The problem seemed to grow more and more significant over time, implying 
that the size of the database rather than the rate at which data is coming 
in is responsible. Perhaps the database is being indexed in a fashion 
which requires portions of it to be shifted as new data comes in.

Love the project!

Original issue reported on code.google.com by factory....@gmail.com on 21 Oct 2008 at 11:44

[GoogleCodeExporter]

Based on my experiences, you're sort of correct.  Parsing is basically static 
in time
cost, and very fast.  Writing to disk is likewise very low cost, and tends to 
remain
low over time.  What becomes expensive over time is 
reading/aggregating/processing
data.  This is exacerbated when there are many open tabs.

Could you provide a list of which tabs are open while you are parsing when you
experience the performance problem, and if performance is acceptable with most 
of
them closed? (Typically Experience and Offense are the only ones I leave open 
while
parsing, and I keep Experience the primary selected one to avoid repaint costs 
from
Offense (though there have been a number of speed optimizations done for that 
tab as
well).)

Original comment by Kinemati...@gmail.com on 22 Oct 2008 at 3:33

[GoogleCodeExporter]

Closing tabs will reduce this? Excellent, it was my first time running the 
parser so 
I had the default settings, which I suppose means all of them. I'll try cutting 
those down. Can I close them all and then reopen after and have them recompile 
all 
the data after the fact? I don't mind not being able to see the data while 
playing, 
as I said, I just don't want to lose any of it.

If not the case currently, maybe that could be the real option to add - a mode 
which 
just fills the database while parsing and waits until you stop parsing and 
specifically request a report to do aggregation. If that's how it works already 
with 
open/closed tabs, then great! I'll try this ASAP.

Also, I was minimizing the parser in the hopes that repaint wouldn't be 
required, 
but I don't think that's part of the slowdown I was seeing anyway.

Original comment by factory....@gmail.com on 22 Oct 2008 at 6:12

[GoogleCodeExporter]

Yes, each tab can be opened and closed at will, either from the Windows menu or 
using
the tab context menu (right click on a tab to either close it or close all 
other tabs). 

[Can I close them all and then reopen after and have them recompile all 
the data after the fact?]

Tabs are merely a view into the database.  Having them open or closed has no 
effect
on what's saved in the database.  When you open a tab it refreshes its data 
from the
database, so it will always be current.  If a tab is closed then it's taken out 
of
the notification queue while parsing and no code is run, so it won't slow your 
system
down.

Marking this bug as invalid, though performance issues will continue to be 
worked on.

Original comment by Kinemati...@gmail.com on 22 Oct 2008 at 8:06

• Changed state: Invalid