watch list corruption

[GoogleCodeExporter]

on startup, if the watch update process completed successfully, create a
backup of watches.list as watches.list.backup. In the event that the list
would be corrupted, the user could always fix it without too much hassle.
Or even better, Specto could heal itself and restore the backup?

Original issue reported on code.google.com by nekoh...@gmail.com on 11 Jan 2007 at 10:33

[GoogleCodeExporter]

Actually, this might not be the best way. We should "safe write"; Specto should 
never
write to watches.list unless having simulated the process and confirmed that it 
would
work without errors in order to prevent corruption. Would that be possible?

So, maybe we should just harden Specto. Or do both (safe writing + backuping)?

Original comment by nekoh...@gmail.com on 26 Jan 2007 at 10:34

• Added labels: Priority-Medium, Security
• Removed labels: Priority-Low

[GoogleCodeExporter]

Argh. Let's switch to XML in the 0.3 series.

I tried grepping through specto to find suspect writes and only saw them in 
watch.py
and watch_web_static.py, everything seemed normal. So my suspicions are on 
iniparser.

Using XML certainly feels safer than the wobbly iniparser we have currently. 
And if
it still corrupts itself at that time, we will know that our code is the 
problem, and
investigate further from that.

Original comment by nekoh...@gmail.com on 17 Jul 2007 at 3:22

• Changed title: bulletproof XML watch list
• Added labels: Milestone-Release0.3, Priority-Critical
• Removed labels: Priority-Medium

[GoogleCodeExporter]

I would prefer to keep our ini-style, because it looks much better and it is 
easier
to add new watches without opening specto->add_watch, ...

And these watches.list corruptions never happen for me...so i can't see where 
the
corruption is being done.

Does it happen when you quit specto when it is busy updating a watch?

Original comment by woutclym...@gmail.com on 20 Sep 2007 at 9:58

[GoogleCodeExporter]

Nope, does not happen in any particular situation. The only thing is that I 
have lots
of website watches and about two email watches. Total is about 10-20. Could you 
try
using specto with that much watches for various websites (reliable or 
unreliable) for
a few days/weeks? It is really weird that I am the only one experiencing this!

Original comment by nekoh...@gmail.com on 20 Sep 2007 at 9:05

[GoogleCodeExporter]

Changing the title, because this issue was originally really meant for my 
mysterious
self-corrupting watch list problem. This is a proposed (theoretical solution) 
which
may or may not work.

Chris Caravaty in issue 75 describes the problem: 

"occasionally, for no reason after a restart... all my watches will be gone. The
watches.list will contain a few entries in this form:

    [some watch name blah blah]
    filesize = 9235

but thats it, nothing shows up in the specto window. This has happend 4 times 
to me,
after mutiple installs, and multiple re-adding all my watches back to specto... 
i
have no idea why this happens"

I, personnally, had this problem numerous times (I lost count), to the point 
where I
keep a backup of my watches.list file. It is really killing me. Could not 
attribute
this to network problems or startup problems of any kind. I tried reproducing it
without success (refreshing watches frantically, removing/adding stuff, 
renaming, etc)

Original comment by nekoh...@gmail.com on 18 Oct 2007 at 3:50

• Changed title: watch list corruption, need bulletproof XML watch list?
• Added labels: Type-Defect
• Removed labels: Type-Enhancement

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

COMMENTED IN CLOSED ISSUE 75:

Comment 2 by adamorjames (http://code.google.com/u/adamorjames/), (sometime in 
the past)

I experience the problem as well. So that is three people.

Original comment by chris.ca...@gmail.com on 18 Oct 2007 at 4:02

[GoogleCodeExporter]

so the issue, to claify for myself: is that watches.list looses the 'watch
information' (error_margin,type,refresh,uri)  and just displays [name blah] & 
filesize

this only happens after restarting or an inopertune shudown.. i dont think it 
can be
reproduce by refreshing/adding/or anything else...

we should start the search by looking through the source for when things are 
written
to watches.list  b/c somewhere, this goes wrong and not all the information we 
want
is written.

also why is the 4th data entry that i mentioned above URI and not URL... i'm 
thinking
this is typo somewhere.

Original comment by chris.ca...@gmail.com on 18 Oct 2007 at 4:13

[GoogleCodeExporter]

an additional note of interest: the corrupted list of:

[watch name blah]
filesize=12345

[watch name blah blah]
filesize=678910

.
.
.

did not contain all of my watches, a few of the ones towards the end/bottom 
were missing
so it appears... that maybe, while checking a watch, the program crashed/webpage
couldn't be found/etc..., and specto wrote the information that it had found so 
far
to the watch list, without appending the critical 'watch information'
(error_margin,type,refresh,uri)

Original comment by chris.ca...@gmail.com on 18 Oct 2007 at 4:24

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

also, i only had website watches in my list.  

Original comment by chris.ca...@gmail.com on 18 Oct 2007 at 4:45

[GoogleCodeExporter]

note that I had this issue in the specto 0.2 series, and the 0.1 as well, so it 
is
not tied to a particular version. Reinstalling from the ubuntu packages will (I
think) change nothing on your side, you just didn't happen to stumble on the bug
again for now :(

Original comment by nekoh...@gmail.com on 18 Oct 2007 at 5:04

[GoogleCodeExporter]

This is a very strange bug...and i am not sure if it will be fixed if we use an
xml-parser (has anybody some experience with ElementTree?) because it looks 
like the
data is lost while writing or just before writing the options...if somebody 
finds out
how to reproduce this bug, let us know, it would make fixing a lot easier!

Original comment by woutclym...@gmail.com on 18 Oct 2007 at 12:18

[GoogleCodeExporter]

Ok, after looking at the code again and again, i think i found a solution...
Why does the web watch create a cache file and doesn't use it?
watches.list should only be used for important watch information, not as a 
cache file.
So the best solution i see is to use the cache file again and write to 
watches.list
only necessary information!

Original comment by woutclym...@gmail.com on 18 Oct 2007 at 12:39

[GoogleCodeExporter]

okay just happened again...
i added a new watch to the list
then i tried to refresh all my watches
it hung somehwere along the way before a website couldn't be reached
after the next restart
all the watches except the new one were gone.

where in the web_watch file are you seeing this woutclymans?

Original comment by chris.ca...@gmail.com on 19 Oct 2007 at 6:20

[GoogleCodeExporter]

here is where the cache file is created:
cacheSubDir__ = os.environ['HOME'] + "/.specto/cache/"
if not os.path.exists(cacheSubDir__):
    os.mkdir(cacheSubDir__)

but it is not used for the filesize:
    def write_filesize(self):
        """ Write the filesize in the watch list. """
        self.new_values = {}
        self.new_values['name'] = self.name
        self.new_values['filesize'] = self.to_be_stored_filesize
        self.specto.watch_io.write_options(self.new_values)

I think this is the reason why the watches.list file gets corrupted...so it 
would be
better if we also write the filesize in the cache file. there is no reason to 
write
it in the watches.list file.

Original comment by woutclym...@gmail.com on 22 Oct 2007 at 6:33

[GoogleCodeExporter]

Makes sense, it could be possible that this is the cause of the problem. We'd 
need to
try putting the filesize in the cache file and see if there are any regressions 
in
accuracy. There are two reasons I can see that may have caused this mess. 
Either:
- putting the filesize in the cache file was a recursive paradoxal thing that 
messed
itself up?
- or I simply lacked the skill to do it properly at that time (very possible)

Original comment by nekoh...@gmail.com on 25 Oct 2007 at 2:25

[GoogleCodeExporter]

I am having the watches.list corruption as well.  I have two ideas about the 
cause here:

I know that at least one of the sites on my watch list will periodically be
unresponsive for an hour or so at a time.  Would that cause corruption?

The other thing is that sometimes my setup (router, cable modem, something) 
will fail
to be granted a new IP address after the lease runs out on my old one and I 
will have
to manually release it and renew it.  If Specto wasn't able to access the 
internet at
all, would that cause corruption?  Hmmm...

Here is a sample of what my watches.list looks like after corruption:

[Sunflow]
filesize = 12981

[Gobolinux]
filesize = 5543

[Blendernation]
filesize = 10152
updated = False
last_updated = Wednesday 09 Jan 2008 02:43

[Relient K]
filesize = 22223
updated = False
last_updated = Thursday 10 Jan 2008 20:16

I also only have web page watches, and there are a total of 23 on my watch 
list. 
Hope this helps!

Original comment by JoshFaul...@gmail.com on 12 Jan 2008 at 6:46

[GoogleCodeExporter]

for the record, I also have a feeling that this has happened to me often after a
gnome logout (while specto was running). But that may be totally unrelated.

Original comment by nekoh...@gmail.com on 26 Mar 2008 at 1:43

[GoogleCodeExporter]

Maybe it could help if we sent *all* write operations to the logger? "I will now
write these values to the watch list", foo, bar

Combine that with the super traceback (issue 99) so we know which line asks for 
the
write, and maybe we could figure out where the problem comes from.

Or does someone know of other methods of investigation?

Original comment by nekoh...@gmail.com on 26 Mar 2008 at 6:34

[GoogleCodeExporter]

I think the problem happens when specto wants to write the filesize to 
watches.list
so i added another cache file that contains the filesize from the webpage.

I really hope this will fix our problem!

Original comment by woutclym...@gmail.com on 27 Mar 2008 at 6:28

• Changed state: Started

[GoogleCodeExporter]

For your information, Wout made a change in his bazaar branch, which puts the
filesize in a separate file instead of the watch list.

Therefore, if anyone would like to run the specto-woutc branch for testing, this
would be pretty useful. I've been running it for a few weeks, problems as of 
yet, but
we would need to be more than 1 to confirm if the bug vanished because of this.

Original comment by nekoh...@gmail.com on 11 Apr 2008 at 6:49

[GoogleCodeExporter]

Perhaps it would be better to update your existing data via a temporary file, 
replacing the original file only once the transaction is complete, and data 
verified?

Original comment by Kalama...@gmail.com on 16 Apr 2008 at 11:09

[GoogleCodeExporter]

oops, sorry, in my previous comment I meant "no problems as of yet".

> Perhaps it would be better to update your existing data via a temporary file
I think that's what wout's change does (it saves the filesize in a second cache 
file
instead of in the watch list)? As I said though, this needs testing. I did not 
see
the corruption show up lately, but it is not a proof until it has been long 
enough
and multiple users confirm it I guess.

Original comment by nekoh...@gmail.com on 17 Apr 2008 at 12:08

[GoogleCodeExporter]

interesting behaviour :
yesterday i've lost all my watches in Specto 0.2.2
i had no backup therefore i had to manually add all my former watches
I used to have a Gmail and a Rhythmbox watches but i couldn't recreate them and 
i
finally had to name them gmail and rhythmbox (with little G and R)
Later, some of my old watches were back and i had both gmail and Gmail and 
Rhythmbox
and rhythmbox
i think that's the reason why i could name them like before : they hadn't 
totally
disappeared 
Maybe that can help to track this bug

Original comment by thibaut....@gmail.com on 1 Jul 2008 at 7:59

[GoogleCodeExporter]

Here guys,
I wanna work on some XML here.

I just wrote a dev page on the wiki : 
http://code.google.com/p/specto/wiki/XMLDevIdeas
I don't want to putsch iniparser just yet, but I have created a branch based on
woutc's dev branch (didn't want to be out of touch).

Give all your ideas right on the wiki page, and we'll try to make it just right!

Cheers
Pascal Potvin

Original comment by pascal.p...@gmail.com on 19 Dec 2008 at 2:29

[GoogleCodeExporter]

Well, I have been using the 0.3 series for exactly a year now, and I haven't
experienced corruption again. If we need an XML, that should be filed as a 
separate
issue.

Original comment by nekoh...@gmail.com on 20 Mar 2009 at 11:40

• Changed title: watch list corruption
• Changed state: Verified

[GoogleCodeExporter]

Issue 75 has been merged into this issue.

Original comment by nekoh...@gmail.com on 16 Apr 2011 at 3:20

[GoogleCodeExporter]

Issue 144 has been merged into this issue.

Original comment by nekoh...@gmail.com on 16 Apr 2011 at 3:26