search

aardappel / procrastitracker

a Windows time tracking application
http://strlen.com/procrastitracker/
483 stars 53 forks source link

The database is corrupt. #67

Open mrexodia opened 3 years ago

mrexodia commented 3 years ago

When restarting Windows 10 2004 I think something goes wrong with handling of the shutdown/restart event.

I get the following message on startup:

The database is corrupt. Please restore the last backup (ProcrastiTracker makes these frequently), and try again. See Procrastitracker start menu for database location. Exiting program..

When asked if I want to restore the database and click 'Yes' it fails and when I go to the %appdata% folder I see:

image

aardappel commented 3 years ago

Not sure why it corrupted, possibly it restarted while in the middle of writing.

Rename db.PT to db_corrupt.PT, and then db_TEMP.~PT to db.PT, then run again.

If it still doesn't work, try renaming db_BACKUP_2020_11_12.PT to db.PT. Sadly in that case you'll lose a week of stats.

mrexodia commented 3 years ago

Thanks! I restored the database already, but this has happened on every single restart for me. That’s why I opened an issue :)

On Fri, 20 Nov 2020 at 05:05, Wouter van Oortmerssen < notifications@github.com> wrote:

Not sure why it corrupted, possibly it restarted while in the middle of writing.

Rename db.PT to db_corrupt.PT, and then db_TEMP.~PT to db.PT, then run again.

If it still doesn't work, try renaming db_BACKUP_2020_11_12.PT to db.PT. Sadly in that case you'll lose a week of stats.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aardappel/procrastitracker/issues/67#issuecomment-730831224, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASYFGLSF2DI5OTP6FHQFUDSQXTH3ANCNFSM4T4HI4RA .

aardappel commented 3 years ago

Hmm.. I have no idea how that is possible. It saves the DB every 10 minutes. So no restart pattern should be able to interfere with that every time I'd think.

mrexodia commented 3 years ago

I guess I just got unlucky multiple times in a row then 🤷‍♂️. I do not reboot often at all and every time I tested I reproduced the issue, but when I tested again today everything was working fine again.

There is a dialog that offers to recover the database and that reproducibly fails to restore the database though (likely because it's trying to restore that 0 byte .TMP file instead of the latest backup).

mrexodia commented 3 years ago

The issue kept happening unfortunately and I discovered that the application silently dies in the background when leaving the PC on for a longer period of time. I couldn't find any crash dumps but I will write some application to monitor the state over time to try and narrow down the issue.

In every case the database ends up being corrupted unfortunately 🙁

aardappel commented 3 years ago

Hmm, that sounds like something is causing it to crash while its writing the DB specifically.

I could add some functionality to add a crash log, and/or send you a debug version of the program?

mrexodia commented 3 years ago

Hello,

I’ll compile it myself, but first I have to figure out why Windows is not logging crashes and crash dumps in the event viewer. The weird thing is that I have a just in time debugger set up and somehow the program managed to crash without popping that up 🤷🏻‍♂️

I’ve had a similar issue with the RTX Voice application and a bunch of other weird things are happening as well, but I’ll try to narrow it down in the coming weeks :)

On Fri, 27 Nov 2020 at 06:05, Wouter van Oortmerssen < notifications@github.com> wrote:

Hmm, that sounds like something is causing it to crash while its writing the DB specifically.

I could add some functionality to add a crash log, and/or send you a debug version of the program?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aardappel/procrastitracker/issues/67#issuecomment-734645877, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASYFGK3HLXSD32725CWYH3SR4XQRANCNFSM4T4HI4RA .

aardappel commented 3 years ago

Oh if you can compile it yourself you can run it from VS (F5), so you could get the debugger to hit, that be even more valuable of course (see both a stack trace and variables).

When it "silently dies", the PC keeps running? no reboot?

I don't think programs do crash dumps unless they are programmed to do so, same with adding meaningful stuff to the event viewer.

Not sure what you mean by "just in time debugger". recommend running it under VS.

What I mean with "I could add some functionality to add a crash log" is code that catches exceptions and writes a stack traces to a log file as it crashes. That is mainly useful when NOT running under the debugger though.

mrexodia commented 3 years ago

When it "silently dies", the PC keeps running? no reboot?

Yeah my PC keeps running, but it appears as though sometimes random applications are terminated (and apparently sometimes corrupting data).

You can configure Windows to automatically save crash dumps for specific applications: https://docs.microsoft.com/en-us/windows/win32/wer/collecting-user-mode-dumps

The just-in-time debugger is an application that gets called by werfault.exe if an application crashes. Visual Studio itself can be configured as such (but I use JitMagic to allow me to choose a debugger). Here you can find more information: https://docs.microsoft.com/en-us/visualstudio/debugger/debug-using-the-just-in-time-debugger?view=vs-2019

aardappel commented 3 years ago

Ah yes, this "just in time debugging" I do have on, didn't know it was called that :)

Either way, let me know if you need any help from my side tracking this down.

aburke20 commented 3 years ago

I'm actually having a very similar issue.

I'm away from home and using a laptop that every week or so crashes to BSOD.

The issue is that it somehow "corrupts" every single "backup" that Procrasti supposedly made. I get that it might corrupt the latest one, but how do none of them work? It's making me start from scratch every time the laptop crashes.

Wouldn't that be an issue with the way it is writing the backups, since they get corrupt where they are new or old, being written or written days ago ?

Any ideas as to a solution ?

mrexodia commented 3 years ago

You can search in the files and find the backup that isn’t 0 bytes in size, if you rename that to “db” it will work.

I have reproduced the issue multiple times now when your pc is getting low on memory and then you reboot it (perhaps try it on a vm). Currently the program is unfortunately back around September 11 because every other data backup got corrupted :(

I’m really not sure what’s causing this, but perhaps the best way is to backup every second and just keep the last ~10 backups and attempt to restore them one by one until it works.

Obviously this is an edge case, but something is triggering this every single time for me (RTX Voice, looking at you) and it makes it corrupt.

I enabled crash dumps but they don’t work at all for some reason. My windows appears to be in low memory and that messes up everything.

On Sun, 13 Dec 2020 at 00:38, aburke20 notifications@github.com wrote:

I'm actually having a very similar issue.

I'm away from home and using a laptop that every week or so crashes to BSOD.

The issue is that it somehow "corrupts" every single "backup" that Procrasti supposedly made. I get that it might corrupt the latest one, but how do none of them work? It's making me start from scratch every time the laptop crashes.

Wouldn't that be an issue with the way it is writing the backups, since they get corrupt where they are new or old, being written or written days ago ?

Any ideas as to a solution ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aardappel/procrastitracker/issues/67#issuecomment-743920688, or unsubscribe https://github.com/notifications/unsubscribe-auth/AASYFGKLNLTPCRHX7ZMPPG3SUP5HFANCNFSM4T4HI4RA .

aardappel commented 3 years ago

The way it saves is that it first writes to db_TEMP.~PT. If it would crash during writing, then only this file should ever be corrupt, and all other files are not touched.

Only once this file has succesfully been written, does it:

As you can see, that is a very "safe" sequence, exactly intended to avoid any file problems.

Also, the copy/delete/rename actions are the built-in Windows routines, not anything custom, i.e. it should have the same effect as when using these commands from the command-line for example.

So the fact that db.PT is 0 bytes in @mrexodia's screenshot is thus quite weird: what would have needed to happen for this is one of these two:

I do see that the actual writing of the bytes does no error checking beyond seeing if the file can be opened. That should normally not matter, as it typically only fails if the disk is full or other rare circumstances. But I could add it just to help tracking these issues down.

https://github.com/aardappel/procrastitracker/blob/acc93cc3fff397632540568b9ddbdb80b109cf87/src/nodedb.h#L2-L37

aardappel commented 3 years ago

Ok, I added stricter error checking on writing: https://github.com/aardappel/procrastitracker/commit/574bb8d398744d773ebd079a14fa42e83d34e68c This should help with tracking these problems down, and should avoid the file being copied if there was any problem writing the file.

You'll now get one of these 3 errors:

aburke20 commented 3 years ago

Thanks so much for all this. It doesn't seem to still not be telling me the specific reason for the error (I'm seeing the same message as always).

Also, I don't know if it's "intentional" or not, but anytime I try to merge a backup, and it turns out that the backup is corrupt, the app crashes ... so basically I'm having to: try to merge a backup, have it tell me it's corrupt, then app crashes, then open the app again. Rinse and repeat as I try out all the backups.

Here are my last backups (I don't think there is any personal info in here, right ?), in case it can help you figure out what is breaking on my end.

https://pixeldrain.com/u/E3KB5wg9

Again, thanks so much for the effort.

edit: also it says "merge", but does that actually merge two sets on databases, or is it just a "replace current database with another one chosen from the list of backups" ?

aardappel commented 3 years ago

@aburke20 what is the same message as always, and when do you see it?

Rather than merge a backup, the intention is that you rename a backup to db.PT and it will just try to load that. Merge should in theory also work, but it is a more complex operation. Not sure why that would crash. You're basically in a running PT that has an empty database and you're trying to merge and it fails? I will try to see if I can repro that.

I will also try with your files, thanks.

Yes, merge actually merges, not replace.

aburke20 commented 3 years ago

-Same: "The database is corrupt. Last available backup: XX/XX/XX. Do you want to restore it ?"

Select NO (since I know what that last backup is, and it isn't the last one: which would have way more tracking data)

-Then: "The database is corrupt. Please restore the last backup. Procrasti makes these frequently ... yada yada yada"

App closes, and I see now it was saying it would close, so it's not a crash (sorry for confusion)

-So no special distinction as to why it was corrupt (which I think is what your latest update was trying to do (unless I'm mistaken).

-Basically right now I would love to be able to merge the data (from what I gave you) from the db.PT, and the one labeled from 11/25, since both of those has the largest data set of the last month or so. If not, then my 11/25 is just fine, and I have gotten that one to work. Other working ones are like 12/08, 12/06, 12/04 ... I think.

Again, no idea why those specific dates are able to be renamed and used as the core db.PT, but not the others ... all the "corrupt days" might be when Windows crashed ?

I'm so sorry for all this, I know I'm basically asking you to help me get your program to work "better", by somehow not failing despite the root initiating cause being my entire laptop crashing to BSOD.

aardappel commented 3 years ago

@aburke20 I so desperately want to play that game too ;) <- as you can tell, I can read your db.PT file fine.

I was also able to load your largest backup file by renaming it to db.PT.

I did discover there was an actual bug in the merge function which I just fixed: https://github.com/aardappel/procrastitracker/commit/738cf1da69acd1a446cddb18550f81e9e9791f03

So merge should now work, however, you should still be better off renaming the last working .PT file to db.PT instead, unless you have 2 files from separate date ranges that you want to merge.

aburke20 commented 3 years ago

Woohoo, look at us testing and debugging ! I'll try the update.

OK, tested the merge, and got exactly what I wanted !

It also seems like you fixing the merge got rid of the "database is corrupt" issue, because I just tried to merge them all and they all merge just fine as if there was never a corruption issue !

As for the game, just promise to go into it with mild to moderate expectations. I don't know if you've heard, but it turns out the game can be a little bit of a let down if you were too excited about what it has to offer. Main story is still very gripping, but overall the game world needs a little work.

Hope you can play it soon !

aardappel commented 3 years ago

That's good to hear!

Yeah, I am actually delaying playing it a bit because of all the bugs.. and see if I can get a 3080 beforehand ;)