Closed wtanksleyjr closed 3 years ago
Thanks for telling me about the 'returning user' issue. I found the bug and the fix will go out today.
The liberated status is stored in the database in the user folder. I'm curious why after deleting the database it is seeing any as liberated. This to me is the real mystery.
The old way was to only recognize live files containing their id in the name. When a file was located, FilePaths.json was updated as a cache. People greatly disliked this so I changed it to being tracked by database only. A 1-time migration copied FilePath.json => FileLocations.json, read FilePath.json, updated the database, then deleted FilePaths.json. (FileLocations.json is used for non-mission critical things; not used to determine Liberated status like FilePath was.). The migration is long-running and happens in the background. If Libation is closed before it's finished, it'll just start from scratch next time.
If you still have access to those user files you deleted, you could restore FileLocations.json, make a back up, rename it to FilePath.json, and restart Libation. After a long run in the background it will mark anything found in there as Liberated and will delete FilePath.json.
Awesome, so I just didn't wait long enough for it to complete. Fair enough, I have a ridiculous number of books.
One more question: where is "FilePath.json" supposed to go exactly, I don't see it in the current install? Is it possible you meant to write "FilePaths.json" instead, since that does exist?
-Wm
On Tue, Sep 7, 2021 at 1:56 PM rmcrackan @.***> wrote:
Thanks for telling me about the 'returning user' issue. I found the bug and the fix will go out today.
The liberated status is stored in the database in the user folder. I'm curious why after deleting the database it is seeing any as liberated. This to me is the real mystery.
The old way was to only recognize live files containing their id in the name. When a file was located, FilePaths.json was updated as a cache. People greatly disliked this so I changed it to being tracked by database only. A 1-time migration copied FilePath.json => FileLocations.json, read FilePath.json, updated the database, then deleted FilePaths.json. (FileLocations.json is used for non-mission critical things; not used to determine Liberated status like FilePath was.). The migration is long-running and happens in the background. If Libation is closed before it's finished, it'll just start from scratch next time.
If you still have access to those user files you deleted, you could restore FileLocations.json, make a back up, rename it to FilePath.json, and restart Libation. After a long run in the background it will mark anything found in there as Liberated and will delete FilePath.json.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/105#issuecomment-914623786, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6LEKXIIBINGDH46FODUAZ377ANCNFSM5DTAE6YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
"FilePaths.json"
Yup
I'm sorry I didn't reply -- I wanted to give it a LONG LONG time to work. I don't see any sign it was doing anything -- no change in the number of books liberated or anything.
HOWEVER, I'm now trying with the new version (you released the bugfix for the "returning user" condition), and this time it's behaving completely differently -- it doesn't pop up any UI, and it's taking forever to start up, repeatedly creating and deleting "LibationContext.db-shm" and ..."-wal" files.
There's nothing in the log, so I can't tell what it's trying to do. Is it possible that it's making progress in loading the liberated books into the database? Or is this a problem?
-Wm
On Wed, Sep 8, 2021 at 4:36 AM rmcrackan @.***> wrote:
Closed #105 https://github.com/rmcrackan/Libation/issues/105.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/105#event-5271720207, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6KPIEIQE5TX365R2STUA5DFVANCNFSM5DTAE6YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
I'm hoping all this madness is related.
I've been thinking about your other issue where FilePaths is not getting turned into database state. All I can think is that an error is happening. If so, FilePaths will not be deleted and this will be in the logs:
Error attempting to insert FilePaths into db
The only time I've seen those db files flicker is when dropbox is fighting for file control. It's possible this is the books being imported except it really shouldn't act that way. Since your library is huge, let's go overkill and give it 20 minutes to see if it starts behaving.
Oh no, my kids turned off the computer. All I can tell you is that (1) it worked after more than a day of flickering DB files (and I don't keep my config folder on Dropbox or anything else); and (2) after it finished, instead of popping up the UI it displayed a long, complex error message in a modal dialog box without generating any log output. Unfortunately, I have no way to find what that error was -- my kids turned off the computer, and no log messages were ever written.
So let's keep this closed, it's good enough ... unless you can make that awful upgrade experience a little more debuggable, maybe write something to the logs for the normal case. I dunno if it's worth it, almost nobody will ever experience this or care about it. IIt just really sucked for my special case of a big library.
-Wm
On Wed, Sep 8, 2021 at 7:33 PM rmcrackan @.***> wrote:
I'm hoping all this madness is related.
I've been thinking about your other issue where FilePaths is not getting turned into database state. All I can think is that an error is happening. If so, FilePaths will not be deleted and this will be in the logs:
Error attempting to insert FilePaths into db
The only time I've seen those db files flicker is when dropbox is fighting for file control. It's possible this is the books being imported except it really shouldn't act that way. Since your library is huge, let's go overkill and give it 20 minutes to see if it starts behaving.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/105#issuecomment-915711960, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6LDOWWXWKQM3QK7DALUBAMIFANCNFSM5DTAE6YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Your library is only ~60% larger than mine. A day is way too long. There's something else going wrong here. I debugged the install/upgrade. Logging in there is a chicken and egg problem: I can't log until I can verify that the install/upgrade state is correct.
Maybe there's something corrupt in the FilePaths file but that wouldn't explain the db flicker. I'm ready to go drastic on this if you're up for it. If and only if you feel comfortable with this, please zip the following files and email me a download link -- don't post it to github. In your settings folder are 3 folders (DownloadsFinal, Images, SearchEngine). I do not want any of those. There are also 6-10 files. Please send all of these EXCEPT the Accounts one -- that's where all of your authentication tokens are.
All right, emailed you a GDrive link. Let me know if you need more permissions or anything.
When I used your files, it failed on my computer because your paths don't exist on my box. No surprise. In Settings.json I fixed the Books path . It still silently dies because I don't have your InProgress path. I fixed that too and it started right up. (You must have permissions to create folders inside of the InProgress folder.)
Book counts. {"total":2819,"booksFullyBackedUp":2080,"booksDownloadedOnly":0, "booksNoProgress":739,"booksError":0} PDF counts. {"total":199,"pdfsDownloaded":199,"pdfsNotDownloaded":0}
For fun, I did the FileLocations => FilePaths trick. It ran for 9 minutes and didn't affect the above numbers.
I could also just delete Settings.json and on the next restart it made me choose some new settings then started right up.
Let me know if you can reproduce any of this.
Oh -- that's my current data, and therefore everything except the podcasts are already loaded.
I don't think I can recreate the pre-import state again, sorry. :(
-Wm
On Fri, Sep 10, 2021 at 1:50 PM rmcrackan @.***> wrote:
When I used your files, it failed on my computer because your paths don't exist on my box. No surprise. In Settings.json I fixed the Books path . It still silently dies because I don't have your InProgress path. I fixed that too and it started right up. (You must have permissions to create folders inside of the InProgress folder.)
Book counts. {"total":2819,"booksFullyBackedUp":2080,"booksDownloadedOnly":0, "booksNoProgress":739,"booksError":0} PDF counts. {"total":199,"pdfsDownloaded":199,"pdfsNotDownloaded":0}
For fun, I did the FileLocations => FilePaths trick. It ran for 9 minutes and didn't affect the above numbers.
I could also just delete Settings.json and on the next restart it made me choose some new settings then started right up.
Let me know if you can reproduce any of this.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/105#issuecomment-917202791, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6PLZ5FL6SRBHCPOOM3UBJVSDANCNFSM5DTAE6YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Can you get it to work though? Using your files I didn't encounter any of the database fighting or errors you described.
As I mentioned, it worked after a day of processing.
The reason you don't get any errors is that I did what you asked, and sent my current state.
I don't think I will be able to get back to the pre-working state -- it's been too long.
So I think this is a dead-end, "unable to recreate." Kinda "oh well," and kinda "ok, let's hope it's just me."
Thank you for trying! Now I'm excited to try the CLI :) .
-Wm
On Fri, Sep 10, 2021 at 2:25 PM rmcrackan @.***> wrote:
Can you get it to work though? Using your files I didn't encounter any of the database fighting or errors you described.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rmcrackan/Libation/issues/105#issuecomment-917221675, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJ7H6P733C5WS5L2DL4MQDUBJZUDANCNFSM5DTAE6YQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
As I mentioned, it worked after a day of processing.
I missed this part. Agreed: since we can't get that state back and can't reproduce the error, this is dead.
The CLI is a first draft and I welcome any feedback. Enjoy!
I finally upgraded to the newer code for the first time since the outage. The upgrade was a surprising experience; it didn't go like I expected.
First, I simply deleted the old Libation executable folder and dropped the new one in its place. I ran it, expecting the upgrade dialog, but it started up as though nothing had changed -- except that only 100-odd of my 1900 books were marked as imported. I imagined there might be something running in the background, so I walked away and left it running for a few hours, but when I came back it was in the same state.
Second, I tried deleting both the Libation executable and the user folder -- everything except the books (I guess I didn't remove the temporary folder, come to think of it, should I have?). That did bring up the "are you a new user or a returning user", so I chose "returning", and then chose the user folder just like before. Here's the second surprise: after asking whether I wanted to create the folder, it asked me again where I wanted to place the folder, and when I clicked "OK" it told me "installation canceled" and quit. No obvious reason for quitting.
Third, I tried "new user". This time was more like I expected, and all was well -- except that it only "sees" the same ~100 books as being Liberated (out of 1900+ books). I'm not sure why those 100 books and not any of the others!
So ... how can I get the new system to see all of my old liberations? And is the "existing user" upgrade code working correctly?