"unable to open database file", intranet library, is read only?

[SimonDedman]

Screenshots of issues above. Setup is: watch folder & qiqqasync directory both in a local network directory which opens and can be edited fine, as far as I can tell. I'm able to setup the intranet library but when I try to full sync I get the above errors.

I have another machine with the same setup and it works fine. I'd been using that machine (home) all day, then tried to sync the other machine (work). Qiqqa was closed on the home machine.

Any ideas gratefully received. Cheers!

[GerHobbelt]

Phew, this might be an issue with SQLite databases and network storage, though that's "usually" reported as crap happening on DropBox shares and the like (cloud storage).

Thanks for the screenshots: what I can glean from there is that I probably need to augment that error report so that we can dig up on what database file exactly SQLite is barfing a hairball.

---

On another note I do recall so old trouble back in '19 with the whole Qiqqa Sync business, which had a few lines from the commercial age being obstructive, but then that should have been gone out the window by window.

Got to dig in, so don't expect anything useful before the weekend.

Plan of attack in my mind right now:

• me: augment the error report so we can see paths and maybe a little more detail
• me: publish that in a new release
• you: install new release and then run that one to see what gives
• you: collect log files (same as #264) and emailing them to me
• me: log/error analysis and hopefully a lucid moment right there 😉

P.S.: just doublechecking here to make sure I can tick off the scare in my brain: both machines still have a working Qiqqa and Qiqqa libraries, right?

[SimonDedman]

Cheers for the wonderfully thought-out plan! :)

Yep - both having working Qiqqa, just getting the intranet library working on the second doesn't work, even though I'm using the same approach as on the first.

One thought is that I don't know with 100% certainty that both machines are setup EXACTLY the same way in terms of reaching the NAS. Now, I think they are. But browsing to the NAS as a network drive took more steps on the home machine (that works) than the work machine, which worked straight off the bat. Now they look and function the same but I don't know if there's any small difference behind the scenes. I suspect this isn't the issue but I figured I'd mention it just in case.

On a completely separate note, I noticed that the Qiqqa getsatisfaction community has been removed; I wonder if it's possible for us to get an archive of this somehow (maybe Wayback machine)? I figure this could be quite useful since it's got a lot of community ideas for workarounds or bug discussions, so it could be a useful resource...

Cheers!

[GerHobbelt]

Before I'm off (near 1 AM now): I do recall some trouble with sync behaviour (enabled/disabled) back in '19. Given your problem, I must look into that part of the code again I expect. At least that's my initial hunch without any backing except "gut feeling, no data". We'll see what happens when I do an updated version of qiqqa for you around tomorrow or day after, depending on how things go with the day job.

On the separate note:

• 218

• and since I was asleep at the wheel on that one (did a web grab, then nuked the webgrab, which was 😨 🤢 klunky anyway, as part of a large SSD cleanup for I needed room. Of course, Murphy being a best buddy I got under speed-dial, that happened after GS shut down. 🥶 Em?... Whoops?!

  • Interesting to note at the time: #218 was Jimme giving a heads up, but when I looked as GS myself as a user the b-tards didn't give a peep about shutting down their 💩 🤦
  • Had a bright moment and did a google / duckduckgo search for qiqqa+GS and got my grubby paws on the not-yet-obsoleted search engine cached versions of those files --> https://github.com/jimmejardine/qiqqa-open-source/tree/master/docs-src/getsatisfaction-mirror/____extracted -- that was a bit of Mechanical Turk work, but alas, I got the goods as far as possible.

  Needs more postprocessing, but that's what I got going through all the search result pages until the well ran dry. Skipped the spam pages that had been cached too, BTW.

  • Wayback Machine would have been an option, iff Qiqqa GS forum had been registered with them for waybacking: they don't do the entire internet and this, AFAICT, was not in their backups. If anyone finds Qiqqa GS in the Wayback machine, then I'd like to hear it, but my google foo delivered nada there.

[OK, that story isn't written in chronologic order. Hope it's intelligible anyway. Anyhoo, I'm off. Gotta go. 👋 ]

[SimonDedman]

Interesting rollercoaster ride of a tale about the GS issues! I suspect there's not tons of useful stuff on there but you never know. I'm sure we'll find all the bugs again anyway!!

[GerHobbelt]

OK, looks like it's the sync target, i.e. the SQLite database file on the NAS, which might be damaged and SQLite barfing a hairball about this.

Have been digging around as I needed to see where I augment the logging, etc. and not yet reproduced your problem, but that should be fine for the next phase: running a Qiqqa update to test this in more detail.

WARNING NOTE

Since I gather from digging through the code and your screenshots of the failure that the NAS copy of your library is probably corrupt (the metadata store at least as the BibTeX etc. is stored in a SQLite database) and before we commence I urge you to back up your Qiqqa libraries from the local machine: nobody would be happy when that stuff gets damaged by a subsequent (newer) Qiqqa run after installing the Qiqqa test release that's going to be referenced in my next comment.

Best practice would be to backup the Qiqqa base directory and everything inside from the local machine and possibly of the NAS as well, just to be on the safe side and my preliminary guess turns out wrong. Personally, I would ZIP (or 7ZIP) the whole kaboodle; might want to use a "multiple volume 7ZIP" archive for that, i.e. tick the box in 7zip when creating the archive and you get archive file segments named archive.001, archive.002, etc.

Screenshots of how that might look:

^^^

would produce something like this

==>

(nitpick: those are screenshots from different actions, so base.7z in the first should be read as 'C9F0D079-EE4C-4A15-8547-72164A7A356D-updated-GHO.7z')

END OF WARNING NOTE

Link to Qiqqa update for testing coming up next....

[GerHobbelt]

OK, new Qiqqa setup for you to try at https://drive.google.com/file/d/13gWzHHtL0dpRAJlSWZjweE3-3RWXFpxY/view?usp=sharing which should give you access to a Google Drive download of setup-v82.0.7576.3604.exe

Make sure you've backed up your Qiqqa data before running this installer.

Once installed, run Qiqqa and see if you can reproduce the problem and see what Qiqqa says then in the logging. This time around Qiqqa should be a bit more verbose and also tell us which database file exactly triggered that b0rk.

(This is an unlabeled release, coded as v82.0.7576.3604 for commit SHA-1: 4433a6acfe2a2f418389e227ba94c319c41b1347 )

Trouble that I expect to find over there:

• given the screenshots posted and the code review I did tonight, the next hurdle would be how to approach recovering a broken SQLite database.

  • the easy way out there would probably be to nuke the *.s3db file in the NAS sync target directory and then let Qiqqa sync it all over again from scratch.

  However: I have not yet tested that 'desired behaviour' and you might end up with a borked library instead for whatever reason. Hence the plea to backup! backup! backup! first! 😅

• Then there's the things I have to do to edit the sync target path, i.e. allow users to point the sync point of their Qiqqa libraries to a new location. That is still wishful thinking as it hasn't been coded yet (and tonight I feel about WPF/XAML like !@#$%^&*)

  That's not directly related to your issue, but since we're touching that chunk of the ware anyway, heck, might fulfil a long pending wish in the issue tracker, even when it's done in a hacky way maybe. 🤷‍♂️

• For the rest, I'm open to surprises. (As in the original plan of attack: if you find anything or nothing, please send the log files for inspection when you feel you're done testing. Thanks!)

  

[GerHobbelt]

Oh, the new Qiqqa should show the Sync Target path in the sync manager dialog. That might also help the diagnosis a bit.

[SimonDedman]

Howdy! A thousand thanks for helping with this, again.

I installed the new version of qiqqa on my work machine. It looks like my original screenshot (second one) was cropped too far down, and the line before was key:

20201029.173706 [Q] ERROR [22] [24.238M] IntranetLibraryDB::GetItemsSummary: Database I/O failure for DB '\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db'.

Sync target path is correct, \NAS\Si Work\Papers & Books\QiqqaSync

In my home machine (working, just did a full sunc successfully which is weird since you'd think it would announce a problem with Qiqqa.LibraryMetadata.s3db) I'll backup the NAS library then nuke that s3db file and try to sync it back (again from home machine) then see if the work machine, uh, works.

To be continued!

[SimonDedman]

So: nuked Qiqqa.LibraryMetadata.s3db, resyned on home machine to rebuild it, successfully (on home machine), but no change on work machine. Thought: could it be that the database file has some form of machine-specific ID that leads it to be readable on the machine that set it up, but not on other machines?

Logs attached for the work machine. !Qiqqa.ErrorsAndUserActions.log

In 'Properties' for that file, details in General (archive) and Security (Allow for all except 'full control') are the same on both machines.

Uninstalled & reinstalled Qiqqa on work machine. Upon reopening, my intranet library is present (with 0 docs as usual). This seems strange; maybe the qiqqa library isn't removed upon uninstall...

I still haven't updated my home machine's Qiqqa to the latest version, since it works fine. Let me know if doing so would be beneficial for diagnostics.

Cheers again bud!

[GerHobbelt]

Thought: could it be that the database file has some form of machine-specific ID that leads it to be readable on the machine that set it up, but not on other machines?

Not that I recall.

Besides, the .s3db file is a plain Sqlite database file and those are portable across machines as far as I know. (Since Qiqqa only runs on Windows, it's even simpler: every machine will then be little endian and 32-bit execution for Qiqqa runs as a 32-bit .NET application, even when the OS is 64-bit, so I cannot imagine anything that might make it load on one and fail on the other. (Aside: This 32-bittiness is due to the embedded webbrowser (XULrunner) forcing the whole application down to 32-bit exec alas.)

Uninstalled & reinstalled Qiqqa on work machine. Upon reopening, my intranet library is present (with 0 docs as usual). This seems strange; maybe the qiqqa library isn't removed upon uninstall...

Indeed, Qiqqa does not delete user config upon install: that's a safety measure so folks don't 'loose' their Qiqqa libraries when upgrading.

The magic ingredient there is the file listing the discovered libraries, which sits in the libraries base directory: Qiqqa.known_web_libraries -- NOTE that when you had "Web Libraries" back in the day of commercial Qiqqa or otherwise mixed & mashed your library collection, then you may discover there are multiple files called Qiqqa.known_web_libraries in the various subdirectories of the Qiqqa base dir: all of these are loaded and combined into a single lump sum set at Qiqqa start of application.

I still haven't updated my home machine's Qiqqa to the latest version, since it works fine. Let me know if doing so would be beneficial for diagnostics.

Might be useful later, but hold off for a bit as I'm working on a new release which should be ready tomorrow there-abouts.

Besides, the .s3db file (and the rest) SHOULD BE cross-version binary compatible with all v79/v80/v81/v82 Qiqqa releases: I haven't changed the formats of any (yet). 😄

Consequently my expectation had been that your Qiqqa setups should either yak about the .s3db database file both, or be able to read & write it without errors.

🤔 🤔 🤔 🤔

Given the results so far and the (to me) very curious behaviour as you describe, let me back-pedal and make darn sure we're on the same page about your layout:

• you have two work machines:
  • one I will call HOME
  • another one I will call OFFICE
• Qiqqa runs on both
• both HOME and OFFICE machines have access to the same NAS storage:
  • this is addressed as a UNC path: '\\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db'
  • both at HOME and at the OFFICE, the Qiqqa library is configured to sync with that UNC path

That last point can only be easily checked with the Qiqqa version that you got from me, as previous versions didn't yet show the sync path in the Sync Management dialog.

Anyway: is the above HOME/OFFICE/NAS layout assumption correct? (If not, please correct me!)

---

There's one more thing you might want to check, just in case:

The 'sync target' (in your case the NAS directory) should also contain a file called Qiqqa.LibraryDetail.txt: you can open that file in a text editor, e.g. Notepad++ to have a look.

This is one example how such a file should look:

{"Id":"INTRANET_FFFF9774-FFFF-FFFF-FFFF-E67433F7FFFF",
"Title":"Issue 142 Library #1 for Qiqqa",
"Description":"yada yada yada."}

The exact format is quite liberal (whitespace & newlines don't matter) as this is a JSON format file.

The important bit is the Id: this is the exact name of the directory your library should be stored in on both machines.

For example, if your Qiqqa base directory on HOME would be (like I have):

  G:\Qiqqa\base\

then the library from the example should be in

  G:\Qiqqa\base\INTRANET_FFFF9774-FFFF-FFFF-FFFF-E67433F7FFFF\

so you should find the library database file there, at least:

  G:\Qiqqa\base\INTRANET_FFFF9774-FFFF-FFFF-FFFF-E67433F7FFFF\Qiqqa.library

Same for the OFFICE machine, where I'll say, for this example, the base directory is at

  C:\Users\Ger\Documents\Qiqqa\

hence the same library there should be in

  C:\Users\Ger\Documents\Qiqqa\INTRANET_FFFF9774-FFFF-FFFF-FFFF-E67433F7FFFF\

with its own local library database file (and other data):

  C:\Users\Ger\Documents\Qiqqa\INTRANET_FFFF9774-FFFF-FFFF-FFFF-E67433F7FFFF\Qiqqa.library

---

So far, that's the only other thing I can imagine that may have occurred somehow, is one of the machines (OFFICE?) losing the sync target reference.

Do note that I'm grabbing at straws here as I have no good explanation for what's happening over at yours, so this will take some diagnosing iterations before we hit it. 😢

That's worst case scenario and the reason why I requested you back up everything: when we don't know what's going on precisely, stuff can get worse before we get it fixed!

Let me know if that Qiqqa.LibraryDetail.txt cross-check delivered anything or my layout assumptions of your situation are wrong.

Stay tuned for another Qiqqa setup tomorrow. (Will be late-ish as this diagnostics round uncovered other bad bits in testing here locally, which are getting fixed.)

[SimonDedman]

Hi again,

My layout is exactly as you described, yes.

My Qiqqa.LibraryDetail.txt contains only the line:

{"Id":"INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73","Title":"SimonDedmanNASsync","Description":"Synced in NAS work papers folder"}

On my HOME machine:

C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73

On my WORK machine:

C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73

Ok, so, same. But obviously the WORK machine's folder has a lot less in it, 5kb vs 4.5gb.

Maybe if I again uninstall qiqqa on WORK, AND delete the user config files??

Edit: I "Forgot" the intranet library on WORK and it did NOT delete that folder (C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73). So I manually moved the folder, restarted Qiqqa, joined the intranet library at NAS again, same issue.

20201030010202:code = CantOpen (14), message = System.Data.SQLite.SQLiteException (0x800007FF): unable to open database file at Qiqqa.DocumentLibrary.IntranetLibraryStuff.IntranetLibraryDB.GetIntranetLibraryItemsSummary() at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder_Intranet.BuildFromRemote(Library library, SynchronisationStates& synchronisation_states) at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder.BuildFromRemote(Library library, SynchronisationStates& synchronisation_states) at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder.Build(Library library, Dictionary`2 historical_sync_file) at Qiqqa.Synchronisation.BusinessLogic.LibrarySyncManager.SynchronizeMetadata_INTERNAL_BACKGROUND(Library library, Boolean restricted_metadata_sync, Boolean is_readonly)

The folder has been created afresh. Possibly deleting the full user config files folder will work though, I'll give it a whirl.

Edit: still no. It does clean out some memories of configurations but the issue remains. The folder is created, again with the correct looking name. The mystery continues!

[GerHobbelt]

Quick prelim release for you to try out: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v82.0.7578.6369

When you install this one on both machines, it should present the proper Sync Point path ('\\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db') as desired -- since you have still trouble with the OFFICE machine, this should be able to check my current working assumption that something on the OFFICE box corrupted/killed the Sync Point setting.

[SimonDedman]

Strange - no change in the path having upgraded the office machine only (so far):

Log attached. Qiqqa-20201031.221242.log

[GerHobbelt]

Strange indeed. Will check your logfile in a bit -- just finishing other stability work on Qiqqa (#264 + the aftermath of 0d35bf65072432e7ab338235024a6d8ed8a42b02).

[GerHobbelt]

Okay. The magic ingredient in that one is this chunk:

20201031.221346 [Q] ERROR [22] [20.369M] IntranetLibraryDB::GetItemsSummary: Database I/O failure for DB '\\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db'.

System.Data.SQLite.SQLiteException
Message: unable to open database file
ErrorCode: 0x0000000E
HResult: 0x800007FF
Source: System.Data.SQLite
StackTrace:    at System.Data.SQLite.SQLite3.Open(String strFilename, String vfsName, SQLiteConnectionFlags connectionFlags, SQLiteOpenFlagsEnum openFlags, Int32 maxPoolSize, Boolean usePool)
   at System.Data.SQLite.SQLiteConnection.Open()
   at Qiqqa.DocumentLibrary.IntranetLibraryStuff.IntranetLibraryDB.GetIntranetLibraryItemsSummary()
TargetSite: Void Open(System.String, System.String, System.Data.SQLite.SQLiteConnectionFlags, System.Data.SQLite.SQLiteOpenFlagsEnum, Int32, Boolean)

20201031.221346 [Q] INFO  [22] [20.006M] About to display client stats: Something unexpected has happened, but it's okay. unable to open database file
20201031.221349 [Q] INFO  [Main] [24.656M] Screen position for window UnhandledExceptionBox stored as 1400|400|640|600|Normal
20201031.221349 [Q] ERROR [22] [24.738M] Unhandled Exception Handler: Something unexpected has happened, but it's okay. unable to open database file - You can continue working, but we would appreciate it if you would send us some feedback on what you were doing when this happened.

System.Data.SQLite.SQLiteException
Message: unable to open database file
ErrorCode: 0x0000000E
HResult: 0x800007FF
Source: System.Data.SQLite
StackTrace:    at Qiqqa.DocumentLibrary.IntranetLibraryStuff.IntranetLibraryDB.GetIntranetLibraryItemsSummary()
   at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder_Intranet.BuildFromRemote(Library library, SynchronisationStates& synchronisation_states)
   at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder.BuildFromRemote(Library library, SynchronisationStates& synchronisation_states)
   at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder.Build(Library library, Dictionary`2 historical_sync_file)
   at Qiqqa.Synchronisation.BusinessLogic.LibrarySyncManager.SynchronizeMetadata_INTERNAL_BACKGROUND(Library library, Boolean restricted_metadata_sync, Boolean is_readonly)
TargetSite: Void Open(System.String, System.String, System.Data.SQLite.SQLiteConnectionFlags, System.Data.SQLite.SQLiteOpenFlagsEnum, Int32, Boolean)

So the machine is telling us it cannot read the .s3db file. At all, since it's not bad records bubbling up from a query, which would look quite different with the bleeding edge of Qiqqa as you have now running.

That does, regrettably, not tell us yet why it finds the thing 'cannot be opened'. To help us diagnose that, I fear I'll have to do a bit more work on the code so Qiqqa can tell us a bit more about the why. Stay tuned for a new setup installer coming your way today.

[GerHobbelt]

Here's the new version to test for you: https://github.com/GerHobbelt/qiqqa-open-source/releases/tag/v82.0.7579.33985

Of particular interest to you & me in this case would be the new diagnostic report appended to every SQLite error: there we should find all available File System info plus a few tests and some preliminary diagnosis of the situation as done by Qiqqa itself.

It'll all end up in the log files, so, as before, I am mighty curious to hear what the OFFICE machine has to say re NAS access now.

---

P.S. note that I will have the day job to attend to the entire week, so things may get delayed or late as working hours are 0700-2200 or there-abouts inc. travel (remote real estate project). I'll try to look into anything you send ASAP anyway. Mighty curious what the heck is going wrong.

[SimonDedman]

Cheers mate, attached. Of interest based on my uneducated eye: "Could not find file 'C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5'." Maybe normal on first sync attempt?

"--> Is a normal file: False" Not sure if this is good or bad!

"SQLite Error Code: 14 Reported error code is NOT a SQLite Extended Error Code. SQLite HResult: -21474816018:X"

"--> VERDICT OK(?): this DOES look like a very normal file."

Strange...

Ger, absolutely ZERO need to explain nor apologise for delays buddy - I appreciate you're doing this in your spare time as a gift to the community and specifically for me (currently). Day job takes precedent, always, of course. Enjoy the travel - seems like a very unusual thing nowadays!

Cheers.

Qiqqa-20201102.173708.LOG

[GerHobbelt]

Thanks!

Just did a quick scan of your log file and the plot thickens... I'll explain in a sec, but the low down of it is that I'll need a look at your .s3db file and dig through it with SQLite tool as I still don't have an explanation what's going wrong there...

Hmmmm. Thinking while writing this message... There's two things you can do, the second of which might "magically" solve this (yup, sometimes engineers wave a dead chicken and turn the prayer wheels; we're at that level of non comprendo, alas.) After those two items, I'll address your notes and the logfile you sent and what it "says".

Actions first:

1. If you're okay with sharing that file, I'm mightly interested in a ZIPped copy of your .s3db file: if you can pack \\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db into a ZIP and send it to me ( ger@hobbelt.com; please mention 'Qiqqa' in email subject line or the spam filters may kill it) so I can have a look at the binary when I have time (next weekend, probably).

2. What I just thought about (also given your sync_md5, which we'll get to in the log analysis section below) is that you MIGHT want to try this:

• DELETE or RENAME the \\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db file so that Qiqqa won't be able to find it, e.g. name it \\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db.bak1. What Qiqqa on the HOME machine SHOULD do, is on sync find out that the s3db is missing, copy an empty template DB in its place and get busy syncing all from scratch. (Remember those backups I said you should have made? This assumes your data is in a safe place to get back if this results in 'bad things happening'; not that I expect but we have one 'weird' machine as we're getting crafty, so better not risk loss of important goods while we EXPECT this to work out, but cannot guarantee it right now.)

Anyway, the idea here is that you RESET the HOME machine's Qiqqa into a state of mind where it believes that is has a Sync Point (\\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db) BUT also discovers that it has NEVER synchronized yet and do a full copy to the NAS. The way to accomplish that change of mind is 2-fold:

A: RENAME or DELETE the .s3db file at the 'NAS' side, so it will at least not do something with a existing file which happens to make the OFFICE box behave so very oddly. So we tickle Qiqqa (HOME) to create a fresh one for us from scratch and take it from there. B: on the HOME box, it is important to RESET Qiqqa into thinking "ow, sync point but apparently I haven't copied a thing yet! Let's do that now!" and that is accomplished by DELETING or RENAMING the corresponding .sync_md5 file in the local library directory on the HOME machine: the .sync_md5 file is where Qiqqa 'tracks' what it has synced previously so it can 'changes only'. We want EVERYTHING, so no previous work, so get rid of that .sync_md5 file.

NOTE: one thing I haven't tested yet, ever, is how Qiqqa will react to the PDF files already existing on the NAS, as Qiqqa should discover those are already there when it starts to look at HOME + NAS to discover the necessary sync actions (copy which metadata records, which PDF files, etc.)

EXPECTED end result is HOME syncing to NAS, taking a while maybe, but I expect that machine to copy all metadata into a fresh .s3db file (which should be present on the NAS once HOME has synced after killing the old .s3db and .sync_md5 files.

OW, forgot to mention but just to make sure: DELETING/RENAMING those files should be done before you start Qiqqa on HOME !

That's action 2 on HOME.

Now we hop over to OFFICE, should see a new \\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db from there, then

• we do the same DELETE/RENAME local .sync_md5 file ( C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5 ) -- given the noises Qiqqa is making in the log, it's already gone. No sweat if it is gone, but if it isn't now's the time to kill the bugger.

• next when you run Qiqqa on the OFFICE box, it should discover the library with a sync point at C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5 like it does now.

Since HOME has been previously enticed into creating a completely fresh .s3db file for us, the OFFICE sync should run with no errors. If it errors, well more logs to send me and bundle the 'new' s3db as well, I'd say. I hope I get a 'it's working now!' message instead, because if this worse case scenario is happening, you got me really stumped. (See also the explanation about your logfile and your notes in your message. If this fails without a clear error cause in HOME or OFFICE log, then we got a real gremlin -- or me missing some horribly wicked detail.)

Anyway, long story short:

Action 1: consider sending me the whole kaboodle, i.e. s3db at least + logs as you go.

Action 2: tickle Qiqqa on HOME to completely rewrite the NAS DB from scratch by DELETING ...\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5 on HOME plus DELETING \\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db on the NAS. (RENAME is as good as DELETE: Qiqqa must not find these files at the expected spots and create them anew.) Then run Qiqqa on HOME to sync, which should result in a new \\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db being created. (Only bit I keep my fingers crossed about is the PDF files' sync as we did not DELETE those off the NAS. should be okay, but this is the only risk bit as far as I can tell.) Once SYNC is complete, go over to OFFICE machine, check if the new s3db on the NAS is visible, DELETE the local sync_md5 if it happens to exist, then start Qiqqa on OFFICE to start the sync there, which SHOULD now SQLite-open the NEW .s3db file and not report an error.

So far, the actions. Logging review I'll do in the next message as writing this as I go made this a pretty long text already.

Good luck! (You & me both :-))

Met vriendelijke groeten / Best regards,

Ger Hobbelt

---

web: http://www.hobbelt.com/ http://www.hebbut.net/ mail: ger@hobbelt.com mobile: +31-6-11 120 978

On Mon, Nov 2, 2020 at 6:48 PM Simon Dedman notifications@github.com wrote:

Cheers mate, attached. Of interest based on my uneducated eye: "Could not find file 'C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5'." Maybe normal on first sync attempt?

"--> Is a normal file: False" Not sure if this is good or bad!

"SQLite Error Code: 14 Reported error code is NOT a SQLite Extended Error Code. SQLite HResult: -21474816018:X"

"--> VERDICT OK(?): this DOES look like a very normal file."

Strange...

Ger, absolutely ZERO need to explain nor apologise for delays buddy - I appreciate you're doing this in your spare time as a gift to the community and specifically for me (currently). Day job takes precedent, always, of course. Enjoy the travel - seems like a very unusual thing nowadays!

Cheers.

Qiqqa-20201102.173708.LOG https://github.com/jimmejardine/qiqqa-open-source/files/5476908/Qiqqa-20201102.173708.LOG

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-720625613, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHTF34P2O7AXPIA6QBLSN3WGPANCNFSM4S36UMAQ .

[GerHobbelt]

(off-topic: doing this in the github issue tracker via web as I want the formatting to be just-right or this will become a horrible read. Back to the subject at hand: analysis of the latest logfile and replying to your notes:)

OFFICE log:

the important stuff starts with this:

20201102.173824 [Q] INFO  [Main] [26.720M] Syncing
20201102.173824 [Q] WARN  [Main] [27.308M] SYNC IGNORE: library 'Library Guest: Local Guest Library' has no Sync Point a.k.a. Share Target set and thus cannot be synced.

Sync process is starting. 'Guest' lib is classically now syncable, no surprises there. (This will change in a future release, but we'll stick with what we have right now.) So far, this is all 'good noises'. Expected output. Onwards!

20201102.173824 [Q] DEBUG [6] [27.317M] SYNC_META:INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73:Starting sync of SimonDedmanNASsync (0/0: 0.0%)
20201102.173824 [Q] INFO  [6] [27.317M] Syncing metadata for SimonDedmanNASsync
20201102.173824 [Q] DEBUG [6] [27.325M] SYNC_META:INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73:Loading sync history (0/0: 0.0%)

Qiqqa inspecting your second library on the OFFICE box. This is the one that comes with a Sync Point (we'll see in a sec). So far, good noises. All clear.

20201102.173824 [Q] WARN  [6] [27.958M] Error loading C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5
System.IO.FileNotFoundException
Message: Could not find file 'C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73\Qiqqa.sync_md5'.
HResult: 0x80070002
Source: mscorlib
StackTrace:    at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)
   at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
   at Utilities.Files.SerializeFile.Load(String filename)
   at Utilities.Files.SerializeFile.LoadRedundant(String filename)
   at Utilities.Files.SerializeFile.LoadSafely(String filename)
TargetSite: Void WinIOError(Int32, System.String)

Error. About the local .sync_md5 file. WITHOUT CONTEXT, I would have said "not a problem. That's just indicative of a lib with sync point which has never been synced yet on that machine." (Hold that thought for a sec...!) Qiqqa 'agrees' with my assessment, as we see in the next log line:

20201102.173824 [Q] INFO  [6] [27.968M] There is no sync history, so creating a new one

Again, 'good noises', IF IT WEREN'T for me wondering aloud: "but Simon has had this setup for a while now! Did he nuke the sync_md5 on the OFFICE previously? (Which is A-okay, Qiqqa should just pick from there and do a full sync on OFFICE, but this is me right now wondering if you did this as part of the ongoing process (might have mentioned to you to do this a while ago) or if this also a 'surprise' to you.

If it is a surprise to you, this merrits a "what the heck?" but otherwise won't be able to help us fix the situation. The actions mentioned in my previous message might resolve the matter. So all we can do now with this is raise eyebrows, wonder and continue with the rest of the logfile:

20201102.173824 [Q] DEBUG [6] [27.977M] SYNC_META:INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73:Building sync state from local history (0/0: 0.0%)
20201102.173824 [Q] DEBUG [6] [27.977M] SYNC_META:INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73:Building sync state from local files (0/0: 0.0%)
20201102.173824 [Q] DEBUG [6] [28.001M] SYNC_META:INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73:Building sync state from Web/Intranet Library (0/0: 0.0%)

That's the sync process starting for real. Just did a quick source code check about those (0/0: 0%) numbers: that's just internal progress counters and we're at the start, knowing nothing yet, so 0/0 is fine. Good noises. Onwards.

20201102.173824 [Q] ERROR [6] [23.221M] IntranetLibraryDB::GetLibraryItemsSummary: Database I/O failure for DB '\\NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db'.

System.Data.SQLite.SQLiteException
Message: unable to open database file
ErrorCode: 0x0000000E
HResult: 0x800007FF
Source: System.Data.SQLite
StackTrace:    at System.Data.SQLite.SQLite3.Open(String strFilename, String vfsName, SQLiteConnectionFlags connectionFlags, SQLiteOpenFlagsEnum openFlags, Int32 maxPoolSize, Boolean usePool)
   at System.Data.SQLite.SQLiteConnection.Open()
   at Qiqqa.DocumentLibrary.IntranetLibraryStuff.IntranetLibraryDB.GetIntranetLibraryItemsSummary()
TargetSite: Void Open(System.String, System.String, System.Data.SQLite.SQLiteConnectionFlags, System.Data.SQLite.SQLiteOpenFlagsEnum, Int32, Boolean)

Kaboom! There's that one again. Bad noise! We already know this, but this SQLite barfing a hairball about not being able to connect (not even running a query yet!) to the NAS DB: the s3db file.

Your bleeding edge Qiqqa software now adds a diagnostic report, which we'll get into next:

20201102.173830 [Q] ERROR [6] [27.420M] --- Diagnosis for reported problem ---
======================================

--> File Attributes:                                Archive

'Archive' attribute. FileSystem legacy stuff from back in the olden days of MSDOS. Almost nobody does anything with that flag, anymore. Qiqqa and SQLite sure wouldn't bother about it being off or on.

Here follows the rest of the detailed FileSystem info dump which I included in the diagnostics report.

--> File Creation Date (UTC):                       29/10/2020 19:16:44
--> File Last Changed Date (UTC):                   29/10/2020 19:34:57
--> File Last Access Date (UTC):                    29/10/2020 19:34:58
--> Is marked as READ ONLY:                         False
--> Is marked as OFFLINE:                           False
--> Is marked as archived:                          True

(Last one: that's the archive bit again. Just from another API. ok.)

--> Is marked as HIDDEN:                            False
--> Is a SYSTEM FILE:                               False
--> Is encrypted by the operating system:           False
--> Is compressed by the operating system:          False
--> Is a mount point:                               False
--> Is temporary:                                   False
--> Is a symbolic link:                             False
--> Is a sparse file:                               False
--> Is a reparse point:                             False
--> Is not content indexed by the operating system: False
--> Is a directory:                                 False
--> Is a device:                                    False

Nothing crazy so far. "Vanilla". Good noises.

--> Is a normal file:                               False

This one may indeed sound confusion, but it his is the OS telling us that the file is a 'normal file' without any attributes set. Since the (A)=Archive bit has been set, this is reported as "normal=false" at OS level, but nothing crazy or off about this. So, confusion aside, definitely not surprising to me.

I should have written this one up as "file without any attributes set" to prevent confusion. Good catch in your notes. Anyway, no cause for concern.

--> File size:                                      27233280 bytes

First time we see the size: this is straight off the disk from the Windows operating system: this is the number you'ld see in the Windows Explorer when you request the File Properties of the s3db file. All okay.

--> Successfully scanned the entire file: length scanned = 27233280 bytes

THIS is important as under hood that was me just READING every byte in the s3db file from NAS: this means that the file proven to be readable by the Qiqqa software, at least at a binary level!

Meanwhile Qiqqa's built-in SQLite kicks up a ruckus, so from this (plus the next line in the log!), it looks like there's some trouble for SQLite about the database file not being in the correct format? Or is SQLite.NET doing things I don't know yet? I downloaded their sourcecode, but that quite a large chunk of code and I haven't had time to dig through that bunch yet, but this part of the diagnostic tells me Qiqqa can read the s3db, so, following logic, it must therefor be something in the format OR in yet-unidentified problems originating from the exact way SQLite opens a database file. We'll have to dig deep to answer that one. (Incidentally, this is why I refuse to work on projects with closed-source libraries: we you cannot read the sourcecode in cases like this, you've hit a very hard wall. There's a commercial crypto-library way back which cost me dearly in reverse-engineering/diagnostics cost (2 man-weeks of digging with sophisticated rev-eng tools) thanks to closed-source. Anyway, we'll have to dig into SQLiteNET if we want to get to the bottom of this, because my new diagnostics just completely the normal FILE READ process and found no fault what-so-ever: "Successfully scanned the entire file: length scanned = 27233280 bytes": no error report in this line, size reported MATCHES the OS-reported filesystem property 'size' from the previous line. As one would expect for any 'normal' file...

--> Successfully opened the file for WRITE ACCESS

Now that line was added by me as my diagnostics code goes through the motions at OS level to open the s3db file for WRITING instead of only READING. The OS just told us: "fine. here you go. Do your thing."

Nevertheless, SQLiteNET comes back with "unable to open database file" so somebody is pulling my leg and all parties point at SQLiteNET by now... We'll have to investigate that one further, alas.

    As this report is about a SQLite database error, it MAY be useful to search
    the Internet for generic help and/or discussion of the reported SQLite error:

    ( ref: https://sqlite.org/rescode.html )
    ( ref: https://sqlite.org/c3ref/c_abort.html )
    SQLite Error Code: 14
    Reported error code is NOT a SQLite Extended Error Code.
    SQLite HResult: -21474816018:X

This is me and my diagnostics report code telling us that SQLiteNET (which, you'll see below, has been configured to output the most elaborate set of errors possible: SetExtendedResultCodes(true)) is bluntly returning a very generic error code: number 14. (The HResult is the translation of that one, where my diagnostics report code has a integer-to-hexadecimal-string-representation conversion bug, which has to be fixed, but only relevant insofar as to fix the -21474816018:X bogus output (something about signed 32 integers and unsigned values, yada yada yada). This number is printed earlier in the log file as 0x800007FF, which does not help us any further than the SQLiteNET error code '14'.

Given the report output, we also know that SQLiteNET didn't deem this worthy of additional error info: 'NOT a SQLite Extended Error Code'. So 'unable to open database file' is about all the information we get for this. Bugger. RTFC is in order to discover why. (RTFC=Read The F** Code; geek-speak for if you want to know what's going down, the source code is your only chance. Very dark. So real. 😄

Let's have a look if there's more to learn from the log:

      SQLite: the define constants (i.e. compile-time options): INTEROP_EXTENSION_FUNCTIONS INTEROP_FTS5_EXTENSION INTEROP_JSON1_EXTENSION INTEROP_PERCENTILE_EXTENSION INTEROP_REGEXP_EXTENSION INTEROP_SESSION_EXTENSION INTEROP_SHA1_EXTENSION INTEROP_TOTYPE_EXTENSION INTEROP_VIRTUAL_TABLE NET_46 PRELOAD_NATIVE_LIBRARY THROW_ON_DISPOSED TRACE TRACE_PRELOAD TRACE_SHARED TRACE_WARNING USE_INTEROP_DLL USE_PREPARE_V2 WINDOWS
      SQLite: the underlying SQLite core library: 3.32.1
      SQLite: SQLITE_SOURCE_ID: 2020-05-25 16:19:56 0c1fcf4711a2e66c813aed38cf41cd3e2123ee8eb6db98118086764c4ba83350
      SQLite: the compile-time options: COMPILER=msvc-1900 ENABLE_API_ARMOR ENABLE_COLUMN_METADATA ENABLE_DBSTAT_VTAB ENABLE_FTS3 ENABLE_LOAD_EXTENSION ENABLE_MEMORY_MANAGEMENT ENABLE_PREUPDATE_HOOK ENABLE_RTREE ENABLE_SESSION ENABLE_STAT4 ENABLE_STMTVTAB MAX_ATTACHED=30 SOUNDEX THREADSAFE=1 USE_URI WIN32_MALLOC
      SQLite: the version of the interop SQLite assembly: 1.0.113.0
      SQLite: the unique identifier for the source checkout used to build the interop assembly: 1911e60e5ee59d01f841dbe35dfd8e5104eae8c8 2020-05-30 14:28:05 UTC
      SQLite: the compile-time options used to compile the SQLite interop assembly: CODEC EXTENSION_FUNCTIONS JSON1_EXTENSION PERCENTILE_EXTENSION REGEXP_EXTENSION SESSION_EXTENSION SHA1_EXTENSION TOTYPE_EXTENSION VERSION_NUMBER=3032001 VIRTUAL_TABLE
      SQLite: the version of the managed components: 1.0.113.0
      SQLite: the unique identifier for the source checkout used to build the managed components: 1911e60e5ee59d01f841dbe35dfd8e5104eae8c8 2020-05-30 14:28:05 UTC
      SQLite: the extra connection flags: None
      SQLite: the default connection flags: Default

All that stuff is me dumping every version number or other release markers I can get my grubby paws on to help me investigate if possibly maybe there's a DLL gone sideways or other installed software crap happenin'. Here? No problem at all. Clean as a whistle. All numbers match the ones on my dev box and are what your Qiqqa installer installed. No surprises there. Good noises.

      (ref: https://sqlite.org/c3ref/extended_result_codes.html )
      SQLite Extended Error Reporting has been ENABLED: SetExtendedResultCodes(true)
---------

That's just me telling folks that this SQLite has been specifically told to turn on advanced error reporting via the API it has provided for that. Nothing special. Just added in the very last release you got from me.

Onwards.

--> VERDICT OK(?): this DOES look like a very normal file.

That is me combining a few of the geeky bits about the file and saying "this is about what I expect to see at OS level for a file that Qiqqa might be working on". As you noted, here's that "normal" again, but this time it is me with my perspective, instead of a deep Windows API with a very specific, different perspective of what's "normal".

Hence this is fine and leads me to conclude that the problem is somewhere hiding in the internals of SQLiteNET (or even SQLite itself!) and I have to find out what might be other reasons for SQLite to dump an error 14 "cannot open" in my lap on one machine, while another does just fine.

I allude that, very indirectly, in the next bit of the logfile:

    HOWEVER, it may have incorrect a.k.a. 'corrupted' content, which made Qiqqa barf,
    or there's something else going on which this simply diagnosis routine
    is unable to unearth.

    Please file a report at the Qiqqa issue tracker and include this logging
    for further inspection.

==================== End of diagnostics report ============================================

For completeness, there's also question what Qiqqa has to say about the sync process after this just happened. So let's pick out the relevant bits of the rest of the log and read those:

20201102.173830 [Q] ERROR [6] [27.495M] Unhandled Exception Handler: Something unexpected has happened, but it's okay. unable to open database file - You can continue working, but we would appreciate it if you would send us some feedback on what you were doing when this happened.

System.Data.SQLite.SQLiteException
Message: unable to open database file
ErrorCode: 0x0000000E
HResult: 0x800007FF
Source: System.Data.SQLite
StackTrace:    at Qiqqa.DocumentLibrary.IntranetLibraryStuff.IntranetLibraryDB.GetIntranetLibraryItemsSummary()
   at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder_Intranet.BuildFromRemote(WebLibraryDetail web_library_detail, SynchronisationStates& synchronisation_states)
   at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder.BuildFromRemote(WebLibraryDetail web_library_detail, SynchronisationStates& synchronisation_states)
   at Qiqqa.Synchronisation.MetadataSync.SynchronisationStateBuilder.Build(WebLibraryDetail web_library_detail, Dictionary`2 historical_sync_file)
   at Qiqqa.Synchronisation.BusinessLogic.LibrarySyncManager.SynchronizeMetadata_INTERNAL_BACKGROUND(WebLibraryDetail web_library_detail, Boolean restricted_metadata_sync, Boolean is_readonly)
TargetSite: Void Open(System.String, System.String, System.Data.SQLite.SQLiteConnectionFlags, System.Data.SQLite.SQLiteOpenFlagsEnum, Int32, Boolean)

20201102.173830 [Q] INFO  [6] [22.635M] About to display client stats: Something unexpected has happened, but it's okay. unable to open database file

That's the code in Qiqqa catching that SQLite error, uttering some noises but nothing extra useful.

From this we at least know that Qiqqa hasn't given up the ghost entirely, so we should see some more sync noises in the log...

20201102.173956 [Q] DEBUG [6] [25.951M] SYNC_META:INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73:Saving sync history (0/0: 0.0%)
20201102.173956 [Q] INFO  [6] [26.961M] Getting list of documents in Intranet Library \\NAS\Si Work\Papers & Books\QiqqaSync
20201102.173958 [Q] INFO  [6] [25.380M] Got LOCAL list of documents (1873) in Intranet Library \\NAS\Si Work\Papers & Books\QiqqaSync
20201102.173958 [Q] INFO  [6] [25.627M] Downloading documents for SimonDedmanNASsync
20201102.173958 [Q] INFO  [6] [25.627M] Queueing 0 PDF download requests.
20201102.173958 [Q] DEBUG [6] [26.421M] SYNC_META:INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73:Finished sync of SimonDedmanNASsync (0/0: 0.0%)

That last line is indeed Qiqqa telling us it's now done with SYNC of the SimonDedmanNASsync library: nothing on the remote side, a DB error at the start of that process, so nothing that can be done. Hence no mention of files or records processed. Expected, yet undesired.

However, a possibly fun fact are the log lines before that one: that's ANOTHER library of yours, as far as I can tell ('QiqqaSync' instead of 'SimonDedmanNASsync') being synced successfully!? So the error seems very specific to a single library in a larger set, or am I reading this wrong?

Anyway, like I said before: the plot thickens and I'll need to dig into SQLiteNET to maybe find out what's going on for real.

Meanwhile, I hope you have a spot of luck with Action 2 (RENAME/DELETE s3db + sync_md5 files, reconstructing the sync that way). Keep me posted and thanks for hanging in there!) 👍

20201102.173958 [Q] INFO  [32] [26.822M] +Saving known Web Libraries to C:\Users\simon\AppData\Local\Quantisle\Qiqqa\Guest\Qiqqa.known_web_libraries
20201102.173958 [Q] INFO  [32] [27.626M] -Saving known Web Libraries to C:\Users\simon\AppData\Local\Quantisle\Qiqqa\Guest\Qiqqa.known_web_libraries

Bit of an aside, but that Qiqqa.known_web_libraries file is where Qiqqa stores the actual Sync Point path: if anyone ever loses that file, they loose the ability for Qiqqa to 'discover' what the Sync Point is for each library and you end up with them all being flagged as local libraries only.

I'll need to do more work on this stuff. (Last weekend's slew of changes saw the old commercial Qiqqa cruft being regarding this being kicked out finally -- it's permeated through the source code. 😞 ) Folks should be able to assign or change the Sync Point of their library and point them at any NAS drive or Cloud Storage they may prefer instead to sync their libs across multiple machines. Alas, that's Future Music. right now we have a very odd Qiqqa library issue here.

I wrote this up also in hopes that maybe one day someone else has need for a bit of "how to read this verbose logfile?" re Qiqqa libraries and the underlying database technology. Let's pray it useful then. 😄

[GerHobbelt]

😭 Last message near the end: I'm wrong. The 'fun fact' I mention there is me not watching out carefully enough: it's the same library, only the local side of its sync process still being able to collect the set of PDFs and then saying there's nothing to do (heck, yeah, s3db remote DB could not be opened, so no sync work can be done!)

The Library name is SimonDedmanNASsync, while it's sync directory on the NAS is called QiqqaSync. That's fine.

Github allows me to EDIT my messages, but I'll the above last message as it is; if I turn this in a document, that 'fun fact' paragraph (and question) should be scratched as it's completely wrong.

Time to spin down here. Catch some Z's. 😴 😴

Best of luck with Action 2 (and 1 too, of course) and let me know how it goes.

[SimonDedman]

Cheers again Ger. I have a big work deadline this afternoon so will get back to this immediately after. S

On Mon, 2 Nov 2020 at 15:33, Ger Hobbelt notifications@github.com wrote:

😭 Last message near the end: I'm wrong. The 'fun fact' I mention there is me not watching out carefully enough: it's the same library, only the local side of its sync process still being able to collect the set of PDFs and then saying there's nothing to do (heck, yeah, s3db remote DB could not be opened, so no sync work can be done!)

The Library name is SimonDedmanNASsync, while it's sync directory on the NAS is called QiqqaSync. That's fine.

Github allows me to EDIT my messages, but I'll the above last message as it is; if I turn this in a document, that 'fun fact' paragraph (and question) should be scratched as it's completely wrong.

Time to spin down here. Catch some Z's. 😴 😴

Best of luck with Action 2 (and 1 too, of course) and let me know how it goes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-720785415, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABDC7RGLL26MAXKN3OC5PDDSN46TDANCNFSM4S36UMAQ .

[SimonDedman]

Hi bud, Right, working through this now,

1. sqdb file just emailed to you

2. /A/B. backup everything then rename Qiqqa.LibraryMetadata.s3db & sync from HOME mechine to have it rebuild the s3db: just done. I did this a few weeks ago, and it works flawlessly. HOME is back to normal.

3. In OFFICE: Forget intranet library in Qiqqa, delete that folder in /Quantile/Qiqqa/ (inc sync.md5). Reopen qiqqa, join intranet library, full sync. Oddly when I join intranet library, I'm then greeted with a blank Home screen:

Nonetheless, manually open intranet library from the menu, sync details,

Start sync: same issue:

Log attached. Qiqqa-20201105.015324.LOG

The confusion continues! :)

Cheers mate

[GerHobbelt]

That empty screen is very weird. At least the Guest lib should be listed there, even if a Ui update would be buggy. This is definitely inexplicable and indicative of a bug somewhere deep.

Havent got a clue yet but this is an important finding as it shows more effects of what i expect to be a single underlying problem.

Writing this from off site, will only have time for this on saturday, so I hope you have time then as well for a couple of iterations tovdig into this.

Can you mail me the zipped logfiles of both machines? What i am looking for in there is (a) qiqqa startup logs where it reports about installed NET versions etc and i want to compare your machines. I also wonder what happened with the Guest library on your OFFICE machine as that empty Home screen is theoretically impossible as qiqqa SHOULD have created a guest lib if it doesnt exist already, so at least one row is expected, not zero rows. (correct behaviour would be 2 rows at least: Guest plus Intranet library)

From the sound of it now it looks like a machine specific problem so we will have to also investigate outside qiqqa itself: which (. NET) libraries etc have been installed and are there any errors with those?

This will take some digging. A very weird problem indeed.

PS does the Guest library exist on the OFFICE box? It is supposed to always contain at least 2 articles about Qiqqa.

Thanks for reporting an for your persistence.

On Thu, Nov 5, 2020, 03:04 Simon Dedman notifications@github.com wrote:

Hi bud, Right, working through this now,

1. sqdb file just emailed to you 2/A/B. backup everything then rename Qiqqa.LibraryMetadata.s3db & sync from HOME mechine to have it rebuild the s3db: just done. I did this a few weeks ago, and it works flawlessly. HOME is back to normal.
2. In OFFICE: Forget intranet library in Qiqqa, delete that folder in /Quantile/Qiqqa/ (inc sync.md5). Reopen qiqqa, join intranet library, full sync. Oddly when I join intranet library, I'm then greeted with a blank Home screen: [image: image] https://user-images.githubusercontent.com/4599748/98188023-2670e900-1f0a-11eb-8e50-906697c452c4.png Nonetheless, manually open intranet library from the menu, sync details, [image: image] https://user-images.githubusercontent.com/4599748/98188094-4d2f1f80-1f0a-11eb-96ff-5efa06e36182.png Start sync: same issue: [image: image] https://user-images.githubusercontent.com/4599748/98188197-7bacfa80-1f0a-11eb-8678-74f7b3d8bb86.png Log attached. Qiqqa-20201105.015324.LOG https://github.com/jimmejardine/qiqqa-open-source/files/5491455/Qiqqa-20201105.015324.LOG

The confusion continues! :)

Cheers mate

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-722078285, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHQUCEPMDPAA6D54RP3SOIB27ANCNFSM4S36UMAQ .

[SimonDedman]

Morning! HOME log attached, after opening qiqqa then syncing the intranet library successfully. Qiqqa.log

This is completely inconsequential I'm sure but github allows me to upload qiqqa.log on the HOME machine's firefox, but on the WORK machine I have to ranem it to qiqqa.LOG. Can't imagine why that would be...!

Re the blank screen: Notwithstanding it still might tie into other problems, that I was able to open the intranet library through the menu, means it was still present just invisible - the guest library was in the same state I believe. So I think this is more of a UI screen drawing glitch than a 'libraries are missing' bug, for what it's worth!

I wouldn't be at all surprised if the problem was with my OFFICE OS install somehow. It's a legit copy of Win10 but for some reason I don't fully trust it, and I can't put my finger on why. If you have any suggestions for programs for me to check the version numbers of, and compare HOME to OFFICE, hit me.

Cheers bud, hope the offsite's going well.

[GerHobbelt]

Very late reply, as with your other problem.

Totally crazy thought: is your office machine maybe using a different filesystem (different from NTFS). The. LOG required rename is a sign there is something very odd going on there. FAT32 instead of NTFS or something like that?? - - even while I seem to recall that Win10 /must/ run on NTFS.

Anyway, when we find out why that rename is required, I expect we have a pretty good chance we also have the qiqqa issue explained. To be looked into.

On Thu, Nov 5, 2020, 18:41 Simon Dedman notifications@github.com wrote:

Morning! HOME log attached, after opening qiqqa then syncing the intranet library successfully. Qiqqa.log https://github.com/jimmejardine/qiqqa-open-source/files/5496010/Qiqqa.log

This is completely inconsequential I'm sure but github allows me to upload qiqqa.log on the HOME machine's firefox, but on the WORK machine I have to ranem it to qiqqa.LOG. Can't imagine why that would be...!

Re the blank screen: Notwithstanding it still might tie into other problems, that I was able to open the intranet library through the menu, means it was still present just invisible - the guest library was in the same state I believe. So I think this is more of a UI screen drawing glitch than a 'libraries are missing' bug, for what it's worth!

I wouldn't be at all surprised if the problem was with my OFFICE OS install somehow. It's a legit copy of Win10 but for some reason I don't fully trust it, and I can't put my finger on why. If you have any suggestions for programs for me to check the version numbers of, and compare HOME to OFFICE, hit me.

Cheers bud, hope the offsite's going well.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-722530966, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHTAKI4AMUR3IH4KQHTSOLPU5ANCNFSM4S36UMAQ .

[GerHobbelt]

Could let this go. Did some googling.

Seems that this: https://community.spiceworks.com/topic/2184726-restrict-certain-characters-in-file-and-folder-names might be a viable option for explaining your trouble, particularly because its an office box in a big "corporate" environment where sysadmins may be extremely risk-averse.

That message in the link and the reply it gets hints arbitrary file naming restrictions can be created using FSRM, which is File Server Resource Manager (googled that one too) which is a tool for system administrators.

Might be useful to ask your sysadmins if they apply such restrictions when configuring any office machines and/or machines connecting to their network.

Wh3n qiqqa inadvertedly happens to hit one of such (arbitrary) checks, that would explain the Access Denied problems we face on your OFFICE machine.

Long shot but certainly worth persuing.

Sysadmins are paranoid enough to do such a thing. No idea if they did but good to check.

On Tue, Nov 17, 2020, 03:02 Ger Hobbelt notifications@github.com wrote:

Very late reply, as with your other problem.

Totally crazy thought: is your office machine maybe using a different filesystem (different from NTFS). The. LOG required rename is a sign there is something very odd going on there. FAT32 instead of NTFS or something like that?? - - even while I seem to recall that Win10 /must/ run on NTFS.

Anyway, when we find out why that rename is required, I expect we have a pretty good chance we also have the qiqqa issue explained. To be looked into.

On Thu, Nov 5, 2020, 18:41 Simon Dedman notifications@github.com wrote:

Morning! HOME log attached, after opening qiqqa then syncing the intranet library successfully. Qiqqa.log < https://github.com/jimmejardine/qiqqa-open-source/files/5496010/Qiqqa.log>

This is completely inconsequential I'm sure but github allows me to upload qiqqa.log on the HOME machine's firefox, but on the WORK machine I have to ranem it to qiqqa.LOG. Can't imagine why that would be...!

Re the blank screen: Notwithstanding it still might tie into other problems, that I was able to open the intranet library through the menu, means it was still present just invisible - the guest library was in the same state I believe. So I think this is more of a UI screen drawing glitch than a 'libraries are missing' bug, for what it's worth!

I wouldn't be at all surprised if the problem was with my OFFICE OS install somehow. It's a legit copy of Win10 but for some reason I don't fully trust it, and I can't put my finger on why. If you have any suggestions for programs for me to check the version numbers of, and compare HOME to OFFICE, hit me.

Cheers bud, hope the offsite's going well.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-722530966 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AADCIHTAKI4AMUR3IH4KQHTSOLPU5ANCNFSM4S36UMAQ

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-728637094, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHXCL7MX2FJBJKMYHM3SQHKT3ANCNFSM4S36UMAQ .

[GerHobbelt]

Could NOT let go. Anyway. Its late. Gotta Zzzz.

On Tue, Nov 17, 2020, 03:33 Ger Hobbelt ger@hobbelt.com wrote:

Could let this go. Did some googling.

Seems that this: https://community.spiceworks.com/topic/2184726-restrict-certain-characters-in-file-and-folder-names might be a viable option for explaining your trouble, particularly because its an office box in a big "corporate" environment where sysadmins may be extremely risk-averse.

That message in the link and the reply it gets hints arbitrary file naming restrictions can be created using FSRM, which is File Server Resource Manager (googled that one too) which is a tool for system administrators.

Might be useful to ask your sysadmins if they apply such restrictions when configuring any office machines and/or machines connecting to their network.

Wh3n qiqqa inadvertedly happens to hit one of such (arbitrary) checks, that would explain the Access Denied problems we face on your OFFICE machine.

Long shot but certainly worth persuing.

Sysadmins are paranoid enough to do such a thing. No idea if they did but good to check.

On Tue, Nov 17, 2020, 03:02 Ger Hobbelt notifications@github.com wrote:

Very late reply, as with your other problem.

Totally crazy thought: is your office machine maybe using a different filesystem (different from NTFS). The. LOG required rename is a sign there is something very odd going on there. FAT32 instead of NTFS or something like that?? - - even while I seem to recall that Win10 /must/ run on NTFS.

Anyway, when we find out why that rename is required, I expect we have a pretty good chance we also have the qiqqa issue explained. To be looked into.

On Thu, Nov 5, 2020, 18:41 Simon Dedman notifications@github.com wrote:

Morning! HOME log attached, after opening qiqqa then syncing the intranet library successfully. Qiqqa.log < https://github.com/jimmejardine/qiqqa-open-source/files/5496010/Qiqqa.log

This is completely inconsequential I'm sure but github allows me to upload qiqqa.log on the HOME machine's firefox, but on the WORK machine I have to ranem it to qiqqa.LOG. Can't imagine why that would be...!

Re the blank screen: Notwithstanding it still might tie into other problems, that I was able to open the intranet library through the menu, means it was still present just invisible - the guest library was in the same state I believe. So I think this is more of a UI screen drawing glitch than a 'libraries are missing' bug, for what it's worth!

I wouldn't be at all surprised if the problem was with my OFFICE OS install somehow. It's a legit copy of Win10 but for some reason I don't fully trust it, and I can't put my finger on why. If you have any suggestions for programs for me to check the version numbers of, and compare HOME to OFFICE, hit me.

Cheers bud, hope the offsite's going well.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub < https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-722530966 , or unsubscribe < https://github.com/notifications/unsubscribe-auth/AADCIHTAKI4AMUR3IH4KQHTSOLPU5ANCNFSM4S36UMAQ

.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-728637094, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHXCL7MX2FJBJKMYHM3SQHKT3ANCNFSM4S36UMAQ .

[SimonDedman]

Interesting that the LOG thing could show the way. My intuition was that that issue more likely related to github since the file is called "Qiqqa.log" on both machines. AFAIK both machines are NTFS - as you say, I'm not aware it CAN be anything else.

Re the spiceworks link: MAYBE but I'd be very surprised tbh. While I'm at a big institution, as soon as I said "linux" they prettymuch just handed me the hardware. (The, "oh, you're one of them" approach). So IIRC I installed xubuntu on the barebones and then installed win10 myself as the secondary boot partition.

But I also installed firefox myself and it's setup the same way on both machines. Both are xubuntu & win10. Both have qiqqa, which saves as C:\Users\simon\AppData\Local\Quantisle\Qiqqa\Logs\Qiqqa.log on both. Yet github invokes the "We don't support that file type. Try again with a GIF, JPEG, JPG, PNG, DOCX, GZ, LOG, PDF, PPTX, TXT, XLSX or ZIP" error for the work machine but not the home machine. On the home machine, github invokes that error for "tmp.BOOGABOOGA" so the github filtering rule is working on the home machine, but seems to be ok with log=LOG, whereas it isn't on the work machine.

Presumably we should be able to diagnose this solely using the Qiqqa.log files, their permissions, formats, filesystem access? Home machine Qiqqa.log details attached, I'll do the work machine as soon as poss.

[SimonDedman]

Certainly some things are different on the work machine, though this MAY be due to different Qiqqa versions? Notably: Newer Qiqqa on work machine has !Qiqqa.ErrorsAndUserActions.log & Qiqqa-20201117.180831.log but not Qiqqa.log; I suspect this is a deliberate change you made during this diagnostic process, but just in case it's relevant.

Difference: on the WORK machine, the option to "encrypt contents to secure data" is greyed out. All else the same. Win10 build WORK: 19041.572, win 10 home version 2004. Trying NTFS disable encryption regedit fix now & rebooting.

[SimonDedman]

Home machine: build is 19041.264, win 10 pro, version 2004. Work machine should be pro, dealing with MS now to get it upgraded since I already activated it with them over the phone. Maybe that'll fix things. Though you'd think win10home shouldn't be a problem...

[GerHobbelt]

Yup, the new log filenames for Qiqqa is me. Every session gets a timestamped rotating log set, plus there is the errors-only log file for collecting error reports without the warn+info+debug noise from the rotated logs.

On Tue, Nov 17, 2020, 19:21 Simon Dedman notifications@github.com wrote:

Certainly some things are different on the work machine, though this MAY be due to different Qiqqa versions? Notably: Newer Qiqqa on work machine has !Qiqqa.ErrorsAndUserActions.log & Qiqqa-20201117.180831.log but not Qiqqa.log; I suspect this is a deliberate change you made during this diagnostic process, but just in case it's relevant. [image: qiqqaNasSync4] https://user-images.githubusercontent.com/4599748/99430063-c104fb00-2900-11eb-904b-6582cc46e3c3.png

Difference: on the WORK machine, the option to "encrypt contents to secure data" is greyed out. All else the same. Win10 build WORK: 19041.572, win 10 home version 2004. Trying NTFS disable encryption regedit fix now & rebooting.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-729113498, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHTIYDM7OCP2WPPIMGLSQK5IJANCNFSM4S36UMAQ .

[SimonDedman]

Hi Ger, merry xmas. I had to reinstall my Home machine due to hardware issues. Didn't have the opportunity to back anything up but everything should be / is saved to the NAS anyway so that hopefully shouldn't matter. However, having installed the latest version of qiqqa (the test release for this issue) the home machine now shows the same issue as the work machine did/does:

Ok, I copied the entirety of the NAS library to a local directory, added THAT as an intranet library, and the sync looks to work. So while this is frustrating insofar as my (only) working (networked) setup on my home machine is now gone,

1. The local setup will work for now at least
2. That the same files work when local but not through a network seems like a vital diagnostic datum for solving this.

The only thing that I can think of is: Could Qiqqa be struggling to interpret the file address of the NAS files, vs the local ones? C:\Users\simon\Documents\QiqqaSync\Qiqqa.LibraryMetadata.s3db vs \NAS\Si Work\Papers & Books\QiqqaSync\Qiqqa.LibraryMetadata.s3db

A. There are no spaces in the local location (but there could be if I had spaces in my own folders) B. Maybe it's looking for a ":\" and doesn't like "\"? Any kinda of regex parsing buried in there that might hate the double escape character string? C. When I access the NAS via windows explorer I initially had to enter my username and password, which was saved. Could it be that access is granted for windows explorer but not for Qiqqa? Even though when I import the intranet library it reads the correct library name from the file, so it presumably has access to that file at least? But maybe that's laundered through windows explorer, and qiqqa can't brose the rest on its own??

Another thought: I setup network qiqqa on home machine successfully using qiqqa v82. I THINK this failed on the work machine. But since the latest version fails on my home machine when networked, I'm minded to trry it out just in case. But before I do that I figure I'll wait to see if you have any thoughts about my diagnoses & where we could go next.

Cheers bud, and a happy new year!

[GerHobbelt]

Happy new year to you too!

It's very late again, so just jotting down here what I just found that made me go "hmmmm" as the magic word for me was the SqliteException, meaning that either I am feeding SQLite.NET something in the database open call that it doesn't like (like possibly your suggestions about paths) or SQLite itself does something I don't know about and then gets shafted on the network drive for reasons only known to SQLite and the OS. 🤔

Anyway, a google for that Error 14 dug up this one: https://stackoverflow.com/questions/21343380/why-am-i-getting-an-sqlite3-error-14-unable-to-open-database-when-calling-step which is not an answer but definitely a strong hint at the root cause. In particular the chown comment in there made me go 🤔 🤔 🤔

Note to self: https://www.pantz.org/software/sqlite/unabletoopendbsqliteerror.html Indeed. The SO page also mentioned temp dir. Be very aware that we are not running in a UNIX realm plus we're using SQLite.NET instead of sqlite, so the bets are off re "temporary path(s)" until we've uncovered exactly where everything goes.

[Edit]: the SQLite.NET we use is a wrapper around native sqlite. Now native sqlite3_open_v2 API doc points at error codes, where our Error 14 is SQLITE_CANTOPEN so that's a bingo. Fortunately their documentation also mentions temporary files, same as mentioned in the SO page that got this started just now. Candidates are 2.2 and 2.3: https://www.sqlite.org/tempfiles.html#walfile

Whatever the precise error-causing item is there, section 5 of that file (https://www.sqlite.org/tempfiles.html#tempdir) is relevant. Meanwhile the docs about changing it (https://www.sqlite.org/c3ref/temp_directory.html) discourage such practices and whether that'll fix it or not, I still wonder why. 🤔 Might need to code a special test version which attempts to write new files in the NAS directory where the sqlite db is stored and see if we get these chown-related errors (i.e. "access denied").

To be worked on...

Anyway, that's probably one step further towards a solution there...

[GerHobbelt]

BTW, re your A-C suggestions and 1+2: A & B would be prime candidates if this was bash scripts or some such, but with C# I don't have those high on my suspects list. (Doesn't mean it's not an option, it's just low in the current ratings. (As you may have noticed, I now focus more on the error 14 then before; context is important and sometimes I just change my mind. 😄 )

Here the context being that your copying of the files to local disk did wonders (no errors!), so that proves it's not in the file contents as they exist on the NAS: it's something to do with the network access...

Now I seem to recall memory mapped I/O over (SAMBA or NFS or ...) network drives was not possible, but I cannot find anything about that in the online docs for Windows or UNIX and my memory is more unreliable than I would like. That "seem to recall" stuff is (or so my brain tells me now 🤡 ) from back in the day when the books from Richard W. Stevens (R.I.P.) were my favorite literature, so that pretty much dates it and a lot has changed, not just socket and network programming. 😄

Anyway, the WAL and SHM files mentioned in the sqlite docs are prime suspect until proven not guilty. 😉 And as I said in my previous msg: while this may be the root cause, it's still curious...

[SimonDedman]

Ger, hello mate, long time, apologies for the crazily slow reply, I've been drowning in work recently.

Good hunting so far. Happy to test any test version of the software if you have any ideas for things which might confirm or deny your hypotheses. Is it worth me trying the latest release? I don't know if you've made any edits relating to this issue since we last spoke?

Cheers fella, hope you're well.

[GerHobbelt]

Hi Simon,

Do note that the latest releases (v83.something) are empathetically marked as unpublished: they are the very bleeding edge and geared mostly towards a couple of folks with serious crash trouble around large Qiqqa libraries. I went totally cowboy with those: they won't nuke your libs (I hope 😉 ) but expect crashes in odd spots as those come straight off the compiler with very minimal testing. The key point of them being tests to see if I at least removed their crashes (and maybe replaced them by a couple of others, that I can then attend to next). Heck, hard to explain exactly, but I consider the v83-series far less "rugged" than the v82 series.

Of course, if you're feeling adventurous, be my guest! 😉 (Went a version is too bothersome for some, you can always re-install latest v82, which should be "good enough for most use".

I was a out of ideas with your NAS problem, but having had it on the backburner, I got some new thoughts on "sync" that might resolve the trouble. (Trouble with that "new idea" is that I'm intent on fundamentally backpedaling to an approach used for network file locking which was developed back in the day of lore and flaky network file systems, which did not support file locking at all (antique NFS). Of course, that comes with its own dangers, which I want to scenario analyze thoroughly so I don't inadvertently destroy people's synced metadata DBs. Will feed you with a few test binaries soon -- expect somewhere around end of next week: I'm in the middle of a hairy migration, where the SORAX PDF render lib was taken out of Qiqqa (issues #9 / #7 / #136 / #209) and replaced by Artifex' mupdf tool. Which resulted in a whole chain of faults popping up.

Anyway, will keep you posted.

Cheers,

Ger

[SimonDedman]

Hi bud, hope you're well. No progress on this issue I take it? I just tried again with the latest build and still no joy. I hid the existing synced libraries and 'started fresh' but still nothing.

Also been having a few weird issues with the previous version I was running - no search results for existing papers but then they are produced when I search again a feww minutes later. Also sometimes every paper I open is blank. We're see if those issues vanish with the new version though.

Since this started back in October I guess I've added 206 papers, mostly PDF, and it occurs to me that IDK what the process will be to re-sync this to the NAS, since these are currently in C:\Users\simon\AppData\Local\Quantisle\Qiqqa\INTRANET_0E4255D5-3AFB-4522-BDD4-74E0E5C23A73 (the name got autogenerated after I backup/hid it, then reinstated it, just now). [note to self: 2020.10.21 tag = new papers at/before then.

Cheers!

[GerHobbelt]

[Edit: edited the Markdown quickly; email reply is !@#$%^&*\ -- this was email, which has been edited into the comment below. This comment didn't allow itself to be formatted. WTF?!]

[GerHobbelt]

[Edit: edited the Markdown quickly; email reply is !@#$%^&*]

Hi Simon,

No progress yet, sorry for that. It's been on my mind all the while but I didn't see a good way through it yet, so paid attention to other matters which are bothering me / Qiqqa for a long time.

Regrettably I'm still kinda stuck re the networking/sync process and how to tackle it to fix/improve it, so that you (and others) can benefit. The hard part here is getting myself mentally unstuck.

Been busy with backend stuff (mupdf: rendering PDF pages as images for viewing/reading on screen + using mupdf tools to extract text, OCR pages (they have incorporated tesseract in their toolchain lately, looks nice) and metadata, all to replace the old tools that did this work under the hood and haven't seen an update or fix in a, ääh, quite some time. Bit of a chicken-and-egg problem there as you can't easily swap out core components like that. I'm happy about where this is going though.

Re your sync problem and the other issues you're observing:

Sync:

The sync issue is quite particular. While this has been lurking in the back of my mind, I think it's somehow "magically" related to SQLite and how it specifically accesses and uses database files, which MAY cause obscure issues over non-native drive connections if the internet is to believe. The argument that popped up in my brain for this a while ago, despite it being surrounded in an extreme amount vagueness, is that a SYNC operation is really TWO actions:

• update the "remote" database, i.e. the sqlite db file on the NAS

• copy the "missing" PDF files to the remote storage's "documents/..." directory tree.

  and I never heard any complaints about PDFs not showing up or being damaged on the remote storage (NAS) for ANYONE, so that's a hint that a straight FILE COPY works as expected, while SQLite's "random access" I/O while it's working on its database, SHOULD be perfectly fine but somehow b0rks over a network connection? (Even though that network connection should be technically invisible as any transmit/receive errors are recovered at the network protocol level (TCP) and any non-recoverable errors there should percolate up all the way into the application layer and result in clear error messages, not just bloody error 14. Grmbl. That's me rambling about my thoughts on this, no proof, no hard data, nothing. And that irks quite some!

  So the "solution" I'm considering (and FEARING a bit TBH) is to change the SYNC process like this:

  • SIGNAL LOCK the remote DB file
  • COPY the remote db file to local scratch space (temp directory, that sort of thing, on LOCAL DISK)
  • do the SQlite work with that one, so that all SqLite activity is LOCAL DISK ONLY.
  • COPY the local updated db file to remote storage, using a NEW NAME.
  • do a (hopefully as close to atomic as possible) DOUBLE RENAME/MOVE on the remote. (CS majors will agree: "nearly atomic" is bloody identical to "NOT atomic AT ALL". Dangerous grounds to tread // meet corrupted files ahead. Got the T-shirt and all that jazz. Trying to keep it in the cupboard, though.)
  • check everything went okay on the remote, otherwise barf a hairball and call in the paramedics. (better make sure this never happens! This's what has me worried.)
  • update the DOCUMENTS on remote as usual.
  • delete the SIGNAL LOCK on the remote

The scary bit there is that:

• this is not something you just code and ship. Any measly little error in there will get me into a heap of trouble, so rigorous testing is preferred. (Guess what, no remote storage like that currently in use around these premises. Something about a zapped bit of electronics that I never bothered to fixed. Using Google Drive and suchlike these days. Hmmmm. Side order of Cascade of Sh*te, anyone?)
• that "SIGNAL LOCK" stuff in that procedure is not really locking in the true CS meaning as I cannot rely on the remote medium being able to do that sort of thing. So I'll be riding on old concepts we use for least common denominator environments: "lock files": if the file exists, you're supposed to wait. THAT is a heap of trouble and I could write entire chapters about. "Atomic operations over arbitrary network-based file systems" is a major headache for everyone. (And Samba and such are pretty near "arbitrary" for this matter, alas.)

An alternative is as tricky to get right: lock file or no lock file, you can always go in and do a file swap (hopefully atomic, but all bets are off there too!) and that's another chapter or book about the same problem really: atomic operations in arbitrary networked file systems.

As my other work on Qiqqa/MuPDF/tesseract/etc. is taking time too, I need to come up with something that unlocks me from this going-circles-in-my-brain state. Still pondering and not happy about it, but I MAY feed you a hacked-together 'Technology Test' tool to test on your system to help me test the viability of the new sync mechanism I'm considering. The big bother here is that I can't explain in any reasonable way why your system is rejecting the current process this way, which could be ignored if it weren't a strong hint that I don't know what to expect when I change a critical systeem like the SYNC process. Gotta start somewhere.

(Yep, going circles in the brain. Folks intimate with Qiqqa internals as-it-currently-exists-live-on-systems will rightfully object to me that the current Qiqqa doesn't have ANY locking provision either, so the corruption-by-sync-from-two+-sites issue is EXISTING and EVERYWHERE, definitely not NEW, so how about I'd say "I don't care now!" and barge on, replacing the existing system with a quality-comparable one? Whoops. You got me. Embarrassing. Not my best feature.)

Re no search results for existing papers but then they are produced when I search again a few minutes later.

Aha! When I first read this I was surprised. But now I write this reply, I realized what you're observing:

When a PDF is added to Qiqqa, it is queued in the background for text extraction, OCR action if that text extraction didn't succeed right away, some other metadata extraction and then on to the next big task: adding the extracted text + metadata to the LuceneNet search engine that's sitting in the back of the app.

All that background task business can take QUITE a while, minutes until done are not a surprise.

And only when the extracted text from that PDF has made it into the LuceneNet search index database will you be able to "find" anything in there. Up till that moment, the PDF document appears as simply NOT PRESENT from a search perspective.

Shame. Had forgotten about this as my own workflow is different: I work in "batches", hence scan and fetch PDFs that look like they have potential (from scanning their abstracts and such on Scholar or where-ever I happen to go at that time), which will load a batch of PDFs into those background processes. Fans spin up, etc. When that scan/collect winds down to a halt I usually have a few PDFs already on screen, picked during that scan process, for a further scan/reading following my own quick gander at the abstract while I hit that download/fetch button and this is why I don't have that search problem: while I read through a few of those papers, the background tasks are busy doing all that text extract+OCR+searchIndexing work and generally are (nearly) done by the time I hit the search box.

PLUS I suspect I have a hidden working assumption internalized in my workflow as I know how long these processes (can) take, thus unconsciously managing my expectations, so I wouldn't be surprised to NOT find the latest PDFs in my search results. Haven't checked/tested my behaviour like that. but now that you mention that, I consider my own workflows and notice that this has happened quite often but never bothered me somehow.

That last bit is a white lie! I did spend effort, repeatedly, digging through the code early on when qiqqa went public to improve the status reporting of the background processes: it's still a jumble (which I don't like), but it's a type of noise that gives me feedback about the "busyness" of the Qiqqa systems, which regulates my behaviour and expectations, so there's no surprise, but "merely" grumpiness about the machine not being fast enough as I try a few things I know should show up, just to see if the material has already been incorporated properly. So, yes, it DOES irritate me quite a lot, now that I think about it, in the same timespan as it does for you (and long after that!), but the knowledge WHY has a bigger impact on my treatment of this than I expected/guessed. Also my biggest bother is with the crap quality of the text extraction + OCR + indexing happening in there, thwarting the waiting itself as a source of irritation. Lots of factors contributing to that, including all the ways you can screw up a PDF (not intentionally).

Also sometimes every paper I open is blank.

Hmmmmm. Consistently so?

I've had LOADS of trouble with the old (= current) Qiqqa PDF viewer like that, which is based on a Sorax library to render PDF pages to images to show on screen for reading.

AFAICT there's two factors contributing here at about equal measure: PDF features/bits-and-pieces that b0rk the Sorax render process, and then some additional oddities in and around that lib I can't explain. (The latter of those two factors is of a temporary nature, i.e. re-opening the PDF often, not always, makes the problem go away. The first issue is a permanent/consistent one: several types of PDFs render as blank forever. (This is why I'm spending a LOT of effort on bulk testing the new muPDF render tools, BTW.))

The good news is that the new MuPDF-based code can handle PDFs a lot better.

The bad news is that I had to take a 12-gage shotgun to the Qiqqa code to make a MVP (Minimum Viable Product) sort of thingy from this, which currently has a few very obnoxious characteristics for people in a hurry:

• I had to nuke all the (cruddy) caching mechanisms in the code to get a clean path from PDF -> MuPDF -> Qiqqa code -> UI/UX on screen.
• The MuPDF process is technically SLOWER as it's interprocess instead of a .NET library, so that's a bit of a disadvantage to start with (MUCH easier to port or upgrade to 64bits platforms and Linux though!)
• Qiqqa is "dumb": it queues page render requests on a FIFO basis, even when you scroll and/or close PDF viewers, so there's plenty render requests in there that are totally useless by the time they are addressed
• That queue is shared on an equal footing basis with the background tasks, which employ it to render PDF pages as images to feed to the old tesseract engine (the Qiqqa you have still runs an antiquated tesseract 3.x version, that was turned into a .NET library years ago. Guess where that has gone ;-) -- this is why I'm working on making it all interprocess, so I, ultimately, don't have to keep .NET "wrapper libraries" alive that get too little attention and use worldwide.

The end result of this is that the bleeding edge codebase does work, has a totally b0rked text extract system right now (as I'm working on that now on the MuPDF side) but is increasingly slow, towards lethargic, in the PDF page view department as the queue fills with background OCR requests and now-already-obsolete render requests for pages you hurriedly scrolled through minutes ago.

To fix that, I have to code a bit of a caching and queue management layer on top of the new goodies (priority LIFO queue + image cache that expires with age, instead taking N slots per PDF like the old code). THAT hasn't been done yet. Stuff to do, places to go. :-)

[SimonDedman]

Hi mate, thanks for this.

mupdf updates

Sounds great dude!

SQlite performance, 2-step process, no proof, no hard data, nothing

[disclaimer: bear in mind this is all so far above my level that basically all I'm bringing to the table is logic and a can-do attitude] : would there be any scope to test these 2+ step assumptions with (a series of) tiny scripts? So e.g. I could backup my NAS copy and script 1 might try to change the name of sqlite db file on the NAS. Then script 2 might try to copy a blank text file to & from the NAS. Stuff like that, just to test permissions under the hood? Maybe first in something more basic than SQlite, if that makes sense, then again in SQlite to see if SQlite has permission problems despite permissions being ok otherwise? Edit: this is your 'Technology Test' tool. Sign me up! (whenever you have time)

PDFs not showing up, & nothing in search results

my sense was that the entire local library was 'frozen' for a while after the program starts. But I don't think this is happening any more since I installed the latest version. I think this may also have been due to loads of papers not having been ingested fully/properly e.g. OCR. I left it running for a few hours and I think the backlog is all cleared. Again, probably a function of me regularly messing around with a quite-big database, trying to diagnose the first issue, and causing new issues instead.

MUCH easier to port or upgrade to 64bits platforms and Linux though!

Be still my beating heart. CrossOver + MSOffice + Qiqqa would be amazing.

Stuff to do, places to go

Sounds like you're kicking arse buddy. I'll be cheering you on from the sidelines, mostly confused about what's happening ;)

Cheers!

[SimonDedman]

Potentially related: since installing the older pre-release version I noticed my watch folder wasn't present. When I go to add it again, the NAS isn't present in Network, in the folder browser. But it's present & open & I'm editing files within it using Windows Explorer:

Not sure why this would be, I'll restart to see if that helps, but this might be a clue regarding Qiqqa's problems connecting to the NAS sync library...

[SimonDedman]

Actually, looks like windows doesn't see the NAS now. Joy. Edit: fixed. Windows decided to turn SMB CIFS off, on a whim. Jerk OS.

[SimonDedman]

Re blank PDFs, I have this now on a recent paper:

Page 6 is blank, but has a normal thumbnail and the text seems to be indexed since it shows up in searches.

Edit: and then some minutes later it appears. Really odd.

[GerHobbelt]

That's something I haven't encountered yet. Could you send me that PDF file, please?

I'm pretty sure it'll be another SORAX library quirk, but I want to have a look to see what might be special and how mupdf page renderer deals with it.

Thanks!

On Mon, Apr 5, 2021, 22:02 Simon Dedman @.***> wrote:

Re blank PDFs, I have this now on a recent paper: [image: image] https://user-images.githubusercontent.com/4599748/113620323-180e3100-9652-11eb-90da-7b5a62b12ab4.png

Page 6 is blank, but has a normal thumbnail and the text seems to be indexed since it shows up in searches.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-813617370, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHRHNTXKNPWEXYSZVL3THIJMNANCNFSM4S36UMAQ .

[SimonDedman]

Hyndman&Khandakar.2008.Auto timeseries forescasting forecast package R.pdf

Enjoy!

[SimonDedman]

Hi bud, related to this, I've noticed that the watch folder is missing when I open Qiqqa. That folder is on the NAS also, so it may be that Qiqqa can't connect to it quickly enough and gives up and then deletes the folder location in its settings?

[GerHobbelt]

Qiqqa SHOULD never discard watch directories which are unreachable for whatever reason. SHOULD: haven't verified this in the source code right now (away from computer), but then again I also don't recall any sort of "cleanup" functionality either.

On Thu, Apr 8, 2021, 00:21 Simon Dedman @.***> wrote:

Hi bud, related to this, I've noticed that the watch folder is missing when I open Qiqqa. That folder is on the NAS also, so it may be that Qiqqa can't connect to it quickly enough and gives up and then deletes the folder location in its settings?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/jimmejardine/qiqqa-open-source/issues/257#issuecomment-815299021, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADCIHTRDBKBKBP6KREYPBTTHTKX5ANCNFSM4S36UMAQ .

[GerHobbelt]

Just a couple of notes-to-self re this issue - still unsolved 😢 :

Talked to someone today (off net, RL) who had tangled with quite similar issues regarding mapped network drives. His problems revolved around non-working MSAccess applications. Environment included BSD servers with Samba (SMB) over VPN and other environments.

• The final bit he needed was to add the network path to the Trusted Locations in the registry. --> checked this out, but AFAICT that only applies to Office applications and we are not an MSOffice application, so those registry entries (and policies) should not be relevant. Noting them here anyway, so I can tick them off the list as "looked at" when I run into them again.

• Other problem reported by him: network mapped paths would not be accessible at (user) machine startup until the user physically clicked in windows Explorer to open the mapped drive. Nothing else worked and he couldn't give a ticket reference but told me this happened a few years ago when the company switched their floor machines from XP to Windows 7. The problem appeared to occur with Vista and later (7, 8, 10) and was, reportedly, flagged by an MS rep as "won't fix". He solved it by writing a startup script which would poll to check if the drive was known already (after boot + login this would also take a random time and was hurdle 1), then it would attempt to look in the mapped drive using a DOS-style DIR command until that succeeded, which could take several more minutes before success, then he would check & reset those Trusted Locations registry entries (which would be erased by MS on Patch Tuesdays and arbitrary system updates), then he would run a sample MSAccess app which was written to report database I/O success/fail and quit -- put that app in a poll loop until success, and only then flagged the system as "ready" by starting the real MSAccess application used by the end user. SOP for users mentioned this as "login and click icon XYZ first thing in the morning, before you grab a coffee"

• other probably related material (as our own issue reports itself as SQLite error 14 -- unresolved):

  • https://github.com/dotnet/efcore/issues/19536
  • https://lavag.org/topic/18269-sqlite-and-network-drives/

[GerHobbelt]

❗ Oh, note-to-self about comment just above: check out the SQLite pragma comments once you get mupdf+sqlite C/C++ backend going: then we can test those and check them in relation to network I/O.

[SimonDedman]

Hey bud, happy winter. Have you had any further success with the MuPDF implementation? I'm basically unable to use Qiqqa at the moment because every pdf I open is blank. When I reopen qiqqa to that pdf, I get a blank or blurry first page and nothing else:

If I scroll away and come back, it's blank. If I close & reopen the pdf, blank. Looks like I'm using v82s; I think I rolled back due to other issues in other issue threads. But now I'm only making updates/edits/imports to Qiqqa at home, having given up trying to get the remote/NAS stuff working until I hear from you.

Cheers!