Access Violation: "" is an invalid integer

[benjbrandall]

Access Violation: "" is an invalid integer

This access violation happened when I tried to change the sync directory (previously tested and working ok) to my local Nextcloud folder, at /home/benjamin/Nextcloud/Tomboy. (never before tested)

Here is a video of how to replicate, because I think that the little stutters it does as it crashes will give us more evidence of what the culprit function or window is:

https://we.tl/b-ydKqpcXJsr

If I hit "OK" and don't abort, the message next to the file sync path in the settings window gets stuck on "Ok, please wait, might take a few minutes....", and the next time I open tomboy-ng after quitting, the sync repo is the same as it was before (unchanged, not what I was after).

[davidbannon]

If you can run that in the debugger, you will get a report of which line of code is causing it.

But I'd make a guess. As you are "joining a new repo" your revision number is zero. That number gets turned into a string and written to manifest. I reckon instead of 0, you are somehow getting '', thats is an empty string, blank, whatever. Now, in the existing file sync code, I check that its not blank, in fact must not be blank.

The (new) revision number is the only thing I can think of in the space you are playing that converts string to int.

Davo

[benjbrandall]

Right, my manifest does have rev="0":

<sync revision="1" server-id="C3561500-AA3B-46EC-9500-E8C6D92D1056">
  <note id="E325B331-9A96-4049-8BC1-84FB46F5FA8E" rev="0" />
  <note id="0B430A52-C425-4E7F-8A0C-E259468BD0AB" rev="0" />
  <note id="6F84A864-1B38-4D3B-A954-BE3139B9DF6B" rev="0" />
  <note id="0697E8F8-0CC9-4E86-8DE4-A10A89E519F8" rev="0" />
  <note id="b0a96cd9-89c8-4a32-8f2a-a67c34cd4f49" rev="1" />
  <note id="a2b05611-901b-4e81-a4fd-ac9e2a542d15" rev="1" />
  <note id="a4a966a9-4977-4360-a4aa-0eeddf09313d" rev="1" />
  <note id="ef64abab-3329-4bb0-b076-78a939547f10" rev="1" />
  <note id="f68173cb-5370-444f-9476-a3c10eac3255" rev="1" />

The exact same note repo opened in Tomboy (not -ng) worked without issue (maybe Nextcloud sync doesn't need any special work?! :grin: ). However, I'm suspicious about the dir structure. Pointing -ng to that folder a few times and syncing etc has created a few '0', '1' dirs in there and multiple manifests etc. Should this happen and is it related?

[davidbannon]

The way the (FileSync) remote rep works it that a a new dir is created under "0" every time there is a new revision. Any new files end up in that revision as well as the previous (remote) manifest.

So, when you first arrive, you look in the top level manifest, see you need a file mentioned there so, see ots revision number and you look in the directory of that revision name. There manifest file there can be used to back away from an error, layer by layer. I guess.

So, any rev dir that has no files or no (old) mainfet would be an error IMHO. (or, maybe a 'delete' ?)

File sync ....

Davo

On 28/08/18 17:08, benjbrandall wrote:

Right, my manifest does have rev="0":

| <note id="0B430A52-C425-4E7F-8A0C-E259468BD0AB" rev="0" /> <note id="6F84A864-1B38-4D3B-A954-BE3139B9DF6B" rev="0" /> <note id="0697E8F8-0CC9-4E86-8DE4-A10A89E519F8" rev="0" /> <note id="b0a96cd9-89c8-4a32-8f2a-a67c34cd4f49" rev="1" /> <note id="a2b05611-901b-4e81-a4fd-ac9e2a542d15" rev="1" /> <note id="a4a966a9-4977-4360-a4aa-0eeddf09313d" rev="1" /> <note id="ef64abab-3329-4bb0-b076-78a939547f10" rev="1" /> <note id="f68173cb-5370-444f-9476-a3c10eac3255" rev="1" /> |

The exact same note repo opened in Tomboy (not -ng) worked without issue (maybe Nextcloud sync doesn't need any special work?! 😁 ). However, I'm suspicious about the dir structure. Pointing -ng to that folder a few times and syncing etc has created a few '0', '1' dirs in there and multiple manifests etc. Should this happen and is it related?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tomboy-notes/tomboy-ng/issues/77#issuecomment-416475152, or mute the thread https://github.com/notifications/unsubscribe-auth/AF__MKOqZhP3AcwCkHeykBw4_OmreOjvks5uVOx7gaJpZM4WNOJW.

[erik-brangs]

My manifest.xml got corrupted somehow. It was either a very strange bug or the hardware is defective: the manifest.xml was overwritten with contents that belong to another file that has nothing to do with tomboy-ng. Anyway, this led me to try to recover by creating a new sync folder (because I can't recover the manifest.xml). So I created a new empty folder and synced into it from Windows. When trying to change my snyc folder on Linux to the new sync folder, the problem

TApplication.HandleException "" is an invalid integer

occurred. I'm using tomboy-ng 0.18b from 2018-08-20.

The revision number for all entries in the new sync folder is 0.

The debug output before the access violation is

Backup Dir Checked
TB_Sync - Lists Cleared
TB_Sync - ReadRemoteManifest Done
TB_Sync - ReadLocalManifest Done
TB_Sync - GetListLocalNotes Done
TB_Sync - ManifestList Created Done
Local sync f0c86260-aa87-4157-afd0-ae75ad22e4d4
In both manifests R=2014-07-25T19:39:36.4182110+02:00 L=2014-07-25T19:39:36.4182110+02:00
Local sync a89f4680-d80e-4027-8c54-680cf424a5ac
In both manifests R=2018-09-03T21:08:29.7760000+02:00 L=2018-08-27T23:32:40.2400000+02:00
TApplication.HandleException "" is an invalid integer
  Stack trace:
  $000000000048AB7C
  $000000000066FAF1
  $00000000005D9119
  $0000000000552883
  $00007F277746F123
Empty Lists created

After the problem occurs, indexing of the notes continues:

Looking for notes in [/home/ebrangs/.local/share/tomboy-ng/]
...
Finished indexing notes

[erik-brangs]

Creating a new sync folder on Linux and then trying to sync from it on Windows doesn't work either. The failure presents in another way, however. There isn't any crash but sync process simply seems to time out. In one case, cancelling worked. After restarting tomboy-ng, the sync folder was still the old one.

In all cases, I didn't create a new note directory on the local machine, i.e. I tried to sync the existing local notes with the newly created sync repo.

[erik-brangs]

If I create a new empty directory for the local notes on Windows and just sync from the newly created sync folder which only has the notes from Linux, it seems to work: no error occurs and at first glance all notes seem to be present.

[davidbannon]

Thanks Erik. This sounds horribly like a problem that I spent a long time chasing and thought fixed. It happens when a 'non standard' new sync is initiated. But exactly what that non standard condition is, I don't know. Ben (who started this ticket) and I agreed that his problem was the funny things he is doing (he's building a network sync model) but looks like I was wrong, that bug is still there.

I am, presently, building a totally new sync unit informed by a powerpoint sllde we found online that gives me a better hint on how Tomboy (the original) is intended to work. If nothing else, with better design (because I understand the sync process better now) it should be easier to test, debug and "certify". The current one you are using is too complicated by far.

However, I would be quite uncomfortable seeing a real user like yourself using the new engine until it has undergone a lot of further development and a lot of testing. My guess is several months, I'm personally on a month long holiday as of next week. I am pretty sure it is the bug I have experienced and chased in the past and it relates to "rejoining" a sync repo. That is, joining a repo when you already have some notes (that also exist in the repo) in your normal notes directory.

Tomboy usually ends up making a whole lot of duplicate notes when this happens, I was determined to avoid that mess in -ng but have still have not got it right I'm afraid.

I am quite sure there is no possibility of you having this problem once the repo's rev number is greater than zero and almost certain you won't see it if you start out with a client that has no notes before its initial sync.

However, I value your input Erik, do you want me to chase this problem now or can we wait until we throw the offending code away and start again ?

(The problem happens when we convert a string containing the revno to an integer so it can be manipulated. Somehow (???) some file seems to have a blank string instead of a '0' and that cannot be converted to an integer. I thought I'd found all possible places where it happens ....)

David

[erik-brangs]

What I really need is a way to recover from a broken, incomplete or missing manifest.xml. This seems to be the main problem that keeps happening to me and that I can't fix. Notes with broken XML aren't problematic, I can ignore/edit/recreate those.

If I could recover by restoring a suitable automatically-created manifest.xml backup or if there was a way to automatically create a manifest.xml from the notes, that would be fine. Then I wouldn't have to try to create a new sync repo and sync from there.

[davidbannon]

Erik, my first thought is we need to find out how you are ending up with a broken remote manifest. Are you seeing any other file problems on that particular drive ? Are you syncing two clients to it at the same time ? What physically is that drive ?

Restoring a previous revision should be trivial. The -ng keeps a copy of the previous manifest in the newly created revision directory. I'd need to have a think about the process before I'd recommend you do it manually. Its a question of how the client reacts if it finds itself a rev ahead of the server. A fresh client would be OK.

(The new one we are building is, effectivly stateless and would have no problem dealing with that.)

You could help heaps by identifying whats happening to that manifest file. You mentioned in one case it was partly overwritten by an unrelated file, that sounds like a serious file system error. Are you experiencing other errors on the same file system ?

David

[davidbannon]

Er, I did not mean to shout in above message ....

[davidbannon]

Running a 'find' at the top of my file sync repo shows me that there are more uniq notes than in my client repo. So rebuilding by just scanning the repo would bring back all the notes I have deleted since first sync.

find . | grep note | awk '{FS="/"};{print $4}' | sort | uniq | wc -l

Better to step back through the repo, from the highest rev back, looking for most recent intact manifest, using that and then adding what files seem to be new since then. We'll still miss any deletions that happened between now and the date of that last valid sync but thats closer to what user wants IMHO.

I'm proposing a separate, standalone tool, tb_fixmynotes, that can do this. Doing this would, perhaps, make a new remote manifest at the highest rev that sync directories exist for and so, not panic a client later connecting ? I'll test this against my test sync repo of 120 notes and see how reliable I can make it.

We could also try and extract info from a damaged manifest but that could be a risky business and would involve the user OK-ing each step. That might be v2. By reading the text and decoding the XML programatically, can recover a lot of data XML libraries spit out.

[erik-brangs]

The problems with the remote manifest might be caused by a faulty drive. It's on an external hard drive that has some other problems. I've already ordered a replacement.

However, I also had problems with the local manifest on Windows simply disappearing (i.e. the file was simply gone). That was on a local SSD which hasn't had any noticable failures. I've had multiple instances of this but most of them with the original Tomboy. It might be that this hasn't actually happened with tomboy-ng but I can't say for sure right now.

I'm syncing the clients sequentially, i.e. there are never two syncs running at the same time.

[davidbannon]

Erik, I have made a rough little tool that will rebuild a remote manifest based on the remote repos directories. If the directories (and contents) are intact, it can restore a filesync manifest. However, a few warnings -

1. Tested on Linux only
2. It will revive any notes you have deleted between the current revision and how ever far back you need to go to find a good manifest. A minimal problem but one I cannot get around.
3. By its nature (and the fact I run it on the remote drive) means it does quite a lot of disk i/o if you have a lot of notes. So, if run on failing hardware, could be a bad thing ! Its won't cause hardware to fail but will highlight any failures that are happening.
4. won't do anything to help a lost local manifest.

Erik, I have not personally experienced any of these manifest failures that seem to be dogging you. And I do a lot of syncing, both testing new code and normal use. I am not happy about the sync code in the version you are using but I believe it works OK, my concern is its not scalable (ie to Ben's network model) and its difficult to maintain just because its spaghetti code !

Let me know if you want to try my (potentially nasty) repair tool. It does some benign diagnosing as well as it's risky fix.

I will not be able to make you any binaries other than a 32bit linux one after Monday, I will be away from home for a month and only have a little netbook with me. Davo

[erik-brangs]

Thank you. I'd like to try the repair tool. I'd start with Linux 64bit. I suppose Linux 32bit shouldn't be a problem, either.

[davidbannon]

The main sync engine now rewritten providing better error detection and debugging as well as easy implementation of new transport models. So, this problem gone for good. Just tidying up issues list. I don't think I can help you any more but if I can, please reopen this ticket. Davo