Problems syncing imported data

[xj25vm]

I can't work out from the description if TBSync is supposed to do a two way sync? I've got it configured and connected to a Horde back end, and it has downloaded data from Horde. But it doesn't seem to be pushing anything back from Thunderbird to Horde. Is it supposed to be able to, or not? Thank you

Edit: checking the TBSync account manager, I can see against the calendar status: "status.js-error-in-calendarsync.sendLocalChanges". So it looks like at least it is trying to update data back to Horde

[jobisoft]

In general, yes. TbSync is not automatically sending your local changes as you edit them, but only if you hit the sync button in the TbSync account manager or set up auto sync (sync every x minutes). This is not ideal, but the current implementation. At some later point I will implement push services, but do not ask me when, I cannot tell.

Do you see your changes pushed to the horde server, if you hit the sync button?

[jobisoft]

Can you retry with the latest beta TbSync v0.6.8.3 (2017.12.28) ?

Can i have a debug log of when that error (status.js-error-in-calendarsync.sendLocalChanges) happens?

[xj25vm]

Sorry for the delay. Could we get this reopened, as the error is still there. I've retried with 6.9.1 and I get the same error. Please see attached debug log.

[jobisoft]

Please try again with beta7.

[xj25vm]

With beta7, I get a different error. It counts down 30s on "Waiting for acknowledgement of local changes" and then "Communication timeout". Please note that this is a large calendar, with at least several thousand entries. However, all of them are already identical locally and at the far end. I only added one locally, and tried to sync it back into the backend. I am on a local lan, not through a slow vpn, in case it makes a difference. Please see attached log.

[jobisoft]

please increase the timeout:

Go to thunderbird options -> Advanced -> Edit Config and search for tbsync. You will get en entry for

extensions.tbsync.eas.timeout

which defaults to 30000 (ms) which are 30s. Try to set it higher and see if it makes any difference.

[jobisoft]

something is very odd, the log shows, that he wants to sync 100 and more old entries (from 2005) to the server. can you please start from scratch? disconnect the account and reconnect?

[xj25vm]

Sorry - I just realised that my comment above was wrong. The local copy contains all the entries, going back to 2005 - which I've loaded from a text file export. The back-end is empty. Does that make more sense?

Edit: I've increased the timeout and tried again. There are 9881 entries in the local calendar, which it is trying to sync to the backend. In my case, "Waiting for acknowledgement of local changes" seems to need anywhere between 35s-40s - so a timeout of 60s/60000 seems to work reliably. I can see now it is syncing 50 at a time. I guess it needs the longer timeout because of the large calendar, and this is not a fast machine (laptop with a Celeron 1017U cpu and a regular hdd, on Linux).

Has Lightning moved to using SQLite for its data format, or is it still using that old Mork format, which was slow on large data sets?

[jobisoft]

I am just using the Lighning API, so I do not know how things work under the hood.

So you no longer have issues with beta7? I will later increase the default timeout from 30s to 90s.

Can we close this issue?

[xj25vm]

Maybe I've spoken too soon. It is still in the process of syncing the 9000 calendar entries. Every few hundred entries, I get either "UNKNOWN_NETWORK_ERROR" or "Sync failed. Server responds with status <Sync.Collections.Responses.Add[25].Status = 6>."

If I press "Syncronize this account" again, it continues to transfer another few hundred entries, before I see one of the two errors above again. Does it look like a different problem? Should I open a separate bug report?

[jobisoft]

Keeping this issue is fine.

You have hit two bugs.

1. If TbSync prepared an element to be send to the server, he marks it as done right away and does not wait for ack from the server. So if the server fails to accept (like response.add.status=6), tbSync still thinks it has synced that item and will not send it again. This will be fixed in the next version. So if you clear your remote calender, start from scratch (disconnect/connect) and re-import your data in TB and sync again, all those errors come back.

2. You are importing data, so you are not using the Thunderbird GUI itself to create the calendar items. That's why your data "looks" different from what I am used to and I may have to add more checks, if an element property is actually there. I need at least the last 100 lines of a full debug log of such an error, most important the javascript error and the item data being send.

[xj25vm]

Please find attached the log for "UNKNOWN_NETWORK_ERROR". I left only the last 1600 lines (it had 36000 lines). Please let me know if you need any other data.

tbsync_debug3.log

[jobisoft]

There is not much I can do about that error, tbSync was not able to connect to your server (using standard Thunderbird API).

I can only fix javascript errors.

Any chance you could send me your import file via email, so I can try to reproduce this here directly?I would need the entire file, because we do not know, which entry is the bad one...

[xj25vm]

That's fine - I've sent you the calendar data as an email attachment. Many thanks for your help so far.

[jobisoft]

This is taking very long...

Is there a true need to sync 10 years old calendar data to the server? I am thinking about adding a default limit of 1 year, which can be changed in the settings. What do you think?

[xj25vm]

I think many calendaring software have a similar default limit when syncing older calendar entries - so it would make sense to limit the import/sync to 1 year by default - as long as it can be changed in the settings. I guess it might be still worth persisting with testing this particular data set, to find out why the sync process breaks?

As a side note, I have tested a lot of calendar sharing solutions over the years - and all of them had the same two weaknesses - they couldn't cope with large data sets, and they couldn't cope if the data was slightly different from what they expected - be it a missing field, or the date format, or other things.

At least it is good that the current TBSync interface gives a clear indication of progress - i.e. how many entries have been processed already. I have encountered a lot of calendar import/sync interfaces which just froze for hours on end while importing or syncing large data sets - because the developer never considered that the import or sync might take more than a few minutes - and never implemented appropriate progress indication in the interface.

[jobisoft]

Please check 16.4.2011, Alan Harris- iPad tuition

It has a reminder AFTER the event started. How did that happen? Intentionally?

EDIT: This is not an import issue, the alarm in the importfile is also after

[jobisoft]

@https://github.com/jobisoft/TbSync/issues/26#issuecomment-360471474 Yeah, corner cases are always a pain

[jobisoft]

item.startDate is undefined

Beautiful data set...

[jobisoft]

Please start from scratch* and retry with beta1

• disconnect account, remove all calendar entries on the server, reconnect, re-import, sync

[xj25vm]

item.startDate is undefined

Could I have the date for that appointment, please? I can't really think of any reason why that should be the case.

Please start from scratch* and retry with beta1

I will do and report back here

[jobisoft]

I did not identify the item in question, I just fixed the error. With beta1 I was able to sync the entire dataset to a Kopano server. I could not test with Horde myself.

[xj25vm]

Just an update - about 4000 records out of the 9000 have been synced, before starting to hit "UNKNOWN_NETWORK_ERROR" again. Restarting the sync manually seems to work. Is TBSync suppose to recover automatically after "UNKNOWN_NETWORK_ERROR" and continue?

[jobisoft]

Not at the moment, the sync stops after an error. I plan to implement a resume-on-error later. For now, you could simply set autosync to 1, which will trigger a sync every minute (if not currently syncing).

[xj25vm]

It has just stopped again after 600 records with "UNKNOWN_NETWORK_ERROR". The autosync is set to 15 minutes - but it doesn't seem to resume the sync - although it seems to try - as it shows the error again. Strangely enough, manually clicking the sync button does restart the sync - for a while.

[jobisoft]

I would like to handle the fail-to-resume in a different issue later. Currently it is important to know, if the sync fails due to javascript errors caused by corner cases - or if it runs through (only with network errors).

Please set the autosync to 1

[xj25vm]

I have set autosync to 1, and that indeed resumes the sync. However, now I am stuck on the same set of 50, which keep on failing with "sync failed. Server responds with status <Sync.Collections.Collection.Responses.Add[34].Status = 6>". It is retrying the same set of 50, but the number after "Add[]", between square brackets, is always different. Does that mean it is NOT failing on the same record every time?

After about 6-7 retries, it has done another 100, then it got stuck with the same error as above another 6-7 rounds of retries - now it is doing another 100 or so.

[xj25vm]

Please check 16.4.2011, Alan Harris- iPad tuition It has a reminder AFTER the event started. How did that happen? Intentionally?

Sorry, I didn't notice this comment earlier. The only reason I can think for this is that the appointment was moved after it was created backwards/earlier, and the reminder stayed where it was. I can't think of any other reason.

[jobisoft]

debug.log with those errors please (via email)

[jobisoft]

and stop testing, I have worked on some issues and release a new beta shortly.

[jobisoft]

Continue with beta2.

beta2 marks items as "done" only after we received an ACK from the server. So if you run into n error and hit sync again, it will retry with the exact same items. So, in case of NETWORK-ERROR it will go on (because the error was not with the items), but with javascript errors he will get stuck and throw the same error over and over again.

This might sound bad first, but it is the correct way. We need to fix javascript errors and we do not want to silently drop items. Furthermore, if you restart TB (and thus delete the debug log), you can reproduce the error just by hitting sync.

Can you verify this behaviour? If so, always restart Thunderbird after an error and reproduce the error via sync, to minimize the debug.log.

[xj25vm]

OK - by the time I got to it, there were only 220 items left which have since synchronised. Do you want me to delete the calendar, re-import and re-start the sync with Beta 2?

[jobisoft]

sync the last 220, maybe the error pops up again.

If no more error pop up, a restart from scratch is the only option to get to the bottom of this bug..

[jobisoft]

oh, it is fully synced now already? Yeah, than please start from scratch.

[jobisoft]

Do you have any more errors syncing your imported data? Otherwise I would like to close this issue. The beta2 is no longer avail, use the latest official release of 0.7.3

[xj25vm]

Sorry - I didn't get a chance to re-run the sync yet. Could you keep this issue open a bit longer please - I hope to get to it in the next few days.

[jobisoft]

Sure.

[jobisoft]

Important: For your test please set the following advanced options:

extensions.tbsync.eas.synclimit = 0 extensions.tbsync.eas.maxitems = 1

How these options are set is described in the wiki. After you have set these values, restart Thunderbird before starting your test.

[xj25vm]

The value options for extensions.tbsync.eas.synclimit seems a little counter-intuitive. Wouldn't it have been possible to just have the number of days - with a maximum of 180, and 0 for 'all' - maybe?

[jobisoft]

These are advanced options which are explained in the wiki. They translate directly to EAS options and I do not want to add an extra "transformation" layer there. Also "6 month" cannot be converted to a fixed number of days.

[xj25vm]

OK - thank you for the explanation. Will there likely be an option in the UI to change these settings, at some point in time in the future?

The latest sync is still in progress - I will post here when it is done, or if I encounter errors.

[jobisoft]

I do not think I will add those options to the GUI, usually the default values for these are good. Furthermore the concept behind TbSync is "simple and works", without much need for the user to setup anything. Currently there are a few settings for contacts which will eventually go away as well (not my code).

If the defaults do not fit your needs, you can change them with just a few clicks. Adding those options to some "advanced config" tab in TbSync is a lot of work, which I do not want to spend.

[xj25vm]

And I've hit the first error of the sync - after about 2000 items. Please see attached log - it might contain several retries on the same item, I think. tbsync_debug_2018-01-31a.log

[jobisoft]

The test worked perfecty, there is exactly on item (which failed), which I can now analyse. What kind of server are you sending this to? Office 365? Horde? Kopano?

[xj25vm]

It's a Horde 5.2.10 back-end. There are another 7000 items to sync, though.

[jobisoft]

You should not be able to sync anymore items, because TbSync is stuck on the one you send me, right? I will send you a new version of TbSync, after I fixed the bug. Then you can continue until you encounter the next bug (or until the sync runs fine to the very end).

I am currently fixing another bug, so it might take a while.

[xj25vm]

Correct - the sync it stuck on that item. No hurry with regards to the fix - thank you for your help so far.

[jobisoft]

Any chance you could lockup the log files of your horde server, if there is any indication of what is causing the error? I am sending against an horde as well (need to lookup what version) and I can send the event without error. I compared the WBXML, they are identical (besides synckey and collection id of course).

[jobisoft]

Other approach: Can you mail me your ics file, which you are trying to import? Maybe with that file I will be able to reproduce the error.

[xj25vm]

It's exactly the same file I sent you before - I used the same one so that you can do tests on it if necessary.

I will try to rummage through the Horde log files to see if there is anything relevant there.