All entries in calendar deleted

decibyte commented 9 years ago

I was pointed in this direction by Tanghus. Please forgive me if this is completely misplaced here.

Just before Christmas, my Jolla deleted everything in my calender 6 months back[1] and all future entries. I found a thread about this issue on TJC and posted a comment there.

I have no idea why this happened. I don't have much information to help debug or reproduce the problem. All I have is this extract from my ownCloud server log.

[1] I assume this 6 months limit is because the sync client doesn't sync further back.

tanghus commented 9 years ago

I was pointed in this direction by Tanghus. Please forgive me if this is completely misplaced here.

I'm @tanghus here, just so I can follow the thread :)

decibyte commented 9 years ago

I just realised that the most recent Sailfish OS (1.1.1.27) was released a few days before my issue. Maybe some of the fixes for CalDAV and ownCloud included in this release caused this?

chriadam commented 9 years ago

Previously, caldav account providers were treated as "services" rather than as "providers" in a&sso terminology. In 1.1.1.27 we ran a maintenance script on existing caldav accounts, to attempt to transition them "properly" to the new account types. It sounds like a bug in that script caused this issue (deletion of the events locally), and then a subsequent sync might pushed those deletions server-side... without more detailed logs / dump of your calendar db / etc I can't tell for sure, though.

I believe there were a few issues fixed thanks to a contribution from a community member, which should make it into the next update (update11). With these patches applied and a newly created account, I'm not aware of any issues in particular with sync reliability in update11.

I have also added support for the .well-known/caldav endpoint in the settings/accounts code, but that won't make it into update11 unfortunately - it'll be in update12.

After U11 is released, can you test again and report back any issues? I have bandwidth to investigate and fix CalDAV (and CardDAV) sync issues going forward.

Thanks!

decibyte commented 9 years ago

without more detailed logs / dump of your calendar db / etc I can't tell for sure, though

Could you give me directions on how to extract that information, if it is still available after I removed the calendar? As described in the TJC thread, my workaround was to remove the calendar account from my phone (which turned out to be quite cumbersome, as the calendar had multiplied extensively), share the calendar read-only with another ownCloud user and then use that account on my phone. Thus, I may have also removed some of the logs/dumps/DBs you're interested in?

After U11 is released, can you test again and report back any issues?

Sure thing! What exactly should I do? Would it be fine to add another read/write calendar account with some entries in it and check that everything is still there after I upgrade to U11, or do I need to do something else for this test?

caldav account providers were treated as "services" rather than as "providers" in a&sso terminology

Just out of curiosity, can you point me in the direction of further explanation of the difference?

chriadam commented 9 years ago

Logs can be retrieved by: systemctl --user stop msyncd MSYNCD_LOGGING_LEVEL=8 devel-su -p msyncd then open a new terminal: devel-su journalctl -af | grep caldav and then trigger a sync via the settings/accounts UI.

The databases of interest are the mkcal database (ie, the device calendar db used by the jolla-calendar app) found at /home/nemo/.local/share/system/privileged/Calendar/mkcal/db and the caldav sync anchors database found at /home/nemo/.local/share/system/privileged/Calendars/caldav-sync.db. Both of those are sqlite3 databases.

By testing with U11, I basically mean: a) adding a new account b) checking that synchronisation works c) checking that duplication (of events and of calendars) does not occur d) checking that removing an account causes those synced events/calendars to be removed

We do tests like that internally, but sometimes very minor differences in setup or process followed can cause big differences in outcomes, so the more "points of view" we have, the better IMO.

In regards to accounts&sso stuff, see the docs at http://docs.accounts-sso.googlecode.com/git/libaccounts-qt/html/index.html Basically, a Provider is something like Google, a Service is something like Google-Calendar, Google-Contacts, Google-GMail, Google-Chat, etc.

decibyte commented 9 years ago

Logs can be retrieved by (...)

The grep doesn't really produce anything. MSYNCD_LOGGING_LEVEL=8 devel-su -p msyncd gives a lot of output. Is that what you want?

I've extracted the DBs. Do you have an e-mail address and a PGP-key I can use to encrypt and share the files with you? I'm a bit concerned about sharing my private calendar openly on the internet :)

Interestingly, everything seems to still be there in the mkcal DB. Which is good, as I'm able to recover stuff from there then. But I think it's also bad that things from deleted calendars are still present. Shouldn't that be deleted along with the calendar?

I see the same entries multiple times. Not only for multiple calendars (probably related to issue #43), but also within the same calendar. Is it supposed to be like that?

By testing with U11, I basically mean (...)

But that doesn't really reproduce the use case I had that caused the wiping of my calendar, does it? I will of course still happily follow the steps described if it is of any help.

blizzz commented 9 years ago

I have seen this behaviour before 1.1.1.27. And it also happened twice.

chriadam commented 9 years ago

@decibyte the grep should produce a lot of logs, from the caldav plugin (assuming that the caldav plugin is built as an out-of-process plugin, which I ... believe it is). I'll double check that the grep should have picked up the appropriate output from the journal.

In regards to the databases, my email is chris dot adams at jollamobile dot com An ascii-armoured copy of my PGP public key is:

Would I have your permission to forward your database onto other Jolla developers (specifically, Pekka Vuorela and Bea Lam) to help me determine what went wrong?

The U11 testing may or may not cover the case which caused your calendar to be wiped previously. I have no idea what caused that to occur (although I'm hoping the databases can provide some clues). Without knowing what triggers the case, and why it happened, we can't say for sure whether the steps I mentioned will cover the error condition or not.

If you (or anyone else) know of a repro which predictably causes the wipe to happen, please let me know what it is, as that would be invaluable to fixing the issue.

Thanks, Chris

decibyte commented 9 years ago

I've sent the DB files to you.

Is there a way to downgrade to a previous release and do the upgrade again?

chriadam commented 9 years ago

Not as far as I'm aware - the upgrade can involve low-level changes which aren't cleanly reversible without doing a factory reset, I believe (although I may be wrong). Packages can be downgraded, obviously, but that may leave your device in an unusable state.

chriadam commented 9 years ago

I've discussed this with Bea, and we've arrived at a possible synopsis:

a) the account was transitioned during the upgrade from the old "caldav" account provider to the new "carddav or caldav" account provider. b) this resulted in the deletion of the data associated with that (old) account. c) the calendars (in mkcal terminology: notebooks) were deleted. d) events (in mkcal terminology: incidences) which are deleted are NOT purged by mkcal, instead they're kept and marked as "isDeleted=1" basically.

This would explain the duplication to some extent (as subsequent sync would result in a duplicate-but-not-deleted incidence in the notebook), although multiple duplicates would require multiple deletion-then-resynchronisation cycles, I think.

The "bad" part then follows: why were the deletions upsynced to your server? This I cannot answer, I cannot figure out why those operations would be considered local modifications, but I guess there must be a bug in the caldav plugin code which I haven't found yet.

decibyte commented 9 years ago

I've just upgraded to the new opt-in release. Now my caldav account has been duplicated (it's listed twice under Accounts). I'm happy I've set it up as read-only, as I fear this would have caused another wiping.

Can I share any logs or other data with you to help you investigate the duplication?

chriadam commented 9 years ago

The account has been duplicated? Can you run ag-tool list-accounts and then ag-tool list-settings <id> for the two accounts? You may need to devel-su -p pkcon install libaccounts-glib-tools first.

I can only imagine that the accounts upgrade script has a bug in it...

decibyte commented 9 years ago

list-accounts:

ID         Provider                       Name
--         --------                       ----
17         onlinesync                     mmm-readonly
16         onlinesync                     mmm-readonly
14         jabber                         3xm@detfalskested.dk
6          yahoo                          mmm
3          twitter                        decibyte
1          jolla                          decibyte

list-settings 16:

default_credentials_username = mmm-readonly
Jolla/segregated_credentials/Jolla = 15
CredentialsId = 15
CredentialsNeedUpdate = false
enabled = true
name = mmm-readonly
        onlinesync-caldav
CredentialsId = 15
calendar_colors = #9fc6e7
server_address = https://sky.cannedtuna.org
enabled_calendars = /remote.php/caldav/calendars/mmm-readonly/default%20calendar_shared_by_mmm/
caldav-sync/profile_id = caldav-sync-16
auth/method = password
calendars = /remote.php/caldav/calendars/mmm-readonly/default%20calendar_shared_by_mmm/
calendar_display_names = Kalender
enabled = true
auth/mechanism = password
sync_profile_templates = caldav-sync
        onlinesync-carddav
auth/method = password
server_address = https://sky.cannedtuna.org
auth/mechanism = password
CredentialsId = 15
carddav.Contacts/profile_id = carddav.Contacts-16
enabled = false
sync_profile_templates = ["carddav.Contacts"]

list-settings 17:

default_credentials_username = mmm-readonly
Jolla/segregated_credentials/Jolla = 16
CredentialsId = 16
CredentialsNeedUpdate = false
enabled = true
name = mmm-readonly
        onlinesync-caldav
CredentialsId = 16
calendar_colors = #9fc6e7
server_address = https://sky.cannedtuna.org
enabled_calendars = /remote.php/caldav/calendars/mmm-readonly/default%20calendar_shared_by_mmm/
caldav-sync/profile_id = caldav-sync-17
auth/method = password
calendars = /remote.php/caldav/calendars/mmm-readonly/default%20calendar_shared_by_mmm/
calendar_display_names = Kalender
auth/mechanism = password
sync_profile_templates = caldav-sync
        onlinesync-carddav
auth/method = password
server_address = https://sky.cannedtuna.org
auth/mechanism = password
CredentialsId = 16
carddav.Contacts/profile_id = carddav.Contacts-17
sync_profile_templates = carddav.Contacts

chriadam commented 9 years ago

Hrm, aside from some explicit enabled flags, the settings are all identical. The update script does a backup/restore of accounts (after performing some modifications), which might result in something like this if there is a bug which stops the "old" account from being removed once the process is complete. But, in this case, I wouldn't expect the transition code to be hit at all, since they do not differ in any settings (ie, no transition is required...).

We'll look into it, thanks for the report.

decibyte commented 9 years ago

I don't know if this is of any help, but I just upgraded to .16

I deleted one of the duplicate caldav accounts before the upgrade and there's still only 1 after the upgade.

chriadam commented 9 years ago

We completely updated the way we do delta detection (added/removed/modified sets) in the caldav plugin, so the deletions should no longer be erroneously upsynced. Can you confirm that this is no longer an issue, or are you still seeing this behaviour with the latest version of the plugin?

decibyte commented 9 years ago

@chriadam Unfortunately I lost my Jolla a couple of months ago, so I'm not able to test this anymore. At least not until I get a new one.

chriadam commented 9 years ago

No problem, thanks.

DagNygren commented 6 years ago

This still happens. Now running 2.1.3.7 on a Jolla-C and it happened to me last week. Do have backups and restored the freshest one. But also here deleted all entries 6 months back. Both from my OwnCloud account AND my Davical account.

nemomobile / buteo-sync-plugin-caldav

All entries in calendar deleted #45