Calendar Provider sometimes returns deleted items (possibly related to Google sync)

[Gitoffthelawn]

Try these steps:

1. In Business Calendar 2 (BC2), create a local calendar.
2. In BC2, create a few events.
3. In BC2, delete one of those events.
4. Using Calendar Import / Export (the app in this repo), export the calendar.

If you don't have BC2, you can try whatever calendar app you normally use.

When I try this on Android KitKat, deleted events are exported(!). :-(

I tried using the integrated import / export functionality in BC2 and it had the same problem. When I used their app to import an exported calendar back in (both functions performed within BC2), all the deleted events reappeared. Major QC fail. Huge privacy issue and potentially a security issue.

Where is the problem? Is it an Android issue? A BC2 issue? A bug in this app?

Can this app somehow to be a solution to this serious issue?

[jgriffiths]

Please enable debug logging and attach a log of the export where a deleted item is exported.

[Gitoffthelawn]

@jgriffiths Debug logging for this app just outputs more data to logcat, correct? No special log file is created, right?

[Gitoffthelawn]

@jgriffiths Here is the logcat of an export in which deleted events from a local calendar are included in the export: http://pastebin.com/5KfdFtwk

[jgriffiths]

@Gitoffthelawn there is no logging from the app in that paste. After enabling debugging logging in the developer options of settings, try 'adb logcat | grep ICS_ " to extract the app logs.

No special log file is created, right?

Correct - enabling logging makes the logging visible in release builds through logcat, it doesn't create another file.

[Gitoffthelawn]

@jgriffiths I'll try to get that for you. Might take a little time, as I'm heading into meetings right now.

In the meantime, I did some research, and I'm starting to think this might be an Android bug for offline calendars. If it is, I'm not sure which versions of Android it pertains to, but it's definitely reproducible on KitKat.

If you have a moment, try using your favorite calendar app(s) to perform the steps in my initial post for this issue. If you will, post whether or not any deleted events appear in the export file.

@all If anyone else can try this too, that would be great. Just post which calendar app you tried, and which version of Android was used. Also indicate if there are any synced calendars on the device.

[Gitoffthelawn]

@jgriffiths OK, I've really got to go, but a couple more thoughts...

This could be an issue on devices where there are no synced calendars, and SyncAdapter is not running.

Also, does this app ignore database entries in which the DELETED or DIRTY flag(s) are set?

[Gitoffthelawn]

@jgriffiths I noticed that even before performing the export, the Calendar Selection screen shows the number of events, and it includes all events deleted in BC2.

[jgriffiths]

The app doesn't currently subscribe to calendar changes and so if left running while the calendar is changed externally, the number of active events can be wrong.

Also, does this app ignore database entries in which the DELETED or DIRTY flag(s) are set?

Deleted calendars are ignored. I don't think deleted events are meant to be returned by the calendar provider.

[Gitoffthelawn]

@all and @jgriffiths Any success in reproducing this? I still have a few more tests to conduct, but so far the problem has been exhibited with over 20 attempts. Every time, deleted events are exported.

I would like to know if this issue is something specific to a configuration, or if everyone is experiencing this issue.

[jgriffiths]

@Gitoffthelawn I have very very little time to work on this unfortunately. The time I do have is best spent on bugs reported with sufficient information to identify and fix them, I think - It seems there are multiple people who will report and investigate bugs, but only myself who will fix them right now :-)

I am also not willing/able to install other apps to investigate issues that may or may not be bugs, although I recognise the value in doing so. Maybe in the future if I have more time. In the meantime, I will try to look at fixing bugs where the issue is clearly incorrect and there are logs etc to help; I think this is of the most value to the community.

If you can supply a log that shows a deleted event being exported I can take a look.

[Gitoffthelawn]

@jgriffiths Not a problem. Thanks. I'll get you a log soon. Maybe it will provide insight into what is causing this issue.

I'm not sure, but I have a hunch this is a serious bug in Android itself. I also have a hunch that only someone with your skills will be able to figure that out (flattery, which in this case happens to be true, gets you everywhere). :-)

I have now tested this on two different Android devices from two different manufacturers with two different calendar apps, and they both reproduce the identical issue with a 100% rate of reproducibility.

If you (or any other generous soul) has time to investigate it, you can use whatever calendar app you normally use... you won't need to install any extra software. Here are the steps:

1. In your calendar app, create a local calendar.
2. In your calendar app, create a few events.
3. In your calendar app, delete one of those events.
4. Using Calendar Import / Export (the app in this repo), export the calendar.

[jgriffiths]

I have followed the instructions using Offline Calendar to create the calendar, and the deleted events are not exported - they are not returned by the calendar provider (as expected). I'm on the latest CyanogenMod nightly FWIW.

Before we can attempt to work around this in the export, we need to determine what is causing the issue. Since locally created calendars don't create UUIDs in the database for me, I don't think its reliable to use their presence/absence to export them.

I will need a log file as discussed above to progress.

[Gitoffthelawn]

@jgriffiths Thanks for your patience. :-)

I generated the log file you requested. Here it is: http://pastebin.com/C9MyHgvZ

Since I wanted to narrow down this issue, instead of using Business Calendar 2, this time I used aCalendar+. I got the same end results.

Steps I took (tested on Android KitKat):

1. Disabled Google Calendar Sync
2. Cleared cache of Calendar and Calendar Storage
3. Used aCalendar+ to create a new local calendar.
4. Created an event called "BUY STUFF".
5. Created an event called "GIVE TO CHARITY".
6. Deleted "BUY STUFF" event.
7. Opened Calendar Import / Export (this repo).
8. Selected the calendar.
9. Exported the calendar.

Results: It exported both events. Expected results: Only export "GIVE TO CHARITY" event.

The same issue transpired when performing identical tests with Business Calendar 2 Pro.

[jgriffiths]

@Gitoffthelawn Thanks for the log. I can confirm that there is no difference in the columns we query that can be used to differentiate one case from the other.

When I get a moment, I will add a further debugging option which may help discover a difference. Until then if you have any links describing this bug and the workaround of deleting the UUID, please add them here. There may be some other way to determine if the user wanted the event deleted.

Thanks for your help in tracking this issue down BTW.

[Gitoffthelawn]

@jgriffiths You're quite welcome, but more importantly, thank YOU!

Yes, the data looks identical for the fields being inspected. All my reports are from my testing and research; I have not found external links.

When you add the further debugging option, you may also consider adding a "refresh" button. That will make debugging and testing a little easier, but it's not hard to exit and reload the app, so it's not a big deal if it's too much work.

[jgriffiths]

@Gitoffthelawn I've added the debugging option ("Query all columns (Y/N)"), you'll need to take a new release to get it. When you enable it, please re-run your test and link the resulting log which should have a lot more dumped columns in it. Hopefully that will tell us something.

I'm not sure what you mean by a refresh button btw. If you can describe what it would do I may be able to find time to implement it at some point.

[Gitoffthelawn]

@jgriffiths Thanks for the debugging option. Will you be pushing it to F-Droid and Google Play?

By "refresh button", I mean a UI button for the app to reload the calendar database. That way, if the user keeps this app open while adding/deleting events in the calendar, they can press "refresh" (in this app) to see the event counter change in the UI.

[jgriffiths]

@Gitoffthelawn I've raised issue #47 for this. A refresh button is not the correct solution, but it should be possible to keep the calendar details up to date transparently to the user which is even better I think.

For releases we need to poke Dominik, I will ask him to re-release once #47 is fixed.

[Gitoffthelawn]

@jgriffiths I agree. I proposed a refresh button simply because it would be less work to implement. But if you make it automatic, all the better!

[fhaftmann]

Ready for release in 2.7.