Problems retrieving CardDav entries with 0.34

[GitOTR]

I updated my installation from 0.27 to 0.34 the last days. I am using eM Client on Windows 8.1 for managing my CardDav address book. After the update syncing takes very long and there are only synced 100 of 250 contacts. It does not matter whether I use dav.php or card.php as url. I switched back to 0.27 and the problem is gone.

[evert]

Hi @GitOTR ,

Baikal got a new version of sabre/dav, which became a lot stricter in terms of what it considers valid vcards. My best guess is that you have an invalid vcard in your database that was accepted earlier, but now causing sabre/dav to throw an error.

If you're interested, you could export all your contacts and I could take a look at the export file and see if I spot the error and then try to repair it for you. Sorry that this is such a pain!

[evert]

No feedback received.

[GitOTR]

Sorry for taking so long to answer. But I was quite busy. As you may understand I don't like the idea of sending my complete private addresses to someone else. I cloned my old 0.27 installation and updated it to 0.34 once again. Afterwards I compared all entries in both address books in eM client. I did a VCF export and examined the missing contacts in 0.34 with them that are available in 0.34. There were two identical data records regarding the identifiers. But only one was visible in 0.34.

Example from 0.27 that was not transferred: BEGIN:VCARD VERSION:3.0 UID:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx X-WAB-GENDER:2 N:Name;Name;;; SORT-STRING:Name\, Name FN:Name Name TITLE:  ORG: ; EMAIL;TYPE=WORK,PREF:email@email.email TEL;TYPE=HOME,VOICE:+11 (111) 11111111 TEL;TYPE=CELL,PREF:+11 (111) 1111111 ADR;TYPE=HOME:;;Street 1/1;Town;;12345; PRODID:-//eM Client/6.0.24928.0 END:VCARD

In eM Client I copied the entry from 0.27 contact folder to 0.34 contact folder. After this the data record was also available in 0.34 and stays visible. I did this with the rest of the missing entries and there were also no problems.

This is the updated entry in 0.34 for the above one in 0.27: BEGIN:VCARD VERSION:4.0 UID:urn:uuid:xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx N;SORT-AS="Name, Name":Name;Name;; FN:Name Name TITLE:  ORG: ; SEX:1 EMAIL;TYPE=WORK;PREF=1:email@email.email TEL;TYPE=HOME,VOICE:+11 (111) 11111111 TEL;TYPE=CELL;PREF=1:+11 (111) 1111111 ADR;TYPE=HOME:;;Street 1/1;Town;;12345; PRODID:-//eM Client/6.0.24928.0 END:VCARD

So in the end I have all my data from 0.27 available in 0.34. But if I start eM Client it takes minutes to sync the database from Baikal to eM Client. Using the 0.27 database this process is done in seconds.

[evert]

Interesting. It's hard to say what the real cause of this is. The original vcard validates just fine here. Since you modified it manually, it's possible that the original bug is now gone. Could have been an unexpected hidden unicode character for example.

[GitOTR]

I checked the exported file in several editors and there is no hidden special character. But I cannot say what is store in the SQL database. I will try to update my 0.27 installation to 0.4x the next weekend and update my comments. But one thing is still not solved. Why does syncing take so much longer with 0.34 than with 0.27. May it be because a lot of profile pictures in my contacts?

[staabm]

@GitOTR could you create a blackfire.io profile of the slow operation/request?

[evert]

It's hard to say @GitOTR why it's slower. We've actually done a lot of optimizations in the past to actually make things faster. Are you just noticing that the initial sync is slow, or also subsequent syncs after the initial one? At the very least when you sync again, in theory it should be super fast compared to baikal 0.2.7.

[GitOTR]

I just took a look at blackfire.io. Could this also be used when the trace is triggered by an external program running independently from the website? The free version does not support tracing SQL. Wouldn't that be the interesting part?

It does not depend on whether the sync is done for the first time or I re-sync my database. Just found out that eM sync log does also contain the number of synced contacts. So I found out that the missing contacts in 0.34 were always in the database but not shown in eM. I will check whether I can find another CardDAV client to check whether this is a eM bug.

[GitOTR]

Yesterday I tried to update from 0.27 to 0.43 and had the same problem as described in the beginning. Missing entries and very slow speed when syncing.

I reverted back to 0.27 and did a fresh installation of 0.43 in a parallel folder. Then added both accounts to eM. I copied my cards from 0.27 account to 0.43 account. Afterwards I also copied my calendar data to the new database. With theses upgrade procedure the problems do not occur anymore. Syncing is as fast as with 0.27 and all my contacts are available. May there be a problem with the update script?

[GitOTR]

I just checked the db.sqlite after it was upgraded by Baikal 0.43. And there two addressbooks and calendars entries. addressbooks and addressbooks_1459009879 calendars and calendars_1459009879 Shouldn't the tables with the number be deleted after upgrade is complete? At least all other db.sqlite I checked do not have any duplicate entries like that.

[evert]

I decided to always keep the original tables, in the off chance that there is something wrong with the upgrade script. It's basically a 'last resort' type of thing. I wanted to minimize risk of data-loss.

You can safely delete those tables afterwards though.

[evert]

Hey @GitOTR,

want to give 0.4.4 a shot? I have a feeling it might solve all your problems

[GitOTR]

Unfortunately the update made things worse.

With 0.44 the calendar is not synced on iOS. I checked with Lightning on Win and there is no problem. This case I did not test with a fresh database. If I find some spare time on the weekend I will do this test too.

Regarding the contacts the update did not help. I tried two scenarios:

1. update of 0.27 database done by 0.44 -> contacts are only partly synced with iOS and eM, nevertheless in Baikal dash board the correct number is shown
2. copy contacts from 0.27 account to 0.44 account in eM -> eM shows all contacts, on iOS there is no contact synced.

One further idea is to check the behavior on real web-space and not only on my DiskStation.

I know it would be most helpful if I would send you my db.sqlite. But this is something that I really would like to avoid. I will think about this the next days.

[evert]

A charles session demonstrating this problem would also really help.

[evert]

Probably even more than giving me the sqlite db. Note that you don't need charles on the server. Just on a desktop computer, and then you can point iOS to use the charles proxy.

[evert]

I got your charles dump! Unfortunately it's helping me a lot less than I had hoped :(

What I'm seeing is that most requests are really fast. But then the client does a multiget response, and the server responds with:

Fatal error: Maximum execution time of 240 seconds exceeded in /volume1/web/baikal.test/vendor/sabre/vobject/lib/Property.php on line 252

The client only requests 50 vcards, so that's a LONG time for that many vCards. The only reasonable possibility I see now is that maybe...

You have some really really massive vCards in your system. Like, many megabytes and it's just taking a long time to parse all that. This is possible if your vCards have big attachments such as photos.

How big was your vCard export?

[GitOTR]

I have 151 contacts in my database. The size is 16 MB, after I deleted all photos from my contacts the size went down to 51 KB. The chunk of 50 vcards is also what I see when I sync the complete database for the first time with eM. After a period of time another 50 vcards will be synced. But afterwards the remaining vcards won't be synced. What is really interesting. When I copy my contacts from 0.27 to 0.43 database using copy & paste in eM the size of the database stays the same but eM is syncing all contacts in a short time. Somehow the migration script seems to do the update of vcards in another way than eM does when copying the contacts from one database to the other. The database without the photos will be synced with iOS without problems. All contacts and calendars are visible. With Baikal 0.27 the size of the database including photos is no problem. Does Baikal 0.44 use a different picture library than 0.27?

[evert]

No, we don't do anything specifically with pictures. But one big difference between the old and the new version, is that we now parse every vCard before sending it back to the client. We do this to figure out the version of the vcard (3 or 4). And we then convert to vCard 4 if a client requests it.

emClient specifically will always request vCard 4 if the server advertises support for it (we do). So my guess is that what's happening now is:

1. You upgraded from 0.27, and all vCards are stored as vCard 3.
2. emClient accesses the new server, asks for vCard 4.
3. We convert all vCards from 3 to 4 before sending it back to emClient.
4. This is taking too long.

The reason it's probably not slow when you let emClient import the data, is because emClient will send the vCards as version 4. And if emClient later on requests those same vCards, sabre/dav does not need to do the conversion.

So what can be done? I could

1. Create a cache for the vCard 3 -> 4 conversion, so the conversion only happens the first time.
2. Start profiling the parser and converter. Given a backup file of x MB I should be able to measure how long its taking, and where most of the time is spent. Based on that I should be able to create optimizations that make this fast enough for you so this will work.
3. Our plan was also to take every attachment/photo from any vCard, and take them out of the vCard and store them somewhere else. This will also make a huge difference, but also a bit more of a long-term feature.

We'd really like 3. Ideally I would like to avoid 1, but 2 is probably the easiest to achieve right now.

In the meantime, you can:

1. Increase the PHP time limit. This is not ideal, and it will still be slow... but chances are that the requests will eventually complete given enough time.
2. Upgrade to PHP 7. This will make a real difference.

[GitOTR]

Fine. So for eM the problem could be solved by the described procedure to update the vCards to v4.

Which vCard version does iOS request? Even with Baikal 0.44 and vCards v4 not all contacts are synced. Furthermore the calendar sync does not work anymore with the database including the photos. Could this be a result of the activation of the sync plugin?

I will check in my DiskStation how I could increase the PHP time limit. There is a new DiskStation OS available. But I don't know whether it supports PHP 7. I will give it a try after the first bugfix update.

[GitOTR]

JFTR. With Baikal 0.46 all problems are gone. Update from 0.27 went smooth and all data are available on the first run and also no performance problems anymore when fetching my contacts via CARDDav.

[evert]

Awesome! We did make some performance improvements, so it's great to hear that it has had an effect for you.