Contacts - vCard import-export issues

[hansbkk]

I am doing some pretty detailed testing of several aspects of vCard and CardDAV integration, with both Android and desktop apps and tools. I am focusing on strict fidelity to the "google way" of implementing extended fields in its export from gContacts via vCards, which is obviously a major 800-lb gorilla de-facto standard in its own right, seems to be pretty compliant with the v3,0 RFC, and also (more importantly) is I believe in line with the "apple way", the other 800-lb gorilla.

Besides these initial reports, I plan to log others here as well as I find them, rather than separate issues for the moment.

But more importantly, I want to also find out if Cozy is implementing CardDav via code from another project, so I can also report upstream as appropriate. Edit - looks like Cozy's devs forked from this? https://github.com/mikedeboer/jsdav

[hansbkk]

1°) In the Google interface, there is a pre-defined pair of strings "Home fax" and "Work fax", which get exported as follows in the vCard:

TEL;TYPE=WORK;TYPE=FAX:212 555 2345 TEL;TYPE=HOME;TYPE=FAX:212 555 3456

These both show up in Cozy as "FAX" alone, so it is impossible to tell which is the home fax number and which is the work number.

My work-around solution for now is to enter these as a custom string rather than using Google's predefined labels, which will export as:

item1.TEL:212 555 2345 item1.X-ABLabel:Work Fax

and this works fine, but obviously it isn't ideal to require the user to have to remember which google labels work with Cozy and which don't.

[hansbkk]

Google's representation for birthday

BDAY:1961-04-05

gets dropped completely on import into Cozy

Cozy's vCard export represents it as:

X-;TYPE=BIRTHDAY:1961-04-05

[hansbkk]

3 Wow this is a big one! Every one of the IM handles from Google's export gets just silently dropped!

X-AIM:AIM X-GTALK:gTalk X-ICQ:ICQ X-JABBER:Jabber X-MSN:MSN X-QQ:QQ X-SKYPE:Skype X-YAHOO:Yahoo

Workaround would be to enter these in notes only.

Cozy exports them (Twitter & Skype only) as:

X-;TYPE=TWITTER:@AAAFirst X-;TYPE=SKYPE:AAFirstAAALast

and in fact Cozy uses the 'blank' "X-" label for company, all relationships, custom dates

rather than the more useful labels allowing grouping: item10.X-ABDATE:2019-12-31 item10.X-ABLabel:Arbitrary Date

and

item12.X-ABRELATEDNAMES:Big Brother item12.X-ABLabel:$!!$

[frankrousseau]

Wow! Thank you for that exhaustive bug report. We'll see how we can handle it.

[frankrousseau]

Yes, we forked jsDAV to make it progress faster. But mainly, CardDAV fields are managed by this library: https://github.com/cozy/cozy-vcard/

[hansbkk]

No worries. Do you want me to cross-reference with an issue logged against that specific library?

Some general observations from my initial research on this topic area (excuse me if much of this is already well-known by the devs).

The vCard / CardDAV "standards" (that's what's so great about standards there are so many of them 8-) are a real mess, and very hard to work with, given that most tools out there are still on (the very limited) v2.1, google and Apple implementations I believe are both on v3.0 and obviously v4.0 compatibility will be required down the road.

Between the two, Google's implementation of CardDAV is apparently a real mess and shouldn't be touched even with the proverbial barge pole - stick with vCard.

Also, IMO rather than trying to strictly implement the standards themselves, from a practical POV it would be best to work from real-world vCard exports from iOS and Google Contacts and make sure Cozy fully supports those fields. The next highest priorities IMO would be Outlook's exports, then Thunderbird address books + the SOGO plug-in, and syncevolution.

The real ideal would be fidelity to any and all custom extensions out there in the wild, but that may then make it difficult to implement a user-friendly web-based contact editor (is yours based on CardDavMATE/CalDavZAP ?)

Only within the higher-priority target of supporting these real-word implementations, should then strict compliance with the RFCs supported. IOW don't let adherence to the standards break compatibility with what your users will be trying to do.

Here is a current sample of a "complete" gContact record as exported in vCard format:

https://gist.github.com/hansbkk/b28f3bbb97b50c0275bd

Finally my opinion on the back-end storage issue. I think the CardDAV server should keep its data as a simple folder collection of .vcf files, one per contact, and the UID for each record as the filename - "UID.vcf."

This will allow a user who wants to implement another file transfer / sync protocol to access their contacts "database" via (s)ftp(s), WevDAV, NFS, BT Sync, FolderSync, unison, or whatever.

Also, integration with a VCS like git would be ideal for backup purposes, since it is very common for users to suffer data loss of their contact record using tools with different implementation standards.

I highly recommend the devs have a look also at some other implementations with design philosophy along these lines: servers: SabreDAV (used by Owncloud and many other projects), DaviCal, and Radicale the last very "light" and flexible http://radicale.org/technical_choices/#idid14

sync tools: http://github.com/untitaker/vdirsyncer http://ntbab.dyndns.org/apache2-default/seite/contactsync.html

dialer/editor: Contacts VCF (Android) CardDavMATE (web-based)

[jsilvestre]

Hi @hansbkk, thank you for that very details report.

If you don't mind, we'll continue the discussion on the dedicated repository (https://github.com/cozy/cozy-sync/issues and https://github.com/cozy/cozy-vard/issues) so it's easier for us to follow. I am going to create new issues and reference existing one here so we don't lose any of you thoughts and bug reports.

I created a new issue about your comment: https://github.com/cozy/cozy-home/issues/287#issuecomment-73349752, you will be able to follow it here: https://github.com/cozy/cozy-vcard/issues/7

I also created a new one for https://github.com/cozy/cozy-home/issues/287#issuecomment-73350314, you will be able to find it here https://github.com/cozy/cozy-vcard/issues/6

About your big one (https://github.com/cozy/cozy-home/issues/287#issuecomment-73350793), there is no way we can guess every single custom fields clients will add, so the best would be to implement a suggestion we have been made a while ago: https://github.com/cozy/cozy-sync/issues/26

Now some more generic information.

As you mentioned, vCard implementations are a damn mess. So the first thing to understand is that there is not a perfect solution. Also, note that we have limited resources and that we must also work on an email client, calendar synchronization, files client with desktop and mobile synchronization, all that without sacrificing the platform itself. So the next thing is that we target the MVP for most of those matters (most viable product), not perfection (at least for now).

That being said, I don't think we've hit the MVP on contacts sync yet, and anyways, there is always rooms for improvements, so I'm glad you push that matter.

vCard

We currently support 2.1 and 3.0 in import, but we export to 3.0 only. And yet we don't fully support the standard anyway. We have also implemented some specific apple/google/android/.../ cases.

The real problem here is that there is no complete vCard implementation in JavaScript, just a 4.0 (but nobody uses 4.0....). I am eagerly waiting for the one of FirefoxOS, they did a great job with the iCal one.

Also, since all providers are displeased by vCard, they started adding custom fields ("X-something"). It's great vCard is extensible, but we can't actually guess every single X-field from every single provider. Even doing it for the main providers represents a huge work (which is why we don't support all of them yet).

One possibility is to keep all "X-field" we don't support as-is so at least it's not lost during synchronization (see https://github.com/cozy/cozy-sync/issues/26).

Sync

Due to our architecture (both data and software), it's not the best solution to sync .vcf on the disk, because it's just not how Cozy works. All the data are kept in the "data system" (which is basically a database with permissions, using CouchDB to store the data themselves). It's as good as file system for whoever wants to implement another protocol, I'd say it's even better because you can actually request it by some criterions without changing the file names or the file tree. Speaking of that, this is exactly what we do: we have a separate application that handles WebDav synchronization. Another app could do it with another protocol :)

Our sync is based on jsDav, a JavaScript port of SabreDav (probably the best WebDav implementation out there), you figured the right tools (and so did we)!

The synchronization itself in Cozy works pretty well, the real work is in the vCard support.

Editor

It's a matter of coherence and technical feasability, I haven't looked at your suggestion yet, but I'm pretty sure they are not Node webapp written for Cozy ;-) As a matter of fact, we created our own. It's not perfect, but has been built to be extensible, so we can manage the "X-" fields at some point (it might already be the case, I can't remind).

The next step

I hope I answered all your questions, feel free to ask more. I suggest you open a new issue (https://github.com/cozy/cozy-vcard/issues) for all each property you want us to support, so it is easier for us to prioritize. We can't do it all right away, but we're committed to continuously improve!

Cheers.

[frankrousseau]

Thank you for all for that detailed information. That will really helps us to improve the VCard management. I think the best approach, as you suggest, is to import Google and Apple contacts and make it works.

[frankrousseau]

I close this issue. Things are now too messy. I'm going to open every related issue on Cozy VCard project.

Sorry @hansbkk for the inconvenience.

[hansbkk]

No problems here. . .

Currently active issues related to those raised here:

https://github.com/cozy/cozy-sync/issues/26#issuecomment-73535007