Contact syncing not working on Mac OS

zakkak commented 5 years ago

Contacts appear on console when using -D but not in the Contacts.app

Possibly related to #54 #65

tasn commented 5 years ago

Could maybe also be #66. I hope I can finally fix #66 in the next few days which will hopefully fix this whole array of issues. Extremely annoying, I know, and I'm sorry. It's just that Radicale doesn't support vcard 4.0 and I therefore need to implement it myself.

zagatta commented 5 years ago

I have the same problem.

I can download the VCF from the local webserver but the contacts are not showing up in the Contacts app

tasn commented 5 years ago

Any interesting information when running etesync-dav with -D?

zagatta commented 5 years ago

I cannot interpret if it is interesting, but here it is. I (hopefully) removed all unique identifiers, since this is public:

etesync.log

tasn commented 5 years ago

I don't see anything relevant there. It works for me though. :| What version of etesync-dav are you using? MacOS version?

zakkak commented 5 years ago

Hi @tasn

When running with -D I can see my contacts printed on the terminal

$ mac-etesync-dav --version
EteSync DAV version:  0.6.0
Radicale Storage EteSync version:  0.8.0
Radicale version:  2.1.11

MacOS Mojave v10.14.5

tasn commented 5 years ago

I fixed #65 and a few other things which may be related. Could you please try again with v0.7.1?

zakkak commented 5 years ago

I am trying to setup carddav with the latest version and I am getting "Unable to verify account name or password".

When I get the above etesync-dav -D prints nothing

tasn commented 5 years ago

Unable to verify account name at what stage? What are you trying and what gives you the error?

tasn commented 5 years ago

Where are you getting the above though? In Contacts.app? So before anything is running? So let's get again as many details as possible, first of all what do you get with etesync-dav --version? Second, what are the exact settings you used in Contacts.app? Does calendar work or you haven't tried?

It looks like it doesn't even reach etesync-dav, so the problem may be before. Have you correctly set up SSL as instructed in: https://github.com/etesync/etesync-dav/blob/master/README.md#macos-mojave ?

zakkak commented 5 years ago

The issue turned out to be that etesync-dav was failing to load the correct configuration. Passing the configuration through command line fixes the above issue.

etesync-dav -C ~/Library/Application\ Support/etesync-dav/radicale.conf

Note that I am running etesync-dav from /Applications/etesync-dav.app/Contents/MacOS.

Despite allowing me to add the account I still can't see my contacts. I use the following configuration:

For completeness and future reference I respond to your questions below.

Unable to verify account name at what stage?

This is in the Internet Accounts -> New CardDav account dialog.

Where are you getting the above though? In Contacts.app?

No, in the System Preferences when adding a new CardDav account

What are you trying and what gives you the error?

I am trying to setup my MacOS to sync contacts with etesync.

So before anything is running?

Not sure what the question is here

So let's get again as many details as possible, first of all what do you get with etesync-dav --version?

$ ./etesync-dav --version
EteSync DAV version:  0.8.1
Radicale version:  2.1.11

Second, what are the exact settings you used in Contacts.app?

Does calendar work or you haven't tried?

I didn't try, but removing the account and trying to add it again fails with the same message

It looks like it doesn't even reach etesync-dav, so the problem may be before. Have you correctly set up SSL as instructed in: https://github.com/etesync/etesync-dav/blob/master/README.md#macos-mojave ?

Yes, that was done for the previous version.

tasn commented 5 years ago

You shouldn't be using the configuration file, etesync-dav has finally broken free from that. Are you using the etesync server on etesync.com? Or self-hosting? If the latter, please make sure ETESYNC_URL is correctly set when running.

tasn commented 5 years ago

Also, please take a look at https://github.com/etesync/etesync-dav/blob/master/macos-instructions.md You should be using "advanced" (according to those instructions).

tasn commented 5 years ago

Actually, I think I know what it may be. Because we no longer use the configuration file it's not picking up the ssl configuration. Try running etesync-dav like this: etesync-dav --ssl --certificate ~/Library/Application Support/etesync-dav/etesync.crt --key ~/Library/Application Support/etesync-dav/etesync.key

Assuming you used the same paths from the guide. Do things now work?

tasn commented 5 years ago

I think I fixed everything with 0.9.1, please try that.

Edit: I fixed the SSL issue, so calendar will now work again. While it looks like Contacts are syncing correctly, I also can't seem to figure out how to get them to work again. I wonder when this regression happened.

zakkak commented 5 years ago

Are you using the etesync server on etesync.com?

Yes

Also, please take a look at https://github.com/etesync/etesync-dav/blob/master/macos-instructions.md You should be using "advanced" (according to those instructions).

For CardDav it says "Account Type: Manual"

Actually, I think I know what it may be. Because we no longer use the configuration file it's not picking up the ssl configuration.

Passing the configuration file as an argument though made it work (without the need for etesync-dav --ssl --certificate ~/Library/Application Support/etesync-dav/etesync.crt --key ~/Library/Application Support/etesync-dav/etesync.key)

I think I fixed everything with 0.9.1, please try that.

Just downloaded and installed 0.9.1

$ ./etesync-dav --version
Traceback (most recent call last):
  File "etesync-dav", line 12, in <module>
ImportError: cannot import name 'SSL_KEY_FILE' from 'etesync_dav.config' (/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/_MEIvELx4s/etesync_dav/config.pyc)
[67061] Failed to execute script etesync-dav

$ ./etesync-dav --ssl --certificate ~/Library/Application\ Support/etesync-dav/etesync.crt --key ~/Library/Application\ Support/etesync-dav/etesync.key
Traceback (most recent call last):
  File "etesync-dav", line 12, in <module>
ImportError: cannot import name 'SSL_KEY_FILE' from 'etesync_dav.config' (/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/_MEIOexPu1/etesync_dav/config.pyc)
[67117] Failed to execute script etesync-dav

tasn commented 5 years ago

So weird. It worked on my Mac. I'll take a look tomorrow.

-- Tom

On Fri, 26 Jul 2019, 20:39 Foivos, notifications@github.com wrote:

Are you using the etesync server on etesync.com?

Yes

Also, please take a look at https://github.com/etesync/etesync-dav/blob/master/macos-instructions.md You should be using "advanced" (according to those instructions).

For CardDav it says "Account Type: Manual"

[image: image] https://user-images.githubusercontent.com/1435395/61977024-f1cb6d80-aff5-11e9-81cd-b6aca53ab1fe.png

Actually, I think I know what it may be. Because we no longer use the configuration file it's not picking up the ssl configuration.

Passing the configuration file as an argument though made it work (without the need for etesync-dav --ssl --certificate ~/Library/Application Support/etesync-dav/etesync.crt --key ~/Library/Application Support/etesync-dav/etesync.key)

I think I fixed everything with 0.9.1, please try that.

Just downloaded and installed 0.9.1

$ ./etesync-dav --version Traceback (most recent call last): File "etesync-dav", line 12, in ImportError: cannot import name 'SSL_KEY_FILE' from 'etesync_dav.config' (/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/_MEIvELx4s/etesync_dav/config.pyc) [67061] Failed to execute script etesync-dav

$ ./etesync-dav --ssl --certificate ~/Library/Application\ Support/etesync-dav/etesync.crt --key ~/Library/Application\ Support/etesync-dav/etesync.key Traceback (most recent call last): File "etesync-dav", line 12, in ImportError: cannot import name 'SSL_KEY_FILE' from 'etesync_dav.config' (/var/folders/s6/bpyt31h91h79xltdlc1v32600000gq/T/_MEIOexPu1/etesync_dav/config.pyc) [67117] Failed to execute script etesync-dav

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/etesync/etesync-dav/issues/70?email_source=notifications&email_token=AAA2Q7SBI5ZSL7US4IZJHWLQBNHGLA5CNFSM4HP6LLVKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD25Q3UI#issuecomment-515575249, or mute the thread https://github.com/notifications/unsubscribe-auth/AAA2Q7XQCOR3I4RIBVDQW63QBNHGLANCNFSM4HP6LLVA .

tasn commented 5 years ago

A couple of comments:

About my comment above regarding the "manual" - please ignore that. I was wrong, as you said. :)
I fixed things in 0.9.1, you no longer need to pass --ssl --certificate and --key, just run normally.

As for the issue. It seems like the binary I uploaded was malformed. I now uploaded a new binary that fixes this issue. Sorry about the noise.

Regarding the original issue (contact sync): I think there may still be a problem. I need to figure out this regression but you are right, it doesn't seem to be syncing for me either (only contacts and only on the mac).

zakkak commented 5 years ago

Could maybe also be #66. I hope I can finally fix #66 in the next few days which will hopefully fix this whole array of issues. Extremely annoying, I know, and I'm sorry. It's just that Radicale doesn't support vcard 4.0 and I therefore need to implement it myself.

It looks like the problem is exactly the opposite. Contants.app only supports vcard 3.0 (allthough I was able to successfully import a vcard 4.0 manually).

Note that I tried creating a contact in the Contacts.app and it was successfully synced with my android device (in the debug logs it appears it was added as a vcard 3.0). The other way around though is not working!

tasn commented 5 years ago

Are you sure it doesn't support 4.0? Got a link to support that? I couldn't find anything online. What you are describing still sounds like #66 to me. Radicale is reporting it's a 3.0 collection so Contacts.app is limiting you to only 3.0. I'm not sure, I really need to figure this one out.

zakkak commented 5 years ago

Are you sure it doesn't support 4.0? Got a link to support that?

I base my claim on the fact that when exporting a contact it is exported in v3.0 format. There is also this old link https://alessandrorossini.org/the-sad-story-of-the-vcard-format-and-its-lack-of-interoperability/ and an Apple "vCard4-as-vCard3" mention in https://www.davx5.com/faq/cant-manage-groups-on-device/

Radicale is reporting it's a 3.0 collection so Contacts.app is limiting you to only 3.0

I am not sure this is the case. Even if I export a contact manually (no etesync-dav involvement) I get a v3.0 vcard.

tasn commented 5 years ago

I don't know what vCard4-as-vCard3 actually means in practice, as I haven't been able to find information about it. Anyhow, v3 and v4 are fairly compatible, so I just need to figure out what's there that's breaking things. I wonder if there's a way to get debug output from Contacts.app.

zakkak commented 5 years ago

I wonder if there's a way to get debug output from Contacts.app.

Yes, that would be great. A google search gave me this defaults write com.apple.AddressBook ABShowDebugMenu -bool true, which enables a Debug menu in the app, however I don't see anything useful yet.

zakkak commented 5 years ago

From Contacts.app preferences

tasn commented 5 years ago

This picture makes me so sad. :(

zakkak commented 5 years ago

I have downloaded the vcf file from the webui and imported it to Contacts. The contacts were added (but twice each :/) and contacts containing unicode (Greek) characters where broken.

tasn commented 5 years ago

If memory serves, vCard v3.0 needs you to set the charset while v4 assumes utf-8, which would explain what you're seeing. I think at this point the only solution is to make etesync-dav serve only version 3.0 vCards (convert the 4.0 ones), and accept both. This will also fix #66 in a sense.

zakkak commented 5 years ago

Looks like your memory is correct

v3 https://tools.ietf.org/html/rfc2425#section-5.5

5.3. Required parameters

Required parameters: charset

The "charset" parameter is as defined in [RFC-2046] for other body parts. It is used to identify the default character set used within the body part.

v4 https://tools.ietf.org/html/rfc6350

3.1. Charset

The charset (see [RFC3536] for internationalization terminology) for vCard is UTF-8 as defined in [RFC3629]. There is no way to override this. It is invalid to specify a value other than "UTF-8" in the "charset" MIME parameter (see Section 10.1).

tasn commented 5 years ago

I figured. My last suggestion will fix this issue as well. It shouldn't be too hard to do I think, at least for the common cases.

zakkak commented 5 years ago

🤞

tasn commented 5 years ago

Just to confirm again what we already figured out before: Contacts.app just ignores version 4.0 vCards. I'm really considering at this point, to just move all of EteSync to vCard 3.0 as trying to use 4.0 has been nothing but pain, or at the very least, as said above, create a conversion layer in etesync-dav. I'll start with a very hacky attempt and let's see how it goes.

tasn commented 5 years ago

OK, I implemented a fairly horrible workaround that should hopefully fix issue. We just transform vCard 4.0 to 3.0 when serving them in etesync-dav. This change doesn't make me very happy, but it at least "fixes" this issue. In the future we can implement a better, more complete, transformation. What's even more annoying is that it used to work, and it just seems to be a regression with either macOS or potentially our code. I really don't know, but it looks like the mac is at fault.

tasn commented 5 years ago

Fixed in v0.11.0

zakkak commented 5 years ago

Thanks @tasn despite being "horrible" as you say, it works and that's good enough for now :)

tasn commented 5 years ago

@zakkak, well, it's in because it's "horrible" and "works". If it was horrible and broken I wouldn't have pushed it. ;)

J/K, thanks for the feedback, yeah, I think it'll do the trick for now. I still don't understand why it stopped working, but I guess they changed something in Mojave. Who knows.

lasscap commented 5 years ago

It workes for my now, too. But unfortunately only for one of my two address books. It is the one which I created later.

The other one is not even shown.

tasn commented 5 years ago

Not even shown where? In Contacts.app or https://localhost:37358 when you click on your username?

lasscap commented 5 years ago

Not shown in Contacts.app. In http://localhost:37359/ under my username it is shown.

tasn commented 5 years ago

Have you tried adding them again? Maybe there was an issue before. I tested here when I closed this issue, and it seemed to work, but I'm happy to test again early next week if you can confirm even adding again didn't fix it.

solarkraft commented 1 year ago

it works and that's good enough for now :)

Welp, until you try to use groups/lists/labels. macOS ignores 4.0 ones, Android supports both 4.0 and 3.0 and the EteSync web client ignores 3.0 ones. What a mess. The best (but still awful) workaround is to create all labels on macOS and just avoid using the web client. Oh man.

In the future we can implement a better, more complete, transformation

It would be nice if the future was here sooner rather than later, but it seems like this project is unmaintained ...

etesync / etesync-dav

Contact syncing not working on Mac OS #70