Contact Group information is lost and not Imported.

zgypa commented 1 year ago

v.1.0.8 Google Integrator will not import Google Groups.

Behavior

At the end of the import, all contacts are in the "Not Grouped" category, instead of being grouped like the vcf import does. Group information is not retained anywhere else in any other field.

Expected Behavior

Similar to VCF file import, i would expect new groups to be created, to match the ones in the Google Contacts.

julien-nc commented 1 year ago

Thanks for the remark. It is now done and will be included in the next release (coming soon).

Only user defined groups are imported (the ones presented as "labels" in the google contacts interface). So it does not include the "myContacts" one. Every contact is in this system group anyway :grin:.

julien-nc commented 1 year ago

This is already included in v1.0.9-1-nightly. It would be great if you could try it.

zgypa commented 1 year ago

Doesn't seem to be working. i'm testing 1.0.9-2-nightly.

I have a very large database, about 6.5k contacts. I have installed the nightly by:

downloading the build locally
disabling the current one with occ app:disable
Returns that 1.0.8 was disabled.
moving the current app out of the way with mv ./integration_google{,-1.0.8}
untaring the download in the nextcloud/apps directory
enabling it with occ app:enable integration_google which returned it enabled version 1.0.9

Import takes a very long time (about 30m). And during the import, i can see the contacts coming in, but none of them has a group assigned. I'm letting it run until the end, in case groups are supposed to get created at the end, but i doubt it.

Maybe i did someting wrong? Maybe the fix is in -1 but not -2?

Wait! Hold on, my bad. Actually, looks like the importer first start with all the contacts without a group, for some reason... Now i see contacts with groups coming in.

julien-nc commented 1 year ago

I'm letting it run until the end, in case groups are supposed to get created at the end, but i doubt it.

Indeed. Groups are not even created. It is a contact field/attribute. The contacts app deduces the groups from the contacts content.

Maybe i did someting wrong? Maybe the fix is in -1 but not -2?

It is also in 1.0.9-2-nightly

I tried again and it works fine on my side. How are your groups defined? Are you using the google labels?

zgypa commented 1 year ago

I guess you missed that last paragraph, which says that the Groups got created/imported, sorry for adding it at the end.

However, i have a new issue now: The import seems to stall at some 5400 contacts (can tell precisely, because the great GUI of Nextcloud just shows "54.." instead of the full number, but i should get about 6100, and i'm getting a whole lotta logs like this one:

[no app in context] Error: Exception: The arguments array must contain 2 items, 1 given in file '/usr/local/www/nextcloud/lib/private/L10N/L10NString.php' line 88 at <<closure>>

0. /usr/local/www/nextcloud/lib/private/AppFramework/App.php line 172
   OC\AppFramework\Http\Dispatcher->dispatch(OCA\Notification ... {}, "listNotifications")
1. /usr/local/www/nextcloud/lib/private/Route/Router.php line 298
   OC\AppFramework\App::main("OCA\\Notificati ... r", "listNotifications", OC\AppFramework\ ... {}, ["v2","ocs.notif ... "])
2. /usr/local/www/nextcloud/ocs/v1.php line 63
   OC\Route\Router->match("/ocsapp/apps/no ... s")
3. /usr/local/www/nextcloud/ocs/v2.php line 23
   require_once("/usr/local/www/nextcloud/ocs/v1.php")

GET /ocs/v2.php/apps/notifications/api/v2/notifications
from 10.1.49.83 by afm at 2022-12-22T15:29:28+00:00

julien-nc commented 1 year ago

Ok nice.

I think this notification error is not related with contact import because it does not trigger any notification. So either the google API does not return the full contact list or they are skipped by the integration_google NC app. Let's try to find out.

I added some debug log in the contact import. This is included in v1.0.9-3-nightly. Just change the "loglevel" NC config value to 0, trigger the import (in a fresh address book) and filter the NC server log with integration_google. You should see the skipped contacts and in the end those 2 messages:

N contacts seen (the actual number of contact we got from the API)
N contacts imported (the ones that were successfully created in NC)

zgypa commented 1 year ago

Well, i suppose the Google API does the return the full list for the following reasons:

The number of contacts in the Data Migration pane of Nextcloud shows the correct number (6162)
My previous attempt to import with 1.0.8 imported all of them.

As a matter of fact, it might not be related with import, as i have noticed they appear even when not importing anything. Here is one full log entry from 1.0.9-2-nightly:

{
    "reqId": "TVuYPHvJn7pMhkTSSsIs",
    "level": 3,
    "time": "2022-12-22T14:16:21+00:00",
    "remoteAddr": "192.168.48.1",
    "user": "afm",
    "app": "no app in context",
    "method": "GET",
    "url": "/ocs/v2.php/apps/notifications/api/v2/notifications?format=json",
    "message": "The arguments array must contain 2 items, 1 given in file '/usr/local/www/nextcloud/lib/private/L10N/L10NString.php' line 88",
    "userAgent": "Mozilla/5.0 (Macintosh) mirall/3.6.4git (build 12826) (Nextcloud, osx-20.6.0 ClientArchitecture: x86_64 OsArchitecture: x86_64)",
    "version": "25.0.2.3",
    "exception": {
        "Exception": "Exception",
        "Message": "The arguments array must contain 2 items, 1 given in file '/usr/local/www/nextcloud/lib/private/L10N/L10NString.php' line 88",
        "Code": 0,
        "Trace": [
            {
                "file": "/usr/local/www/nextcloud/lib/private/AppFramework/App.php",
                "line": 172,
                "function": "dispatch",
                "class": "OC\\AppFramework\\Http\\Dispatcher",
                "type": "->",
                "args": [
                    {
                        "__class__": "OCA\\Notifications\\Controller\\EndpointController"
                    },
                    "listNotifications"
                ]
            },
            {
                "file": "/usr/local/www/nextcloud/lib/private/Route/Router.php",
                "line": 298,
                "function": "main",
                "class": "OC\\AppFramework\\App",
                "type": "::",
                "args": [
                    "OCA\\Notifications\\Controller\\EndpointController",
                    "listNotifications",
                    {
                        "__class__": "OC\\AppFramework\\DependencyInjection\\DIContainer"
                    },
                    [
                        "v2",
                        "ocs.notifications.Endpoint.listNotifications"
                    ]
                ]
            },
            {
                "file": "/usr/local/www/nextcloud/ocs/v1.php",
                "line": 63,
                "function": "match",
                "class": "OC\\Route\\Router",
                "type": "->",
                "args": [
                    "/ocsapp/apps/notifications/api/v2/notifications"
                ]
            },
            {
                "file": "/usr/local/www/nextcloud/ocs/v2.php",
                "line": 23,
                "args": [
                    "/usr/local/www/nextcloud/ocs/v1.php"
                ],
                "function": "require_once"
            }
        ],
        "File": "/usr/local/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
        "Line": 165,
        "Previous": {
            "Exception": "ValueError",
            "Message": "The arguments array must contain 2 items, 1 given",
            "Code": 0,
            "Trace": [
                {
                    "file": "/usr/local/www/nextcloud/lib/private/L10N/L10NString.php",
                    "line": 88,
                    "function": "vsprintf",
                    "args": [
                        "%s file was imported from Google Drive.|%s files were imported from Google Drive.",
                        [
                            161
                        ]
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/lib/private/L10N/L10N.php",
                    "line": 127,
                    "function": "__toString",
                    "class": "OC\\L10N\\L10NString",
                    "type": "->",
                    "args": []
                },
                {
                    "file": "/usr/local/www/nextcloud/lib/private/L10N/LazyL10N.php",
                    "line": 56,
                    "function": "n",
                    "class": "OC\\L10N\\L10N",
                    "type": "->",
                    "args": [
                        "%s file was imported from Google Drive.",
                        "%s files were imported from Google Drive.",
                        161,
                        [
                            161
                        ]
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/apps/integration_google/lib/Notification/Notifier.php",
                    "line": 103,
                    "function": "n",
                    "class": "OC\\L10N\\LazyL10N",
                    "type": "->",
                    "args": [
                        "%s file was imported from Google Drive.",
                        "%s files were imported from Google Drive.",
                        161,
                        [
                            161
                        ]
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/lib/private/Notification/Manager.php",
                    "line": 368,
                    "function": "prepare",
                    "class": "OCA\\Google\\Notification\\Notifier",
                    "type": "->",
                    "args": [
                        {
                            "__class__": "OC\\Notification\\Notification"
                        },
                        "en_GB"
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/apps-pkg/notifications/lib/Controller/EndpointController.php",
                    "line": 111,
                    "function": "prepare",
                    "class": "OC\\Notification\\Manager",
                    "type": "->",
                    "args": [
                        {
                            "__class__": "OC\\Notification\\Notification"
                        },
                        "en_GB"
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
                    "line": 225,
                    "function": "listNotifications",
                    "class": "OCA\\Notifications\\Controller\\EndpointController",
                    "type": "->",
                    "args": [
                        "v2"
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php",
                    "line": 133,
                    "function": "executeController",
                    "class": "OC\\AppFramework\\Http\\Dispatcher",
                    "type": "->",
                    "args": [
                        {
                            "__class__": "OCA\\Notifications\\Controller\\EndpointController"
                        },
                        "listNotifications"
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/lib/private/AppFramework/App.php",
                    "line": 172,
                    "function": "dispatch",
                    "class": "OC\\AppFramework\\Http\\Dispatcher",
                    "type": "->",
                    "args": [
                        {
                            "__class__": "OCA\\Notifications\\Controller\\EndpointController"
                        },
                        "listNotifications"
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/lib/private/Route/Router.php",
                    "line": 298,
                    "function": "main",
                    "class": "OC\\AppFramework\\App",
                    "type": "::",
                    "args": [
                        "OCA\\Notifications\\Controller\\EndpointController",
                        "listNotifications",
                        {
                            "__class__": "OC\\AppFramework\\DependencyInjection\\DIContainer"
                        },
                        [
                            "v2",
                            "ocs.notifications.Endpoint.listNotifications"
                        ]
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/ocs/v1.php",
                    "line": 63,
                    "function": "match",
                    "class": "OC\\Route\\Router",
                    "type": "->",
                    "args": [
                        "/ocsapp/apps/notifications/api/v2/notifications"
                    ]
                },
                {
                    "file": "/usr/local/www/nextcloud/ocs/v2.php",
                    "line": 23,
                    "args": [
                        "/usr/local/www/nextcloud/ocs/v1.php"
                    ],
                    "function": "require_once"
                }
            ],
            "File": "/usr/local/www/nextcloud/lib/private/L10N/L10NString.php",
            "Line": 88
        },
        "CustomMessage": "--"
    }
}

I will next try the import as you suggest.

zgypa commented 1 year ago

@julien-nc : here we go. See at bottom for the entries from logs.

I do have a lot of entries without names, because i have inherited the address book from my dad, which used to use his Address Book as personal DB with all sorts of info in it (location of objects in the house, notes on specific subjects, ...).

I don't mind removing these from my address book. But i wouldn't mind having them somewhere. An idea could be to simply Giving them a "No Name" group, with empty strings as first name and last name, instead of Skipping entries without names.

Bottom line: I think that a more correct behavior for the importer would be to import with empty name, rather than skipping altogether. The importer tool should import, not clean up. Therefor a contact in google API without name array, should get a name array with empty strings as names.

{
    "reqId": "77HmY5PVTH0rL5bPn9Jl",
    "level": 0,
    "time": "2022-12-26T22:13:50+00:00",
    "remoteAddr": "10.1.49.83",
    "user": "afm",
    "app": "integration_google",
    "method": "GET",
    "url": "/apps/integration_google/import-contacts?key=0&newAddressBookName=1.0.9-3-nightly",
    "message": "6162 contacts seen",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0",
    "version": "25.0.2.3",
    "data": {: "integration_google"
        "app": "integration_google"
    }
}
{   "reqId": "77HmY5PVTH0rL5bPn9Jl",
    "level": 0,
    "time": "2022-12-26T22:13:50+00:00",
    "remoteAddr": "10.1.49.83",
    "user": "afm",
    "app": "integration_google",
    "method": "GET",
    "url": "/apps/integration_google/import-contacts?key=0&newAddressBookName=1.0.9-3-nightly",
    "message": "5390 contacts imported",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0",
    "version": "25.0.2.3",
    "data": {: "integration_google"
        "app": "integration_google"
    }
}

And the skipped once, which count up to exactly 772 = 6162-5390:

{
    "reqId": "77HmY5PVTH0rL5bPn9Jl",
    "level": 0,
    "time": "2022-12-26T21:40:05+00:00",
    "remoteAddr": "10.1.49.83",
    "user": "afm",
    "app": "integration_google",
    "method": "GET",
    "url": "/apps/integration_google/import-contacts?key=0&newAddressBookName=1.0.9-3-nightly",
    "message": "Skipping contact with no names array",
    "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0",
    "version": "25.0.2.3",
    "data": {
        "app": "integration_google"
    }
}

julien-nc commented 1 year ago

Could you try v1.0.9-5-nightly?

No more skipping contacts with empty names. I went a bit further there and clarified/improved the import logic:

created card's URI is now the google contact resource name
the URI is used to check if a remote contact exists locally
- it exists
  - the google one was edited more recently: we update it
  - the local one was edited more recently, we skip this remote contact
- it does not exist: we create it
the UI always shows this after the import:
- total number of contacts seen
- number of created ones
- number of updated ones

zgypa commented 1 year ago

Wow, cool!

So thus i shall try to import over the previous import rather than a new address book?

julien-nc commented 1 year ago

As the URI of locally created elements has changed, better start from scratch, create a new address book. Then you can make some changes on the google side and import again in the same address book.

zgypa commented 1 year ago

Looks a lot better! Some things i'm noticing:

I have added a comment in #96 regarding the URLs of websites coming in with extra backslashes (https\\:// instead of https://).
i have noticed extra backslashes in Company name which contain multiple commas. You should be able to reproduce with a company name such as Name, name, name, name? which gets imported as Name, name\, name\, name?. So the first comma is correct, the subsequent commas are escaped. I see this in the Nextcloud contacts broswer, but in macOS Contacts.app the company name appears correct. See if you can reproduce. If you cannot, i shall dig deeper in logs for you.
i have noticed that my personal contact is not importing the photo and the company name. I have not looked at the logs yet, but i did notice that i had 2 company names associated with my contact. 2 separate entries. See if you can reproduce. If you cannot, i'll have to dig deeper in the logs for you.

julien-nc commented 1 year ago

extra backslashes in Company name which contain multiple commas

This is a display bug in the Contacts app. The comas are (and should be) escaped in vCard. The value obtained from google is not coma-escaped, then it is stored escaped in the internal vCard, then it is incorrectly displayed by the Contacts app. I tried to manually enter Name, name, name, name? as company name for a freshly created contact, then reloaded the page and see Name, name\, name\, name?

i have noticed that my personal contact is not importing the photo and the company name

What do you mean by "my personal contact"?

zgypa commented 1 year ago

I suspected it could have been an NC contacts app bug. Thanks for confirming. Do you know the proper place where to report this bug?

My personal contact is just the contact which contains my information. Nothing special.

I was wondering if contacts with multiple business/company entries get imported correctly, or if maybe having multiple company entries triggers the importer to somehow fail importing photo.

julien-nc commented 1 year ago

To report the bug in contacts: https://github.com/nextcloud/contacts/issues

I was wondering if contacts with multiple business/company entries get imported correctly, or if maybe having multiple company entries triggers the importer to somehow fail importing photo.

If there are multiple companies in the json payload provided by the google API, only the first one will be considered/imported. See https://github.com/nextcloud/integration_google/blob/master/lib/Service/GoogleContactsAPIService.php#L421-L437

Anyway, the google contacts frontend does not allow adding multiple companies. Same for the NC Contacts app.

Importing contact photos is supposed to work. It works fine on my side. Is it failing for all your contacts? If it's just failing for some contacts, I would be interested to get a few of them in a vcf file and try to reproduce the issue.

zgypa commented 1 year ago

I was using fullcontact.com which was syncing iCloud and Goole contacts and probably responsible for creating the multiple companies. I only had a handful of them, and i am fine just letting go and accepting the import with only the first business.

I'm not sure, can't remember what happened, but i can now see my image. Maybe i have added it manually, can't recall.

Regardless, the original problem of this issue (contact groups information being lost) is now solved, and so i'm closing this ticket.

nextcloud / integration_google

Contact Group information is lost and not Imported. #124

Behavior

Expected Behavior