mstilkerich / rcmcarddav

CardDAV plugin for RoundCube Webmailer
GNU General Public License v2.0
257 stars 81 forks source link

not all carddav contacts are displayed #221

Closed pafcioooo closed 6 years ago

pafcioooo commented 6 years ago

I have a nextcloud address book with ca. 2400 entries. I have installed roundcube carddav plugin. I have configured default address book for the plugin in plugins/carddav/config.inc.php. Now address book is displayed but only with ca. 700 entries.

I am not sure if it is connected with the problem but in logs/error log there are following entries while operating on contacts in roundcube:

[05-May-2018 22:32:19 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:32:37 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:32:37 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:37 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:41 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:32:47 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:32:54 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  array_merge(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1035
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  array_intersect(): Argument #1 is not an array in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 1039
[05-May-2018 22:33:21 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
[05-May-2018 22:33:31 Europe/Warsaw] PHP Warning:  Invalid argument supplied for foreach() in /home/poczta/public_html/roundcubemail/plugins/carddav/carddav_backend.php on line 645
blind-coder commented 6 years ago

This error happens when you remove the require_always attribute from a preconfigured addressbook, I honestly never tried doing that, because it didn't make sense to me to try. Can you please add 'require_always' => [], to your preset and give it a try if it syncs correctly? Mind the refresh_time if you do.

pafcioooo commented 6 years ago

I have checked that I had 'require_always'=> true, so I changed it to 'require_always' => [],. I have also changed refresh_time to 15 minutes with 'refresh_time' => '00:15:00',. Then I have waited 30 minutes, logout and login, and even restart apache. Nothing helped. After that I noticed that my own carddav settings in RC gui do not got changed even though I have updated plugins/carddav/config.inc.php file. So then I changed my refresh time in GUI to 15 minutes - but it did not helped either.

Then I have found that in logs/error there is following error entry (I have change real values here to XXXXX): <g84rp7j0> DB Error: [1366] Incorrect string value: '\xF0\x9F\x8C\xA9' for column 'surname' at row 1 (SQL Query: INSERT INTO carddav_contacts (firstname,surname,showas,email,name,etag,vcard,abook_id,uri,cuid) VALUES ('XXXXX','XXXXX🌩','INDIVIDUAL','XXXXXX@XXXXX','XXXXXX XXXXX🌩','\"9ce5a49d7ef767d9c9b1a97735e33ced\"','BEGIN:VCARD\r\nVERSION:3.0\r\nPRODID:-//Sabre//Sabre VObject 4.1.2//EN\r\nUID:4b778423-88c8-4bb5-868e-ca29c5efa7a0\r\nN:XXXXX🌩;XXXXX;;;\r\nFN:XXXXX XXXXX🌩\r\nREV:20171203T194303Z\r\nTEL;TYPE=cell:+XXXXXX\r\nEMAIL:XXXXX@XXXXXX\r\nPHOTO

So I logged into nextcloud and deleted 🌩 from the contact. Then all contacts got synchronized.

Thanks

PS. I have also noticed that "use modern group" is disabled and it can not be enabled.

blind-coder commented 6 years ago

Which database backend are you using? MySQL, PostgreSQL or SQLite? Can you share your preset addressbook settings (sans username/password/url)? Lastly, can you share an anonymized VCF file including the emoji character? This really should work.

pafcioooo commented 6 years ago

ad. 1 I use MySQL ad.2 Do you mean settings from config.inc.php? ad. 3 Here you go, but I am not sure if it helps since this is with already deleted emoji character - I did not try to enter it once more. You can see the emoji character in my previous comment. cb53696f-bc11-4af9-93af-0876bafe2748.zip

This is also interesting since I have another contact with emoji - only different ☁ and it was imported correctly.

I also remember that I had similar issues while importing emails to database and this is know problem with database encoding...

blind-coder commented 6 years ago

Thanks! Yes, I mean the config.inc.php addressbook preset. The database should be able to handle emoji characters. Or at least RCMCardDAV really should gracefully skip over "broken" VCF entries.

pafcioooo commented 6 years ago

Ad.2 :-)

$prefs['Personal'] = array(
        // required attributes
        'name'         =>  'Personal',
        // will be substituted for the roundcube username
        'username'     =>  '%u',
        // will be substituted for the roundcube password
        'password'     =>  '%p',
        // %u will be substituted for the CardDAV username
        'url'          =>  'https://<main_domain_here>/remote.php/dav/addressbooks/users/%u/contacts',
        'active'       =>  true,
        'readonly'     =>  false,
        'refresh_time' => '00:15:00',
        //'require_always'=> true,
        'require_always' => [],
        //'fixed'        =>  array( 'username' ),
        'hide'        =>  false,
);

You are welcome and I thank you:-)

pafcioooo commented 6 years ago

Just remind myself, this can be helpfull:

blind-coder commented 6 years ago

Oh wow, thanks for this! I'll create a dbmigration for that.

blind-coder commented 6 years ago

I've created a hotfix that addresses this issue (at least in parts). I need to do a bit more to properly fix this as right now indices are smaller than the column they index, which can lead to issues if you have really long names/uris/cuids or whatnot. This affects the database structure of all supported databases, though, so I'll need to look a bit deeper here.