Import of contacts hangs

stbland commented 8 years ago

Import of contacts hangs after displaying progress bar.

Application version : 0.1.5

Browser console show error :

GET XHR  https://internal_name/apps/import-from-google/socket.io/?transport=polling

->

HTTP/1.1 500 Internal Server Error

Cozy application log ( https://internal_name/logs/import-from-google ) :

express deprecated res.send(status): Use res.sendStatus(status) instead at build/server/controllers/leavegoogle.js:37:7
info - Google contact 0 already synced and uptodate
info - Create 1 contact
info - Create 3 contact
info - Create 5 contact
_http_outgoing.js:342
    throw new Error('`value` required in setHeader("' + name + '", value).');
    ^

Error: `value` required in setHeader("Content-Length", value).
    at ClientRequest.OutgoingMessage.setHeader (_http_outgoing.js:342:11)
    at FormData.<anonymous> (/usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/form-data/lib/form_data.js:344:13)
    at /usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/form-data/lib/form_data.js:288:7
    at /usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/async/lib/async.js:726:13
    at /usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/async/lib/async.js:52:16
    at done (/usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/async/lib/async.js:241:17)
    at /usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/async/lib/async.js:44:16
    at /usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/async/lib/async.js:723:17
    at /usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/async/lib/async.js:167:37
    at /usr/local/cozy/apps/import-from-google/node_modules/cozydb/node_modules/form-data/lib/form_data.js:133:7

frankrousseau commented 8 years ago

Hello @stbland,

Thank you for reporting. Which version of Node.js do you use?

stbland commented 8 years ago

@frankrousseau

The version of Node.js is 4.3.0.

nono commented 8 years ago

@frankrousseau @aenario I think we should release a new version on npm. It looks like we have some commits that should help here (a5ab68201fb912ca930947d431ba655600914eb3 and c152dcec9354790ce47ead855be3c8562dd19400 mostly).

clochix commented 8 years ago

Any news?

frankrousseau commented 8 years ago

Import from google is not distributed on NPM. @nono what did you mean in your last message about the publication?

nono commented 8 years ago

There was no development branch on github. I thought it was distributed via npm. I created the branch since. And @aenario made a commit to build the last version, it should fix this issue.

alexxtasi commented 8 years ago

Hi all I think I have some similar problem (installed cozy on raspeberry pi 3 from the cozy image) The import from google app hangs after a while.

On my first attempt to import, the app just freeze and the result was to import only 11 contacts from google (I have many more).
then I tried again (checking also to import photos), the same 11 contacts was there, two pictures and the app hanged. The log file was:

cozy.local/logs/import-from-google

[2016-09-21 10:09:29:352] info - americano | add plugin: cozydb
[2016-09-21 10:09:29:781] info - Cozy DB | contact - all request creation...
[2016-09-21 10:09:29:961] info - Cozy DB | succeeded
[2016-09-21 10:09:29:962] info - Cozy DB | contact - byName request creation...
[2016-09-21 10:09:30:51] info - Cozy DB | succeeded
[2016-09-21 10:09:30:52] info - Cozy DB | event - all request creation...
[2016-09-21 10:09:30:137] info - Cozy DB | succeeded
[2016-09-21 10:09:30:138] info - Cozy DB | event - byDate request creation...
[2016-09-21 10:09:30:217] info - Cozy DB | succeeded
[2016-09-21 10:09:30:218] info - Cozy DB | event - byCalendar request creation...
[2016-09-21 10:09:30:319] info - Cozy DB | succeeded
[2016-09-21 10:09:30:320] info - Cozy DB | account - all request creation...
[2016-09-21 10:09:30:393] info - Cozy DB | succeeded
[2016-09-21 10:09:30:394] info - Cozy DB | account - byEmailWithOauth request creation...
[2016-09-21 10:09:30:485] info - Cozy DB | succeeded
[2016-09-21 10:09:30:485] info - Cozy DB | photo - byTitle request creation...
[2016-09-21 10:09:30:558] info - Cozy DB | succeeded
[2016-09-21 10:09:30:559] info - Cozy DB | album - byTitle request creation...
[2016-09-21 10:09:30:645] info - Cozy DB | succeeded
[2016-09-21 10:09:30:646] info - Cozy DB | tag - byName request creation...
[2016-09-21 10:09:30:729] info - Cozy DB | succeeded
[2016-09-21 10:09:30:730] info - Cozy DB | cozyinstance - all request creation...
[2016-09-21 10:09:30:805] info - Cozy DB | succeeded
[2016-09-21 10:09:30:805] info - Cozy DB | user - all request creation...
[2016-09-21 10:09:30:823] info - Cozy DB | succeeded
[2016-09-21 10:09:30:825] info - Cozy DB | account - define indexes...
[2016-09-21 10:09:30:830] info - Cozy DB | album - define indexes...
[2016-09-21 10:09:30:831] info - Cozy DB | contact - define indexes...
[2016-09-21 10:09:30:832] info - Cozy DB | event - define indexes...
[2016-09-21 10:09:30:832] info - Cozy DB | cozyinstance - define indexes...
[2016-09-21 10:09:30:833] info - Cozy DB | photo - define indexes...
[2016-09-21 10:09:30:834] info - Cozy DB | tag - define indexes...
[2016-09-21 10:09:30:835] info - americano | cozydb loaded.
[2016-09-21 10:09:34:654] info - americano | Routes loaded.
[2016-09-21 10:09:35:448] info - americano | Configuration for production loaded.
import-from-google server is listening on port 9289...
[2016-09-21 10:09:35:545] info - config | LocalizationManager initialized
[Wed, 21 Sep 2016 10:37:17 GMT] - GET / - 200 - 109.967 ms -  872
[Wed, 21 Sep 2016 14:11:56 GMT] - GET / - 200 - 29.968 ms -  872

but I see no error there !!

one last attempt and while the import dialogue shows import 4 from 9 photos (some like that), the app hangs and also cozy

When unplugged the pi and boot it again the log was the same!! :

[2016-09-21 18:11:54:791] info - americano | add plugin: cozydb
[2016-09-21 18:11:55:221] info - Cozy DB | contact - all request creation...
[2016-09-21 18:11:55:393] info - Cozy DB | succeeded
[2016-09-21 18:11:55:395] info - Cozy DB | contact - byName request creation...
[2016-09-21 18:11:55:485] info - Cozy DB | succeeded
[2016-09-21 18:11:55:486] info - Cozy DB | event - all request creation...
[2016-09-21 18:11:55:573] info - Cozy DB | succeeded
[2016-09-21 18:11:55:575] info - Cozy DB | event - byDate request creation...
[2016-09-21 18:11:55:657] info - Cozy DB | succeeded
[2016-09-21 18:11:55:658] info - Cozy DB | event - byCalendar request creation...
[2016-09-21 18:11:55:743] info - Cozy DB | succeeded
[2016-09-21 18:11:55:744] info - Cozy DB | account - all request creation...
[2016-09-21 18:11:55:813] info - Cozy DB | succeeded
[2016-09-21 18:11:55:815] info - Cozy DB | account - byEmailWithOauth request creation...
[2016-09-21 18:11:55:881] info - Cozy DB | succeeded
[2016-09-21 18:11:55:883] info - Cozy DB | photo - byTitle request creation...
[2016-09-21 18:11:55:948] info - Cozy DB | succeeded
[2016-09-21 18:11:55:950] info - Cozy DB | album - byTitle request creation...
[2016-09-21 18:11:56:23] info - Cozy DB | succeeded
[2016-09-21 18:11:56:25] info - Cozy DB | tag - byName request creation...
[2016-09-21 18:11:56:93] info - Cozy DB | succeeded
[2016-09-21 18:11:56:95] info - Cozy DB | cozyinstance - all request creation...
[2016-09-21 18:11:56:160] info - Cozy DB | succeeded
[2016-09-21 18:11:56:161] info - Cozy DB | user - all request creation...
[2016-09-21 18:11:56:176] info - Cozy DB | succeeded
[2016-09-21 18:11:56:179] info - Cozy DB | account - define indexes...
[2016-09-21 18:11:56:183] info - Cozy DB | album - define indexes...
[2016-09-21 18:11:56:185] info - Cozy DB | contact - define indexes...
[2016-09-21 18:11:56:186] info - Cozy DB | event - define indexes...
[2016-09-21 18:11:56:187] info - Cozy DB | cozyinstance - define indexes...
[2016-09-21 18:11:56:188] info - Cozy DB | photo - define indexes...
[2016-09-21 18:11:56:189] info - Cozy DB | tag - define indexes...
[2016-09-21 18:11:56:191] info - americano | cozydb loaded.
[2016-09-21 18:12:01:644] info - americano | Routes loaded.
[2016-09-21 18:12:02:561] info - americano | Configuration for production loaded.
import-from-google server is listening on port 9289...
[2016-09-21 18:12:02:654] info - config | LocalizationManager initialized
[Wed, 21 Sep 2016 18:12:39 GMT] - GET / - 200 - 71.208 ms -  872
[Wed, 21 Sep 2016 18:13:41 GMT] - GET / - 200 - 72.611 ms -  872

Is it just that it's a big load of contacts and photos and the little pi freezes ? Or it's related with the bug you are discussing ? Forgive me for my newbie questions.... and thanks

clochix commented 8 years ago

Hello @alexxtasi, thanks for your feedback. Could you have a look a the file /usr/local/var/log/cozy/import-from-google-err.log on your server filesystem. Errors may be logged there.

alexxtasi commented 8 years ago

Hi, the /usr/local/var/log/cozy/import-from-google-err.log file is empty (there is a /usr/local/var/log/cozy/import-from-google-err.log-backup file with few lines) Since the .log is empty I'll try the import procedure again and come back again to report.

alexxtasi commented 8 years ago

Well... I made another import attempt. But before that, I deleted all google data that were previous imported (those three photos, 11 contacts, even the empty calendar!) and reboot the RPi. I even went to my Google account profile and deleted the Cozy instance that was connected (I thought I should do the hole thing from the begging :-) ).

Then I started the "Import from Google" app, connect it with my google profile once again, but did not select to import photos this time (to decrease data maybe ? just a hard guess).

Now the dialogue stack showing Calendar import complete! 1 imported events on 1 and Contact import running... 7 imported contacts on 581 and also the ssh connection I had open, is broken.

...I let the app run for some hour, but it is still stack there, so refresh the page, which brings me to the cozy login page. When login, the Import From Google dialogue comes up once again from the beginning!

logs:

/usr/local/var/log/cozy/import-from-google-err.log is empty
/usr/local/var/log/cozy/import-from-google-err.log-backup - empty
/usr/local/var/log/cozy/import-from-google.log :

[2016-09-23 13:23:01:414] info - americano | add plugin: cozydb
[2016-09-23 13:23:01:838] info - Cozy DB | contact - all request creation...
[2016-09-23 13:23:02:16] info - Cozy DB | succeeded
[2016-09-23 13:23:02:18] info - Cozy DB | contact - byName request creation...
[2016-09-23 13:23:02:122] info - Cozy DB | succeeded
[2016-09-23 13:23:02:123] info - Cozy DB | event - all request creation...
[2016-09-23 13:23:02:220] info - Cozy DB | succeeded
[2016-09-23 13:23:02:222] info - Cozy DB | event - byDate request creation...
[2016-09-23 13:23:02:318] info - Cozy DB | succeeded
[2016-09-23 13:23:02:319] info - Cozy DB | event - byCalendar request creation...
[2016-09-23 13:23:02:400] info - Cozy DB | succeeded
[2016-09-23 13:23:02:400] info - Cozy DB | account - all request creation...
[2016-09-23 13:23:02:481] info - Cozy DB | succeeded
[2016-09-23 13:23:02:482] info - Cozy DB | account - byEmailWithOauth request creation...
[2016-09-23 13:23:02:563] info - Cozy DB | succeeded
[2016-09-23 13:23:02:564] info - Cozy DB | photo - byTitle request creation...
[2016-09-23 13:23:02:682] info - Cozy DB | succeeded
[2016-09-23 13:23:02:684] info - Cozy DB | album - byTitle request creation...
[2016-09-23 13:23:02:778] info - Cozy DB | succeeded
[2016-09-23 13:23:02:780] info - Cozy DB | tag - byName request creation...
[2016-09-23 13:23:02:852] info - Cozy DB | succeeded
[2016-09-23 13:23:02:855] info - Cozy DB | cozyinstance - all request creation...
[2016-09-23 13:23:02:930] info - Cozy DB | succeeded
[2016-09-23 13:23:02:931] info - Cozy DB | user - all request creation...
[2016-09-23 13:23:02:945] info - Cozy DB | succeeded
[2016-09-23 13:23:02:948] info - Cozy DB | account - define indexes...
[2016-09-23 13:23:02:952] info - Cozy DB | album - define indexes...
[2016-09-23 13:23:02:953] info - Cozy DB | contact - define indexes...
[2016-09-23 13:23:02:954] info - Cozy DB | event - define indexes...
[2016-09-23 13:23:02:955] info - Cozy DB | cozyinstance - define indexes...
[2016-09-23 13:23:02:956] info - Cozy DB | photo - define indexes...
[2016-09-23 13:23:02:957] info - Cozy DB | tag - define indexes...
[2016-09-23 13:23:02:959] info - americano | cozydb loaded.
[2016-09-23 13:23:06:826] info - americano | Routes loaded.
[2016-09-23 13:23:07:648] info - americano | Configuration for production loaded.
import-from-google server is listening on port 9289...
[2016-09-23 13:23:07:736] info - config | LocalizationManager initialized
[Fri, 23 Sep 2016 15:13:49 GMT] - GET / - 200 - 87.530 ms -  872

the timestamp of the import-from-google-err.log shows that it is created before I run the app... so the way I see it, it seems that the import app is listening on port 9289 at 13:23:07:648 and hangs at [Fri, 23 Sep 2016 15:13:49 GMT] - GET / - 200 - 87.530 ms - 872 showing no errors !

Between all the cozy's logs (I suppose this is the directory to store logs), the latest created (based on time) are:

-rw-r--r-- 1 root staff 76556 Sep 23 15:18 data-system.log
-rw-r--r-- 1 root staff  7262 Sep 23 15:13 home.log
-rw-r--r-- 1 root staff  4464 Sep 23 15:13 proxy.log
-rw-r--r-- 1 root staff  2535 Sep 23 15:13 import-from-google.log
-rw-r--r-- 1 root staff 12280 Sep 23 15:13 data-system-err.log

let me know if some of them is useful and I'll copy it here

it's confusing to me is all this related to a some bug, or I am wrong-configuring something here?

regards

clochix commented 8 years ago

Well, that's really weird, I don't see any error. I wonder if the whole Cozy stack has not restarted, dropping the logs. Could you run (as root) cozy_management show_reporting and paste the output to some pastebin? (this command display a complete diagnosis of your server).

alexxtasi commented 8 years ago

yes it is weird! And more weird is the fact that I did little configuration (only installed some cozy apps just to test if the suit me) - I didn't mesh with it. Another thing is that sometimes I have to provide cozy's ip to connect (nginx does some tricks) but I think it's irrelevant.

Anyway, you'll find the output of sudo cozy_management show_reporting here (I think it's for 30 days ?). Hope it's useful.

clochix commented 8 years ago

:-/ In don't see any error.

Do you know how to open the developer console in your browser of choice? If son could your open the console, try another import, and see if you get any JS error message inside the browser console?

alexxtasi commented 8 years ago

well correct me if wrong I use Firefox, so I pressed F12, then clicked on "Web Console" and there on "JS" I checked "Errors" and "Warnings".

I started the google-import procedure again, and checked to import only calendars and contacts (no pictures) and .... "this is the moment I felt embarrassed" it worked!! I really cannot understand it! I pressed the Take me back home button (or something like that), the cozy's home screen was there and then those errors came up:

- The connection to wss://cozy.local/apps/import-from-google/socket.io/?EIO=3&transport=websocket&sid=rgv2C7czdnlmhuh0AAAC was interrupted while the page was loading. vendor.js:632:5172
- The connection to wss://cozy.local/socket.io/?EIO=3&transport=websocket&sid=JYFRrc3Bi5xf8FaJAAAM was interrupted while the page was loading. vendor.js:32583:15
- unreachable code after return statement vendor.js:26952:12384
- WARNING: Missing translation for key: "downloader description" vendor.js:27291:42
- WARNING: Missing translation for key: "music description" vendor.js:27291:42
- Use of getPreventDefault() is deprecated.  Use defaultPrevented instead. vendor.js:3571:28

(the two warnings were the from the start)

Now all 581 contacts are inside cozy ! Error log is empty again and /usr/local/var/log/cozy/import-from-google.log after lots of Create contact lines ends with [2016-09-28 16:41:10:881] info - leave | import from google complete... which all seem normal !!

Any guess ? I am I the only one to have test it via the RPi image and have this problems or not ?

I will try this again trying to import google pictures also ... and see what happens

alexxtasi commented 8 years ago

I repeated the import again, now checking all 3 options (contacts, calendar and photos). The import was successful once again !! (the photos were just a few).

No errors on the browser's dev console, only the same 6 as mentioned in the previous post (after pressing the Back home button at the end of the process).
Nothing in /usr/local/var/log/cozy/import-from-google-err.log

Really confusing to me, but it worked

ArchangeGabriel commented 8 years ago

I didn’t look at all the details here, but I think I had the same error. FWIR, checking only contacts or contacts+calendar resulted in a hang, but checking all 3 options does work.

Unfortunately, I’m not able to try it again right now because of ArchLinux Couchdb 2.0 upgrade which made my Cozy unavailable for now (until someone provide me a solution, so if you have any…).

Note that I have no pictures, and almost no calendars, mostly contacts (and a lot of them, see https://github.com/cozy-labs/import-from-google/issues/41).

Scorpio2177 commented 8 years ago

I'm having SERIOUS issues with my Gmail not syncing, & my preinstalled Android contacts & calendar apps not syncing. I've done EVERY possible troubleshooting steps there are... MORE THAN ONCE... Can anyone PLEASE help me? Thank you, Scorpio2177 #41

https://drive.google.com/folderview?id=0B8Pv0NriKBOmOVRfSkdkU2ZGNkk

cozy-labs / import-from-google

Import of contacts hangs #50