search

nextcloud / integration_google

🇬 Google integration into Nextcloud
GNU Affero General Public License v3.0
107 stars 33 forks source link

Cannot Import Photos or Drive when using reverse proxy #31

Closed mike-lloyd03 closed 5 months ago

mike-lloyd03 commented 4 years ago

Problem

I followed the instructions and have authenticated my Google account with the integration app. The app can see how many documents and photos I have but when I try to import, nothing happens. The Google API dashboard shows requests being made with 200 response codes but no data is transferred.

Current Setup

Apache Configuration

<VirtualHost 127.0.0.1:8080>
    Alias "/" "/var/www/nextcloud/"

    <Directory /var/www/nextcloud/>
        Options +FollowSymlinks
        AllowOverride All

        <IfModule mod_dav.c>
            Dav off
        </IfModule>

        SetEnv HOME /var/www/nextcloud
        SetEnv HTTP_HOME /var/www/nextcloud
    </Directory>

    ErrorLog /var/log/apache2/nextcloud-error_log
    CustomLog /var/log/apache2/nextcloud-access_log common
</VirtualHost>

nginx Configuration

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name <external-site>;

    ssl_certificate /etc/letsencrypt/live/<external-site>/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/<external-site>/privkey.pem;

    client_max_body_size 0;
    underscores_in_headers on;

    location ~ {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        add_header Front-End-Https on;

        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 64;

        proxy_buffering off;
        proxy_redirect off;
        proxy_max_temp_file_size 0;
        proxy_pass http://127.0.0.1:8080;
    }
}

Nextcloud Configuration

<?php
$CONFIG = array (
    'instanceid' => <instanceid>,
    'passwordsalt' => <passwordsalt>,
    'secret' => <secret>,
    'trusted_domains' =>
    array (
        0 => <external_url>,
    ),
    'datadirectory' => '/media/data/nextcloud-data',
    'dbtype' => 'mysql',
    'version' => '20.0.1.1',
    'overwrite.cli.url' => <external_url>,
    'dbname' => 'nextcloud',
    'dbhost' => 'localhost:3306',
    'dbport' => '',
    'dbtableprefix' => 'oc_',
    'mysql.utf8mb4' => true,
    'dbuser' => 'ncuser',
    'dbpassword' => <dbpassword>,
    'installed' => true,
    'memcache.local' => '\OC\Memcache\APCu',
    'overwriteprotocol' => 'https',
);

Nextcloud Logs

[integration_google] Warning: Google API error getting album list, no "albums" key in []

GET /index.php/apps/integration_google/photo-number
from 127.0.0.1 by Mike at 2020-11-16T17:04:34+00:00

[PHP] Error: Error: count(): Parameter must be an array or an object that implements Countable at /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php#56 at <<closure>>

0. <<closure>>
   OC\Log\ErrorHandler::onError(2, "count(): Parame ... e", "/var/www/nextcl ... p", 56, {accessToken: "y ... ]})
1. /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php line 56
   count(null)
2. /var/www/nextcloud/apps/integration_google/lib/Controller/GoogleAPIController.php line 136
   OCA\Google\Service\GoogleContactsAPIService->getContactNumber("ya29.A0AfH6SMAm ... s", "Mike")
3. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 169
   OCA\Google\Controller\GoogleAPIController->getContactNumber()
4. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 100
   OC\AppFramework\Http\Dispatcher->executeController(OCA\Google\Contr ... "}, "getContactNumber")
5. /var/www/nextcloud/lib/private/AppFramework/App.php line 152
   OC\AppFramework\Http\Dispatcher->dispatch(OCA\Google\Contr ... "}, "getContactNumber")
6. /var/www/nextcloud/lib/private/Route/Router.php line 308
   OC\AppFramework\App::main("OCA\\Google\\Co ... r", "getContactNumber", OC\AppFramework\ ... {}, {action: null,_r ... "})
7. /var/www/nextcloud/lib/base.php line 1009
   OC\Route\Router->match("/apps/integration_google/contact-number")
8. /var/www/nextcloud/index.php line 37
   OC::handleRequest()

GET /index.php/apps/integration_google/contact-number
from 127.0.0.1 by Mike at 2020-11-16T17:04:32+00:00

[PHP] Error: Error: Undefined index: connections at /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php#56 at <<closure>>

0. /var/www/nextcloud/apps/integration_google/lib/Service/GoogleContactsAPIService.php line 56
   OC\Log\ErrorHandler::onError(8, "Undefined index: connections", "/var/www/nextcl ... p", 56, {accessToken: "y ... ]})
1. /var/www/nextcloud/apps/integration_google/lib/Controller/GoogleAPIController.php line 136
   OCA\Google\Service\GoogleContactsAPIService->getContactNumber("ya29.A0AfH6SMAm ... s", "Mike")
2. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 169
   OCA\Google\Controller\GoogleAPIController->getContactNumber()
3. /var/www/nextcloud/lib/private/AppFramework/Http/Dispatcher.php line 100
   OC\AppFramework\Http\Dispatcher->executeController(OCA\Google\Contr ... "}, "getContactNumber")
4. /var/www/nextcloud/lib/private/AppFramework/App.php line 152
   OC\AppFramework\Http\Dispatcher->dispatch(OCA\Google\Contr ... "}, "getContactNumber")
5. /var/www/nextcloud/lib/private/Route/Router.php line 308
   OC\AppFramework\App::main("OCA\\Google\\Co ... r", "getContactNumber", OC\AppFramework\ ... {}, {action: null,_r ... "})
6. /var/www/nextcloud/lib/base.php line 1009
   OC\Route\Router->match("/apps/integration_google/contact-number")
7. /var/www/nextcloud/index.php line 37
   OC::handleRequest()

GET /index.php/apps/integration_google/contact-number
from 127.0.0.1 by Mike at 2020-11-16T17:04:32+00:00
Bergum commented 4 years ago

I have the same pronlem

julien-nc commented 4 years ago

Thanks for the precise bug report. The error about contacts is fixed in v0.0.22. It would be nice if you could try it.

The only thing related to photos in your logs is [integration_google] Warning: Google API error getting album list, no "albums" key in []. This means Google API is returning an empty array when asking for album list.

Could you try the "live API" there https://developers.google.com/photos/library/reference/rest/v1/albums/list?hl=en_US ? Just click "execute" in the right sidebar. I'm interested to see what's the response (appearing right under the "execute" button).

mike-lloyd03 commented 4 years ago

Thanks. The response I get is a json object with the following format:

{
  "albums": [
    {
      "id": "<id>",
      "title": "<title>",
      "productUrl": "<productUrl>",
      "mediaItemsCount": "<mediaItemsCount>",
      "coverPhotoBaseUrl": "<coverPhotoBaseUrl>",
      "coverPhotoMediaItemId": "<coverPhotoMediaItemId>"
    }
  ],
  "nextPageToken": "<token>"
}
julien-nc commented 4 years ago

Well apparently the response the integration_google app is receiving is something like [] or {} :grin:. I don't know why...and there's not much I can do without further information. The only difference between the live API and the Nextcloud app is the "OAuth app" that is used. With the Nextcloud app, it's the one that you configured.

It's disturbing that the app can tell you the number of photos and then not get the album list because the same https://photoslibrary.googleapis.com/v1/albums request is done in both cases.

mike-lloyd03 commented 4 years ago

What information can I give you to help?

mike-lloyd03 commented 4 years ago

Also interesting, it knows which albums are shared because checking/unchecking the 'Ignore Shared Albums' checkbox updates the photo count.

mike-lloyd03 commented 4 years ago

Okay I just checked my Google API console. Under 'OAuth Consent Screen', there's a warning saying:

We are currently experiencing service disruptions and are unable to provide quota information at this time.

That could be it.

julien-nc commented 4 years ago

What information can I give you to help?

I don't know :grin:.

It would help if you could add this:

if (preg_match('/v1\/albums$/i', $url)) {
    file_put_contents('/tmp/test', $body);
}

between lines 113 and 114 (just before return json_decode($body, true);) of lib/Service/GoogleAPIService.php.

Then launch the photo import and when the [integration_google] Warning: Google API error getting album list, no "albums" key in [] warning appears in the logs, get the content of /tmp/test file. It is the raw response of the API.

That could be it.

I doubt it as you can see get the photo number...you can get this album list when loading the migration settings page but not when importing. Weird stuff, really :grin:.

julien-nc commented 4 years ago

If you prefer, I can produce a nightly build of the app and you would only have to install it manually. Would you prefer that?

mike-lloyd03 commented 4 years ago

Let me give this a try real quick.

mike-lloyd03 commented 4 years ago

Now I'm getting this error banner five times when I load the page:

This application requires JavaScript for correct operation. Please enable JavaScript and reload the page.

julien-nc commented 4 years ago

Yeah there is a syntax error in the snippet. I fixed it. Sorry.

mike-lloyd03 commented 4 years ago

I should have caught that.

Okay the empty array error is still popping up in the log but nothing is being written to /tmp/test

mike-lloyd03 commented 4 years ago

For what it's worth, I get the same problem when trying to import files from Drive including the empty Album array error.

julien-nc commented 4 years ago

Damn, long day, replace $url by $endPoint in the snippet. It should now write something in /tmp/test.

julien-nc commented 4 years ago

Thanks for your patience.

BenjaminAlpert commented 4 years ago

Same problem here

Bergum commented 4 years ago

Photo import at noe working for my part. Files does still not work.

mike-lloyd03 commented 4 years ago

Okay still nothing is being written to /tmp/test. This is what I have in 'GoogleAPIService.php`, lines 111-118:

if ($respCode >= 400) {
    return ['error' => $this->l10n->t('Bad credentials')];
} else {
    if (preg_match('/v1\/albums$/i', $endPoint)) {
        file_put_contents('/tmp/test', $body);
    }
    return json_decode($body, true);
}

Am I missing something?

julien-nc commented 4 years ago

@mike-lloyd03 Is the file created? If it's created but stays empty it means the response you get from Google API is an empty string.

Let's try something else: Could you authorize this URL : https://localhost/dev/server/index.php/apps/integration_google/oauth-redirect in developer console -> OAuth 2.0 Client IDs -> your web app -> Authorized redirect URIs ?

I would need the app's Client ID and secret (you can send them by email).

It will let me try to authenticate to my google account through your OAuth app. I will hopefully be able to reproduce the bug on my side and then...we'll see.

mike-lloyd03 commented 3 years ago

@eneiluj

Sorry it has taken me so long to get back to you. No file is being created at /tmp/test.

I just sent you an email with the details you needed and I added the URL to my Google web app.

Thanks for your help with this.

julien-nc commented 3 years ago

@mike-lloyd03 Don't be sorry, you're helping a lot!

It's working for me with your OAuth app so let's find out what happens exactly. I wonder if it can be because you're using a reverse proxy like @Bergum. @BenjaminAlpert Are you using a reverse proxy too?

One thing is for sure, things are working fine when your Nextcloud does the requests as an authenticated user because you get the photo count in the settings page. The background job used to import photos/drive is launched in a different context. This might be the source of the problem.

I suspect the authentication is not correctly provided to the requests in the background job. Let's find out.

So I made a nightly release with a lot of level 1 logs. Could you install it (just replace the .../nextcloud/app/integration_google directory) and change .../nextcloud/config/config.php to set log level to 1:

'loglevel' => 1,

There will be much more information in your logs. I would need to see what happens there when you import your google photos. We're gonna nail this bug...hopefully.

BenjaminAlpert commented 3 years ago

Yes, I am using Nginx Reverse Proxy. I am also using cron for background jobs.

Bergum commented 3 years ago

You have a new log file.

julien-nc commented 3 years ago

@BenjaminAlpert ok good to know, thanks.

@Bergum Thanks. I guess you were using latest release and not the nightly build. Anyway, fixed 2 more things thanks to your logs:

v0.0.24 is out.

Something interesting in @Bergum log file:

{"Exception":"Error","Message":"fread(): read of 8192 bytes failed with errno=9 Bad file descriptor at /var/www/nextcloud/3rdparty/icewind/streams/src/Wrapper.php#91...

So either:

If someone has time to setup a test instance without reverse proxy on the same machine and try importing photos and drive, it would be interesting to know if the issue we're facing is related to reverse proxies.

Thanks again to all of you for you perseverance.

Bergum commented 3 years ago

I was using the latest nightly. Or, I think i was. I dowloaded i and extrackted

but, im a noob on ubuntu.

Edit: Nope. I failed. Extrackted to the wrong place. New log is coming... .

Bergum commented 3 years ago

New logfile uploaded.

julien-nc commented 3 years ago

@Bergum There is nothing on 2020-11-18 related to drive or photo import in your logfile. What did you do before getting it? I need a full run of those import to see something in the log file. Thanks anyway :grin:

mike-lloyd03 commented 3 years ago

I should be able to just forward traffic to my Apache server directly and forego nginx. I'll try tonight.

Bergum commented 3 years ago

I installed the nightly, set log level, stopped all of the imports that was running, started a new import of all possible imports, and watched until new lines stopped coming into the log app. Then i sendt you the log.

mike-lloyd03 commented 3 years ago

Looks like the problem is definitely with my nginx setup. I bypassed it and the transfer from google drive seemed to be working as expected.

julien-nc commented 3 years ago

@Bergum I think the upload failed. I don't see the file.

@mike-lloyd03 Some progress! I still can't see why the reverse proxy is problematic in the background job context and not when loading the page and getting photo/files number...

maretodoric commented 3 years ago

Hey everyone. I have a similar issue though since not having the same setup (no proxy and NC is located on a VPS) and no errors in logs as it seems. I'm not sure do i open a new issue or continue here, so waiting for feedback.

Bergum commented 3 years ago

Sorry for the long delay. Have you had any success with Proxy yet? Have uploaded the latest log file.

josephkerkhof commented 3 years ago

I am also facing this issue of not being able to import files from Google Drive. For me the integration doesn't produce any log messages (errors or warnings), it just stays at 0% files imported.

Like @mike-lloyd03, I am also using Nginx. My setup is nearly identical to the Nginx configuration given in Nextcloud's documentation.

gfabiano commented 3 years ago

I am also facing this issue of not being able to import files from Google Drive. For me the integration doesn't produce any log messages (errors or warnings), it just stays at 0% files imported.

Like @mike-lloyd03, I am also using Nginx. My setup is nearly identical to the Nginx configuration given in Nextcloud's documentation.

Same problem here, at some point it made some folders but no files imported.

Bergum commented 3 years ago

Same problem here, at some point it made some folders but no files imported.

That is the same with me, it created the folders, but does not dowload the files. Im behind a Proxy

deanbarrow commented 3 years ago

I had this issue too and used Rclone in the end, create google drive and webdav links then copy from one to the other. If you want to migrate photos then you need to repeat the process specifically for the photos remote.

2fst4u commented 3 years ago

I'm having issues with this too and I think I may be running into the same symptoms. I have oauth set up and everything appears fine. In the user settings it gives the option to import calendars, Google Drive and Google Photos but when I try to import it just stays on 0 imported. I thought it might be because my "web app" isn't verified so I've attempted that process but it seems far too difficult for something so simple.

I'm also behind a reverse proxy but mine is haproxy.

OctopusET commented 3 years ago

Same problem here. At the first time about 100 photos had been imported from google photo, But that was it. never works again.

SanchosPancho commented 3 years ago

Same problem here as @OctopusET. But only 60 of 700+ photos imported NC 21.0.3

mjeshurun commented 3 years ago

I managed to migrate Contacts, Calendars and Photos, but the Drive files are stuck on 0%. I'm using Nextcloud 20.0.12 installed with NextCloudPi. Running on Raspberry Pi 4 4GB with Raspberry Pi OS.

FredericESH commented 3 years ago

I have same issue, unable to import Google drive content. Only folder structure is copy, but no files.

Any way to solve this issue ?

JensUweB commented 2 years ago

I have the same issue. Contacts & calendars are imported fine, but google drive doesn't work. I can see some api calls inside cloud console too, but no files or folders are created.

I'm not sure if my nginx config is correct, therefore I'm posting it here so that you can have a look at it. I copied my nginx config from various places so maybe some configs are not nessesary. 😅

server {
       server_name cloud.domain.tld;

        add_header X-Served-By "Served by Me!";
        add_header Strict-Transport-Security "max-age=15768000; includeSubDomains; preload;" always;

        client_max_body_size 5G;

         # Enable gzip but do not remove ETag headers
        gzip on;
        gzip_vary on;
        gzip_comp_level 4;
        gzip_min_length 256;
        gzip_proxied expired no-cache no-store private no_last_modified no_etag auth;
        gzip_types application/atom+xml application/javascript application/json application/ld+json application/manifest+json application/rss+xml application/vnd.geo+json application/vnd.ms-fontobject application/x-font-ttf application/x-web-app-manifest+json application/xhtml+xml application/xml font/opentype image/bmp image/svg+xml image/x-icon text/cache-manifest text/css text/plain text/vcard text/vnd.rim.location.xloc text/vtt text/x-component text/x-cross-domain-policy;

        # Remove X-Powered-By, which is an information leak
        fastcgi_hide_header X-Powered-By;

        rewrite ^/\.well-known/carddav https://$server_name/remote.php/dav/ redirect;
        rewrite ^/\.well-known/caldav https://$server_name/remote.php/dav/ redirect;

       location /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
        }

       location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        # proxy_set_header X-Forwarded-Proto https;
        proxy_set_header Connection 'upgrade';
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Content-Security-Policy upgrade-insecure-requests;

        proxy_cache_bypass $http_upgrade;

        proxy_pass http://172.1.1.2;
       }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/cloud.domain.tld/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/cloud.domain.tld/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    server_name cloud.domain.tld;
        listen 80;
        return 301 https://$host$request_uri;
}
spcano01 commented 2 years ago

FWIW, I also get folders but no files with Google Drive, although Contacts & Calendar works perfectly. Unlike others, I'm using pfSense's HAProxy vs nginx proxy manager for my Reverse Proxy.

Like others, it shows the right number of files/photos to be synced as well as space required.

marbman21 commented 2 years ago

Is this ever gonna be fixed? I am facing the same problem running behind nginx proxy. I can't believe this has been an open issue for over a year and still no resolution.

spcano01 commented 2 years ago

Not sure if this alone fixed Photos import (Drive still is stuck at 0% & importing only folders but no files) - but I created a Service Account attached to my project. And now my Photos are slowly importing!

TannerNelson16 commented 1 year ago

Not sure if this alone fixed Photos import (Drive still is stuck at 0% & importing only folders but no files) - but I created a Service Account attached to my project. And now my Photos are slowly importing!

What type of service account did you attach? I've still had no luck getting this to work with next cloud in this NGINX configuration.

github-actions[bot] commented 1 year ago

Hello :wave: This issue appears to have had no activity for 3 months. We cannot keep track of whether individual issues have resolved themselves or still require attention without user interaction. We're thus adding the stale label to this issue to schedule it for getting closed in 5 days time. If you believe this issue is still valid and should be fixed, you can add a comment or remove the label to avoid it getting closed.

Cheers :blue_heart:

TannerNelson16 commented 1 year ago

Still very interested in a fix for this!