gilesknap / gphotos-sync

Google Photos and Albums backup with Google Photos Library API
Apache License 2.0
1.97k stars 161 forks source link

Unable to do first set up using docker container #414

Closed nitobuendia closed 1 year ago

nitobuendia commented 1 year ago

Issue

Error: AttributeError: 'NoneType' object has no attribute 'replace' when trying to access localhost:8080 to kick-off OAuth process.

Steps to Reproduce

  1. Following these steps after getting the client_secret.json from these steps.

  2. Run docker-compose up (see details below). Output:

gphotos  | 03-06 13:36:01 WARNING  gphotos-sync 0.1.dev1+g208a216 2023-03-06 13:36:01.389495 
  1. Access http://localhost:8080 or http://localhost.mydomain.com:8080 (which also points at 0.0.0.0).

  2. Docker container crashes:

gphotos  | 03-06 13:36:01 WARNING  gphotos-sync 0.1.dev1+g208a216 2023-03-06 13:36:01.389495 
gphotos  | 03-06 13:36:12 ERROR    
gphotos  | Process failed. 
gphotos  | Traceback (most recent call last):
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 492, in main
gphotos  |     self.setup(args, db_path)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 341, in setup
gphotos  |     self.auth.authorize()
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos  |     flow.run_local_server(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 507, in run_local_server
gphotos  |     authorization_response = wsgi_app.last_request_uri.replace("http", "https")
gphotos  | AttributeError: 'NoneType' object has no attribute 'replace'
gphotos  | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos  | 03-06 13:36:12 WARNING  Done. 
gphotos exited with code 1

The error looks similar to this comment here. One "strange" thing is that the redirect_uri here is always http://localhost:8080 even if I access http://localhost.mydomain.com:8080 where localhost.mydomain.com is pointing at 0.0.0.0.

I can follow the URL, which would redirect back to localhost:8080 with the state and code. However, the container has crashed by then. If I restart the container before completing the process, then I get a different error:

gphotos  | 03-06 13:44:57 WARNING  gphotos-sync 0.1.dev1+g208a216 2023-03-06 13:44:57.726472 
gphotos  | 03-06 13:45:01 ERROR    
gphotos  | Process failed. 
gphotos  | Traceback (most recent call last):
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 492, in main
gphotos  |     self.setup(args, db_path)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 341, in setup
gphotos  |     self.auth.authorize()
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos  |     flow.run_local_server(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 508, in run_local_server
gphotos  |     self.fetch_token(authorization_response=authorization_response)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 300, in fetch_token
gphotos  |     return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/requests_oauthlib/oauth2_session.py", line 244, in fetch_token
gphotos  |     self._client.parse_request_uri_response(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 220, in parse_request_uri_response
gphotos  |     response = parse_authorization_code_response(uri, state=state)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 278, in parse_authorization_code_response
gphotos  |     raise MismatchingStateError()
gphotos  | oauthlib.oauth2.rfc6749.errors.MismatchingStateError: (mismatching_state) CSRF Warning! State not equal in request and response.
gphotos  | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos  | 03-06 13:45:01 WARNING  Done. 
gphotos exited with code 1

Details

gphotos-sync version: 0.1.dev1+g208a216 (from docker exec -ti gphotos gphotos-sync --version)

docker-compose.yaml

version: '3.3'

services:
    gphotos:
        image:  ghcr.io/gilesknap/gphotos-sync:latest
        restart: "no"
        container_name: gphotos
        ports:
            - '8080:8080'
        volumes:
            - ./config:/config
            - ./photos:/storage
        command: "/storage"

Command to run: docker-compose up to start container.

gilesknap commented 1 year ago

Please can you try changing your image as follows:

        image:  ghcr.io/gilesknap/gphotos-sync:3.04
nitobuendia commented 1 year ago

@gilesknap Thanks for checking and your help!

I was expecting :latest to be up to date for the latest version. Although, I understand now why the version was not 3.04 (or above) on the container.

I removed the container and recreated with image: ghcr.io/gilesknap/gphotos-sync:3.04 on docker-compose.yaml. The version on the container is also updated (gphotos | 03-06 14:06:34 WARNING gphotos-sync 3.4 2023-03-06 14:06:34.139985). (Is it expected to see 3.4 instead of 3.04 as per the image?)

Unfortunately, the error is still the same after opening http://localhost:8080 :

gphotos  | 03-06 14:06:34 WARNING  gphotos-sync 3.4 2023-03-06 14:06:34.139985 
gphotos  | 03-06 14:08:07 ERROR    
gphotos  | Process failed. 
gphotos  | Traceback (most recent call last):
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 492, in main
gphotos  |     self.setup(args, db_path)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 341, in setup
gphotos  |     self.auth.authorize()
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos  |     flow.run_local_server(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 507, in run_local_server
gphotos  |     authorization_response = wsgi_app.last_request_uri.replace("http", "https")
gphotos  | AttributeError: 'NoneType' object has no attribute 'replace'
gphotos  | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos  | 03-06 14:08:07 WARNING  Done. 

If I follow the authorization URL, after recreating the container, it would again throw an expected CSRF error:

gphotos  | 03-06 14:35:04 WARNING  gphotos-sync 3.4 2023-03-06 14:35:04.542168 
gphotos  | 03-06 14:35:05 ERROR    
gphotos  | Process failed. 
gphotos  | Traceback (most recent call last):
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 492, in main
gphotos  |     self.setup(args, db_path)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 341, in setup
gphotos  |     self.auth.authorize()
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos  |     flow.run_local_server(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 508, in run_local_server
gphotos  |     self.fetch_token(authorization_response=authorization_response)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 300, in fetch_token
gphotos  |     return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/requests_oauthlib/oauth2_session.py", line 244, in fetch_token
gphotos  |     self._client.parse_request_uri_response(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 220, in parse_request_uri_response
gphotos  |     response = parse_authorization_code_response(uri, state=state)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 278, in parse_authorization_code_response
gphotos  |     raise MismatchingStateError()
gphotos  | oauthlib.oauth2.rfc6749.errors.MismatchingStateError: (mismatching_state) CSRF Warning! State not equal in request and response.
gphotos  | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos  | 03-06 14:35:05 WARNING  Done. 
gphotos exited with code 1

Loading http://localhost.mydomain.com:8080/ instead leads to the same results.

nitobuendia commented 1 year ago

As a workaround, I will try creating my own OAuth token and passing it to .gphotos.token

gilesknap commented 1 year ago

Yeah sorry about that, this project needs a bit of love but I've not been giving it much attention due to #119. I did a workaround to the redirector by monkey patching the Google library. I made a release in a container only and incorrectly tagged it as 3.04 instead of 3.0.4.

I did a proper fix here https://github.com/googleapis/google-auth-library-python-oauthlib/pull/202 but have not re-released gphotos-sync since as this took so long to get accepted I lost momentum.

Also have an issue with PRS overwriting the latest docker image which needs fixing.

None of this helps your issue. It looks like it's not going to be easy to debug. The offending line is here https://github.com/googleapis/google-auth-library-python-oauthlib/blob/v0.8.0/google_auth_oauthlib/flow.py#L520

It seems that the call above to wsgiref.simple_server.make_server should have populated wsgi_app.last_request but failed to do so. This would take some debugging.

It may be best to wait until I have re-released the container against the most recent version of google-auth-library-python-oauthlib which has apparently reached v1.0.0 since I last looked.

Will try and take the time to look at this on Sat 11th March.

nitobuendia commented 1 year ago

Not at all. Thanks a lot for having a look and replying. Let me know when you redeploy and super happy to test. No rush.


From my side, I tried generating the OAuth token using Apps Script and a variant of this code and it seemed to work yesterday. I need to verify the tool can renew the token once it expires, so I will be monitoring it.

Update: The workaround does not seem to work. While the token works when it's generated and photos/videos can be downloaded, after some time (I guess once it's expired), I start getting Auth (401) errors:

03-07 12:39:28 ERROR    Request failed with status 401: b'{\n  "error": {\n    "code": 401,\n    "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",\n    "status": "UNAUTHENTICATED"\n  }\n}\n' 

My guess is that once it's time to renew, either the app is not renewing the expired token or, more likely, it's not able to renew the token. One potential issue here is that I generated it using "web" client instead of a "desktop" client (although I adapted client_secrets.json to match desktop ("installed" instead of "web"). I am trying now with a "desktop app" oauth token and see if that makes a difference.

Update 2: Generating the token externally as a Desktop one (as per instructions) seems to have worked.

gilesknap commented 1 year ago

Hi @nitobuendia I've released a new version that uses the latest google auth library. https://github.com/gilesknap/gphotos-sync/pkgs/container/gphotos-sync/76608572?tag=3.1.0

It would be useful to know if this has any bearing on your issue.

Thanks.

nitobuendia commented 1 year ago

Seems still broken to me :(

docker-compose.yaml

version: '3.3'

services:
    gphotos:
        image:  ghcr.io/gilesknap/gphotos-sync:3.1.0
        restart: "no"
        container_name: gphotos
        ports:
            - '8080:8080'
        volumes:
            - ./config:/config
            - ./photos:/storage
        command: "/storage"

Run: docker-compose up

docker-compose up
[+] Running 1/0
 ⠿ Container gphotos  Created                                                         0.0s
Attaching to gphotos
gphotos  | 03-11 11:59:01 WARNING  gphotos-sync 3.1.0 2023-03-11 11:59:01.777368 

Access http://localhost:8080 -- Question: is this step correct?

Docker container exits with code 1. Same http to https rename issue.

gphotos  | 03-11 12:00:24 ERROR    
gphotos  | Process failed. 
gphotos  | Traceback (most recent call last):
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 506, in main
gphotos  |     self.setup(args, db_path)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 353, in setup
gphotos  |     self.auth.authorize()
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos  |     flow.run_local_server(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 446, in run_local_server
gphotos  |     authorization_response = wsgi_app.last_request_uri.replace("http", "https")
gphotos  | AttributeError: 'NoneType' object has no attribute 'replace'
gphotos  | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos  | 03-11 12:00:25 WARNING  Done. 
gphotos exited with code 1

Access OAuth URL: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline

Then, (re)start docker container.

docker-compose up
[+] Running 1/0
 ⠿ Container gphotos  Created                                                         0.0s
Attaching to gphotos
gphotos  | 03-11 12:01:29 WARNING  gphotos-sync 3.1.0 2023-03-11 12:01:29.459681 

Complete the OAuth URL process.

Lands on http://localhost:8080/?state=<redacted>&code=<redacted>&scope=https://www.googleapis.com/auth/photoslibrary.sharing%20https://www.googleapis.com/auth/photoslibrary.readonly

Docker container exits with code 1. Same CSRF issue.

gphotos  | 03-11 12:04:34 ERROR    
gphotos  | Process failed. 
gphotos  | Traceback (most recent call last):
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 506, in main
gphotos  |     self.setup(args, db_path)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 353, in setup
gphotos  |     self.auth.authorize()
gphotos  |   File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos  |     flow.run_local_server(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 447, in run_local_server
gphotos  |     self.fetch_token(authorization_response=authorization_response)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 285, in fetch_token
gphotos  |     return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/requests_oauthlib/oauth2_session.py", line 244, in fetch_token
gphotos  |     self._client.parse_request_uri_response(
gphotos  |   File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 220, in parse_request_uri_response
gphotos  |     response = parse_authorization_code_response(uri, state=state)
gphotos  |   File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 278, in parse_authorization_code_response
gphotos  |     raise MismatchingStateError()
gphotos  | oauthlib.oauth2.rfc6749.errors.MismatchingStateError: (mismatching_state) CSRF Warning! State not equal in request and response.
gphotos  | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos  | 03-11 12:04:34 WARNING  Done. 
gphotos exited with code 1

Same happens if I try to access https://localhost.mydomain.com:8080.

gilesknap commented 1 year ago

Thanks for getting back to me.

I'm not clear on what you mean by "is this step correct?" Do you only see it going to localhost:8080 without the rest of the OAutth URL? if so then that is wrong.

I'm going to step you through what I did to make this work in a container on a server today. Please try my steps and let me know if they work for you. Then we can debug the compose if that is causing the issue.

(I'm using podman instead of docker and different folders to you, everything else should be identical, I also use 8090 because there is another gphotos running)

nitobuendia commented 1 year ago

@gilesknap Thanks for this. Reading your steps, I think I know the problem. I was actually opening http://localhost:8080 in the browser after doing docker-compose up. I think I shouldn't do based on your steps, right? I imagine that causes the system to think that I am back from the OAuth link when that's no the case. I was doing that because it never gives me the URL straight away, so I thought I had to do it to trigger it. Only when loading localhost:8080, the URL is generated.

My docker-compose set up seems to work (at least post setup) since I am currently running gPhotos for a few days with 0 issues. When it completes, I will: (1) test with my current set up, without accessing localhost URL first and waiting until the OAuth URL is auto-generated; (2) should that fail, I will follow your exact steps too.

Give me a few days since I don't want to risk interrupting the current process which has been running for a few days. If it takes too long, I will generate new credentials (client_secret separate from the one I use now) and test.

gilesknap commented 1 year ago

OK that makes sense now. I'm going to do some work on the documentation soon. If you have any ideas about how to improve it then let me know.

I think I'm going to put the steps I listed above as the recommended approach as it is the simplest way to get things working as long as your browser can route to your container.

nitobuendia commented 1 year ago

@gilesknap I managed to spend some time on this today.

The issue is definitely not on your documentation. This being said, I think there are two options to cover this use case:

  1. A note under the current set up warning about the "incompatibility". Add on this page a note on the lines of:

Note: While you can use docker-compose to run gphotos-sync, make sure you do initial set up with docker-composer run <container_name>. The OAuth URL may not be generated when starting the container with docker-compose up.

  1. Another approach would be to add a new whole section about running gphotos-sync using docker-compose. For this, you could take my code from the first comment. Then add the command(s) docker-compose run <container_name> and docker-compose up -d perhaps the same note as the previous point.

Do you have PayPal, or BuyMeABeer?

gilesknap commented 1 year ago

Thanks @nitobuendia.

I just saw this https://stackoverflow.com/questions/33066528/should-i-use-docker-compose-up-or-run

It says if you use run you are not exposing the ports. So that will be why you get ERR_CONNECTION_REFUSED.

I can also see a difference between my invocation of docker and what your compose does. I use the flags -it, there is some possibility that this is stopping the oauth library from outputting the auth URL because it sees its not running in a terminal

I wonder if this would work?

version: '3.3'

services:
    gphotos:
        image:  ghcr.io/gilesknap/gphotos-sync:latest
        stdin_open: true # docker run -i
        tty: true        # docker run -t
        restart: "no"
        container_name: gphotos
        ports:
            - '8080:8080'
        volumes:
            - ./config:/config
            - ./photos:/storage
        command: "/storage"

(Its hard for me to try this as I am a podman user these days)

If it does work I;ll add a compose page to the docs as you suggest

nitobuendia commented 1 year ago

That worked. The OAuth was generated fine end to end with docker-compose up after adding your suggested lines:

        stdin_open: true # docker run -i
        tty: true        # docker run -t

For completeness, I've also tried docker-compose run --service-ports after reading your post and similar guidance.

The experience was a bit flaky. The first couple of times it took very long to load the Redirect URL (http://localhost:8080/?state=<redacted>&code=<redacted>&scope=https://www.googleapis.com/auth/photoslibrary.sharing%20https://www.googleapis.com/auth/photoslibrary.readonly) and the container ended up throwing an error:

03-19 06:28:10 ERROR    
Process failed. 
Traceback (most recent call last):
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 506, in main
    self.setup(args, db_path)
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 353, in setup
    self.auth.authorize()
  File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
    flow.run_local_server(
  File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 446, in run_local_server
    authorization_response = wsgi_app.last_request_uri.replace("http", "https")
AttributeError: 'NoneType' object has no attribute 'replace'
03-19 06:28:10 WARNING  Done. 

However, after a couple of runs, it worked fine somehow, but I couldn't reproduce again. Nonetheless, I think the docker-compose up with your fixes would be best

gilesknap commented 1 year ago

Great. Thanks for letting me know.

I'll get the docs updated soon

gilesknap commented 1 year ago

Added document compose to #417