Closed nitobuendia closed 1 year ago
Please can you try changing your image as follows:
image: ghcr.io/gilesknap/gphotos-sync:3.04
@gilesknap Thanks for checking and your help!
I was expecting :latest
to be up to date for the latest version. Although, I understand now why the version was not 3.04 (or above) on the container.
I removed the container and recreated with image: ghcr.io/gilesknap/gphotos-sync:3.04
on docker-compose.yaml
. The version on the container is also updated (gphotos | 03-06 14:06:34 WARNING gphotos-sync 3.4 2023-03-06 14:06:34.139985
). (Is it expected to see 3.4 instead of 3.04 as per the image?)
Unfortunately, the error is still the same after opening http://localhost:8080 :
gphotos | 03-06 14:06:34 WARNING gphotos-sync 3.4 2023-03-06 14:06:34.139985
gphotos | 03-06 14:08:07 ERROR
gphotos | Process failed.
gphotos | Traceback (most recent call last):
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 492, in main
gphotos | self.setup(args, db_path)
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 341, in setup
gphotos | self.auth.authorize()
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos | flow.run_local_server(
gphotos | File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 507, in run_local_server
gphotos | authorization_response = wsgi_app.last_request_uri.replace("http", "https")
gphotos | AttributeError: 'NoneType' object has no attribute 'replace'
gphotos | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos | 03-06 14:08:07 WARNING Done.
If I follow the authorization URL, after recreating the container, it would again throw an expected CSRF error:
gphotos | 03-06 14:35:04 WARNING gphotos-sync 3.4 2023-03-06 14:35:04.542168
gphotos | 03-06 14:35:05 ERROR
gphotos | Process failed.
gphotos | Traceback (most recent call last):
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 492, in main
gphotos | self.setup(args, db_path)
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 341, in setup
gphotos | self.auth.authorize()
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos | flow.run_local_server(
gphotos | File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 508, in run_local_server
gphotos | self.fetch_token(authorization_response=authorization_response)
gphotos | File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 300, in fetch_token
gphotos | return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
gphotos | File "/root/.local/lib/python3.10/site-packages/requests_oauthlib/oauth2_session.py", line 244, in fetch_token
gphotos | self._client.parse_request_uri_response(
gphotos | File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 220, in parse_request_uri_response
gphotos | response = parse_authorization_code_response(uri, state=state)
gphotos | File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 278, in parse_authorization_code_response
gphotos | raise MismatchingStateError()
gphotos | oauthlib.oauth2.rfc6749.errors.MismatchingStateError: (mismatching_state) CSRF Warning! State not equal in request and response.
gphotos | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos | 03-06 14:35:05 WARNING Done.
gphotos exited with code 1
Loading http://localhost.mydomain.com:8080/ instead leads to the same results.
As a workaround, I will try creating my own OAuth token and passing it to .gphotos.token
Yeah sorry about that, this project needs a bit of love but I've not been giving it much attention due to #119. I did a workaround to the redirector by monkey patching the Google library. I made a release in a container only and incorrectly tagged it as 3.04 instead of 3.0.4.
I did a proper fix here https://github.com/googleapis/google-auth-library-python-oauthlib/pull/202 but have not re-released gphotos-sync since as this took so long to get accepted I lost momentum.
Also have an issue with PRS overwriting the latest docker image which needs fixing.
None of this helps your issue. It looks like it's not going to be easy to debug. The offending line is here https://github.com/googleapis/google-auth-library-python-oauthlib/blob/v0.8.0/google_auth_oauthlib/flow.py#L520
It seems that the call above to wsgiref.simple_server.make_server should have populated wsgi_app.last_request but failed to do so. This would take some debugging.
It may be best to wait until I have re-released the container against the most recent version of google-auth-library-python-oauthlib which has apparently reached v1.0.0 since I last looked.
Will try and take the time to look at this on Sat 11th March.
Not at all. Thanks a lot for having a look and replying. Let me know when you redeploy and super happy to test. No rush.
From my side, I tried generating the OAuth token using Apps Script and a variant of this code and it seemed to work yesterday. I need to verify the tool can renew the token once it expires, so I will be monitoring it.
Update: The workaround does not seem to work. While the token works when it's generated and photos/videos can be downloaded, after some time (I guess once it's expired), I start getting Auth (401) errors:
03-07 12:39:28 ERROR Request failed with status 401: b'{\n "error": {\n "code": 401,\n "message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",\n "status": "UNAUTHENTICATED"\n }\n}\n'
My guess is that once it's time to renew, either the app is not renewing the expired token or, more likely, it's not able to renew the token. One potential issue here is that I generated it using "web" client instead of a "desktop" client (although I adapted client_secrets.json
to match desktop ("installed" instead of "web"). I am trying now with a "desktop app" oauth token and see if that makes a difference.
Update 2: Generating the token externally as a Desktop one (as per instructions) seems to have worked.
Hi @nitobuendia I've released a new version that uses the latest google auth library. https://github.com/gilesknap/gphotos-sync/pkgs/container/gphotos-sync/76608572?tag=3.1.0
It would be useful to know if this has any bearing on your issue.
Thanks.
Seems still broken to me :(
docker-compose.yaml
version: '3.3'
services:
gphotos:
image: ghcr.io/gilesknap/gphotos-sync:3.1.0
restart: "no"
container_name: gphotos
ports:
- '8080:8080'
volumes:
- ./config:/config
- ./photos:/storage
command: "/storage"
Run: docker-compose up
docker-compose up
[+] Running 1/0
⠿ Container gphotos Created 0.0s
Attaching to gphotos
gphotos | 03-11 11:59:01 WARNING gphotos-sync 3.1.0 2023-03-11 11:59:01.777368
Access http://localhost:8080
-- Question: is this step correct?
Docker container exits with code 1. Same http to https rename issue.
gphotos | 03-11 12:00:24 ERROR
gphotos | Process failed.
gphotos | Traceback (most recent call last):
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 506, in main
gphotos | self.setup(args, db_path)
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 353, in setup
gphotos | self.auth.authorize()
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos | flow.run_local_server(
gphotos | File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 446, in run_local_server
gphotos | authorization_response = wsgi_app.last_request_uri.replace("http", "https")
gphotos | AttributeError: 'NoneType' object has no attribute 'replace'
gphotos | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos | 03-11 12:00:25 WARNING Done.
gphotos exited with code 1
Access OAuth URL: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
Then, (re)start docker container.
docker-compose up
[+] Running 1/0
⠿ Container gphotos Created 0.0s
Attaching to gphotos
gphotos | 03-11 12:01:29 WARNING gphotos-sync 3.1.0 2023-03-11 12:01:29.459681
Complete the OAuth URL process.
Lands on http://localhost:8080/?state=<redacted>&code=<redacted>&scope=https://www.googleapis.com/auth/photoslibrary.sharing%20https://www.googleapis.com/auth/photoslibrary.readonly
Docker container exits with code 1. Same CSRF issue.
gphotos | 03-11 12:04:34 ERROR
gphotos | Process failed.
gphotos | Traceback (most recent call last):
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 506, in main
gphotos | self.setup(args, db_path)
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 353, in setup
gphotos | self.auth.authorize()
gphotos | File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
gphotos | flow.run_local_server(
gphotos | File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 447, in run_local_server
gphotos | self.fetch_token(authorization_response=authorization_response)
gphotos | File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 285, in fetch_token
gphotos | return self.oauth2session.fetch_token(self.client_config["token_uri"], **kwargs)
gphotos | File "/root/.local/lib/python3.10/site-packages/requests_oauthlib/oauth2_session.py", line 244, in fetch_token
gphotos | self._client.parse_request_uri_response(
gphotos | File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/clients/web_application.py", line 220, in parse_request_uri_response
gphotos | response = parse_authorization_code_response(uri, state=state)
gphotos | File "/root/.local/lib/python3.10/site-packages/oauthlib/oauth2/rfc6749/parameters.py", line 278, in parse_authorization_code_response
gphotos | raise MismatchingStateError()
gphotos | oauthlib.oauth2.rfc6749.errors.MismatchingStateError: (mismatching_state) CSRF Warning! State not equal in request and response.
gphotos | Please visit this URL to authorize this application: https://accounts.google.com/o/oauth2/auth?response_type=code&client_id=<redacted>.apps.googleusercontent.com&redirect_uri=http%3A%2F%2Flocalhost%3A8080%2F&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.readonly+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fphotoslibrary.sharing&state=<redacted>&access_type=offline
gphotos | 03-11 12:04:34 WARNING Done.
gphotos exited with code 1
Same happens if I try to access https://localhost.mydomain.com:8080
.
Thanks for getting back to me.
I'm not clear on what you mean by "is this step correct?" Do you only see it going to localhost:8080 without the rest of the OAutth URL? if so then that is wrong.
I'm going to step you through what I did to make this work in a container on a server today. Please try my steps and let me know if they work for you. Then we can debug the compose if that is causing the issue.
(I'm using podman instead of docker and different folders to you, everything else should be identical, I also use 8090 because there is another gphotos running)
@gilesknap Thanks for this. Reading your steps, I think I know the problem. I was actually opening http://localhost:8080 in the browser after doing docker-compose up
. I think I shouldn't do based on your steps, right? I imagine that causes the system to think that I am back from the OAuth link when that's no the case. I was doing that because it never gives me the URL straight away, so I thought I had to do it to trigger it. Only when loading localhost:8080, the URL is generated.
My docker-compose set up seems to work (at least post setup) since I am currently running gPhotos for a few days with 0 issues. When it completes, I will: (1) test with my current set up, without accessing localhost URL first and waiting until the OAuth URL is auto-generated; (2) should that fail, I will follow your exact steps too.
Give me a few days since I don't want to risk interrupting the current process which has been running for a few days. If it takes too long, I will generate new credentials (client_secret separate from the one I use now) and test.
OK that makes sense now. I'm going to do some work on the documentation soon. If you have any ideas about how to improve it then let me know.
I think I'm going to put the steps I listed above as the recommended approach as it is the simplest way to get things working as long as your browser can route to your container.
@gilesknap I managed to spend some time on this today.
docker run commands works as expected, similar to your experience (docker run -dit --name gphotostest --rm -v /path-to/config:/config -v /path-to/photos:/storage -p 8080:8080 -it ghcr.io/gilesknap/gphotos-sync:3.1.0 --port 8080 /storage
)
When I run docker-compose up
gphotos | 03-18 12:18:27 WARNING gphotos-sync 3.1.0 2023-03-18 12:18:27.270694
). I waited for 5-10 minutes, whereas docker run
was immediate.docker-compose run gphotos
and that generated the OAuth URL fine too. Although when landing on the redirect URL, it returned ERR_CONNECTION_REFUSED
(http://localhost:8080/?state=<redacted>&code=<redacted>&scope=https://www.googleapis.com/auth/photoslibrary.sharing%20https://www.googleapis.com/auth/photoslibrary.readonly
(probably issue on my side). Running docker-compose run gphotos
after token is set up, also works fine.The issue is definitely not on your documentation. This being said, I think there are two options to cover this use case:
Note: While you can use
docker-compose
to run gphotos-sync, make sure you do initial set up withdocker-composer run <container_name>
. The OAuth URL may not be generated when starting the container withdocker-compose up
.
docker-compose run <container_name>
and docker-compose up -d
perhaps the same note as the previous point.Do you have PayPal, or BuyMeABeer?
Thanks @nitobuendia.
I just saw this https://stackoverflow.com/questions/33066528/should-i-use-docker-compose-up-or-run
It says if you use run you are not exposing the ports. So that will be why you get ERR_CONNECTION_REFUSED
.
I can also see a difference between my invocation of docker and what your compose does. I use the flags -it, there is some possibility that this is stopping the oauth library from outputting the auth URL because it sees its not running in a terminal
I wonder if this would work?
version: '3.3'
services:
gphotos:
image: ghcr.io/gilesknap/gphotos-sync:latest
stdin_open: true # docker run -i
tty: true # docker run -t
restart: "no"
container_name: gphotos
ports:
- '8080:8080'
volumes:
- ./config:/config
- ./photos:/storage
command: "/storage"
(Its hard for me to try this as I am a podman user these days)
If it does work I;ll add a compose page to the docs as you suggest
That worked. The OAuth was generated fine end to end with docker-compose up
after adding your suggested lines:
stdin_open: true # docker run -i
tty: true # docker run -t
For completeness, I've also tried docker-compose run --service-ports
after reading your post and similar guidance.
The experience was a bit flaky. The first couple of times it took very long to load the Redirect URL (http://localhost:8080/?state=<redacted>&code=<redacted>&scope=https://www.googleapis.com/auth/photoslibrary.sharing%20https://www.googleapis.com/auth/photoslibrary.readonly
) and the container ended up throwing an error:
03-19 06:28:10 ERROR
Process failed.
Traceback (most recent call last):
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 506, in main
self.setup(args, db_path)
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/Main.py", line 353, in setup
self.auth.authorize()
File "/root/.local/lib/python3.10/site-packages/gphotos_sync/authorize.py", line 95, in authorize
flow.run_local_server(
File "/root/.local/lib/python3.10/site-packages/google_auth_oauthlib/flow.py", line 446, in run_local_server
authorization_response = wsgi_app.last_request_uri.replace("http", "https")
AttributeError: 'NoneType' object has no attribute 'replace'
03-19 06:28:10 WARNING Done.
However, after a couple of runs, it worked fine somehow, but I couldn't reproduce again. Nonetheless, I think the docker-compose up
with your fixes would be best
Great. Thanks for letting me know.
I'll get the docs updated soon
Added document compose to #417
Issue
Error:
AttributeError: 'NoneType' object has no attribute 'replace'
when trying to access localhost:8080 to kick-off OAuth process.Steps to Reproduce
Following these steps after getting the
client_secret.json
from these steps.Run
docker-compose up
(see details below). Output:Access http://localhost:8080 or http://localhost.mydomain.com:8080 (which also points at 0.0.0.0).
Docker container crashes:
The error looks similar to this comment here. One "strange" thing is that the redirect_uri here is always http://localhost:8080 even if I access http://localhost.mydomain.com:8080 where localhost.mydomain.com is pointing at 0.0.0.0.
I can follow the URL, which would redirect back to localhost:8080 with the state and code. However, the container has crashed by then. If I restart the container before completing the process, then I get a different error:
Details
gphotos-sync version: 0.1.dev1+g208a216 (from
docker exec -ti gphotos gphotos-sync --version
)docker-compose.yaml
Command to run:
docker-compose up
to start container.