search

upwork / python-upwork-oauth2

Python bindings for Upwork API (OAuth2)
Apache License 2.0
37 stars 10 forks source link

Refresh token (or token refresh?) issue #11

Closed Kick28 closed 5 months ago

Kick28 commented 6 months ago

@mnovozhylov Hi!

After switching to GraphQL API, there is a problem with refreshing the token automatically.

Here are the error logs:

Apr 04 10:45:36 python3[44349]: Error while fetching the API
Apr 04 10:45:36 python3[44349]: (invalid_grant) Refresh token is expired or revoked or not registered

Every few days I need to manually run the script to update my token. The script I'm running is the get_desktop_client() from example/myapp.py. At first, I thought there was some kind of problem with the grant type, so in the script from above I changed the way of initializing of Upwork config, but unfortunately, this didn't help:

upwork_config = upwork.Config(
    {
        "client_id": config['UPWORK_API_CONSUMER_KEY'],
        "client_secret": config['UPWORK_API_CONSUMER_SECRET'],
        "redirect_uri": config['UPWORK_API_REDIRECT_URL'],
        "grant_type": "refresh_token"
    }
)

So, maybe do you know what is the cause of this problem? Also, if I should provide you with another kind of information (code, config, etc.), just tell me. Thanks in advance for your help!

mnovozhylov commented 6 months ago

Hi @Kick28 ,

a new refresh token is valid for 14 days, while we change expiration date for old refresh token to now + 24 hours. So, once you refresh a token, please, make sure that you pick the new refresh token from the response and refresh it in your security storage. Please, check this comment https://github.com/upwork/python-upwork-oauth2/blob/master/example/myapp.py#L58

Kick28 commented 6 months ago

Yup, double-checked everything, and the new token from the response is saved. The client.get_actual_config() returns the new token as well.

eliyaba commented 6 months ago

@mnovozhylov I am seeing the same issue since change to GraphQL, I have a cron running periodically, after the first 24 hours getting (invalid_grant) Refresh token is expired or revoked or not registered The only way I get it to work is manually refreshing the token.

Any help appreciated.

Kick28 commented 6 months ago

@eliyaba Yep, refreshing the key manually is quite annoying :(

Kick28 commented 6 months ago

@mnovozhylov Sorry to bother you again, but is there any way to fix this issue?

mnovozhylov commented 6 months ago

Hi @Kick28 ,

Please, remove this line first "grant_type": "refresh_token" from the config, it's not needed. Do you assign and save token as shown in the example?

token = {'access_token': 'oauth2v2_********', 'expires_at': 1640339543.5184577, 'expires_in': '86400', 'refresh_token': 'oauth2v2_*************', 'token_type': 'Bearer'}
config = upwork.Config({'client_id': config.client_id, 'client_secret': config.client_secret, 'token': token})

Could you please share your flow? Also, please, specify requests-oauthlib version used in your project

P.S. GraphQL is not responsible for the authentication flow.

Kick28 commented 6 months ago

Hi @mnovozhylov ! The line grant_type was added as a possible way to fix this issue, but this didn't work. Sure, here is my flow: 1) Firstly, I refresh my key with get_dekstop_client() from example/myapp.py. The response from client.get_access_token(authz_code) is saved as a JSON file token.json in the same format you specified. 2) Then, in my application I initialize the upwork.Config and upwork.Client as following:


with open('./config.json') as f:
    config = json.load(f)

with open('./token.json') as f:
    token = json.load(f)

upwork_config = upwork.Config(
    {
        "client_id": config['UPWORK_API_CONSUMER_KEY'],
        "client_secret": config['UPWORK_API_CONSUMER_SECRET'],
        "redirect_uri": config['UPWORK_API_REDIRECT_URL'],
        "token": token
    }
)

client = upwork.Client(upwork_config)

3) Use this client object to perform my requests

The version of requests-oauthlib used is 1.3.1.

eliyaba commented 6 months ago

@mnovozhylov thanks for your response.

I am also using requests-oauthlib==1.3.1, usage quite similar to @Kick28. My redirect url saves the generated token successfully to a Mysql database. Then app uses this code to create a client and refresh the token. This runs successfully in the first 24 hours and then throws (invalid_grant) Refresh token is expired or revoked or not registered Exception.

        if token:
            self.config = Config({
                'client_id': consumer_key,
                'client_secret': consumer_secret,
                'token': token,
                'redirect_uri': redirect_url
        })
        else:
            self.config = Config({
                'client_id': consumer_key,
                'client_secret': consumer_secret,
                'redirect_uri': redirect_url
            })

        self.client = Client(self.config)
        self.config = self.client.get_actual_config() ## used to make sure token is refreshed
        if hasattr(self.config, 'token') and self.config.token:
            token = UpworkAuthToken.insert_or_update(
                self.config.token['access_token'],
                self.config.token['refresh_token'],
                self.config.token['token_type'],
                self.config.token['expires_in'],
                self.config.token['expires_at']
            )
            db.session.commit()

        self.organization_id = self.get_organization_id()

Thank you

eliyaba commented 5 months ago

@mnovozhylov @Kick28 any news regarding this issue? Thank you.

Kick28 commented 5 months ago

@mnovozhylov @Kick28 any news regarding this issue? Thank you.

Nope, still waiting for the fix.

mnovozhylov commented 5 months ago

// FIRST SESSION: TOKEN UNKNOWN

Enable logging for requests_oauthlib if you want to debug your issue.

import logging
import sys
log = logging.getLogger('requests_oauthlib')
log.addHandler(logging.StreamHandler(sys.stdout))
log.setLevel(logging.DEBUG)

to the test script.

Initial config:

config = upwork.Config(
{
"client_id": "xxxxx_your_key_xxxxx",
"client_secret": "xxxxx_key_secret_xxxxx",
"redirect_uri": "https://www.upwork.com/developer/keys/apply"
}
)

Execute script:


Emulating desktop app
Generated new state VtbXjiUO67UXxWFWgGddvHNRNrU2Ho.
Please enter the full callback URL you get following this link:
https://www.upwork.com/ab/account-security/oauth2/authorize?response_type=code&client_id=xxxxx_your_key_xxxxx&redirect_uri=https%3A%2F%2Fwww.upwork.com%2Fdeveloper%2Fkeys%2Fapply&state=VtbXjiUO67UXxWFWgGddvHNRNrU2Ho

https://www.upwork.com/developer/keys/apply?code=xxxxxxxxxxxxxxxx&state=VtbXjiUO67UXxWFWgGddvHNRNrU2Ho Retrieving access and refresh tokens.... Encoding client_id "xxxxx_your_key_xxxxx" with client_secret as Basic auth credentials. Requesting url https://www.upwork.com/api/v3/oauth2/token using method POST. Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'authorization_code', 'code': 'xxxxxxxxxxxxxxxx', 'redirect_uri': 'https://www.upwork.com/developer/keys/apply'} Passing through key word arguments {'timeout': None, 'auth': <requests.auth.HTTPBasicAuth object at 0x7c81826193d0>, 'verify': True, 'proxies': None}. Request to fetch token completed with status 200. Request url was https://www.upwork.com/api/v3/oauth2/token Request headers were {'User-Agent': 'python-requests/2.28.1', 'Accept-Encoding': 'gzip, deflate, br', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8', 'Content-Length': '136', 'Authorization': 'Basic xxxxxxxxxxxxxxxxxxxxxxxxxxxxx'} Request body was grant_type=authorization_code&code=xxxxxxxxxxxxxxxx&redirect_uri=https%3A%2F%2Fwww.upwork.com%2Fdeveloper%2Fkeys%2Fapply Response headers were {'Date': 'Thu, 25 Apr 2024 19:51:40 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'vnd.eo.request-id': '0f413bd7-6a87-4773-b247-b55fa48233f3', 'vnd.odesk.request-id': '0f413bd7-6a87-4773-b247-b55fa48233f3', 'vnd-eo-trace-id': '87a0de7e6b704bdb-MXP, 87a0de7e6b704bdb-MXP', 'vnd-eo-span-id': '598a5b64-0349-4125-9c77-fb0fe9bd921b', 'vnd-eo-parent-span-id': '87a0de7e6b704bdb-MXP', 'vnd.eo.trace-id': '87a0de7e6b704bdb-MXP', 'vnd.eo.span-id': '598a5b64-0349-4125-9c77-fb0fe9bd921b', 'vnd.eo.parent-span-id': '87a0de7e6b704bdb-MXP', 'x-upwork-target-status': 'true', 'vnd-eo-server-sr': '2024-04-25T19:51:40.686Z', 'vnd-eo-server-ss': '2024-04-25T19:51:40.756Z', 'vnd-eo-prana-client-receive': '2024-04-25T19:51:40.756Z, 2024-04-25T19:51:40.757Z', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'referrer-policy': 'origin-when-cross-origin', 'x-xss-protection': '1; mode=block', 'CF-Cache-Status': 'DYNAMIC', 'Set-Cookie': 'xxxxxxxxx', 'X-Content-Type-Options': 'nosniff', 'Server': 'cloudflare', 'CF-RAY': '87a0de7e6b704bdb-MXP', 'Content-Encoding': 'br', 'alt-svc': 'h3=":443"; ma=86400'} and content {"access_token":"xxxxx_oauth2_access_token_xxxxx","refresh_token":"xxxxx_oauth2_refresh_token_xxxxx","token_type":"Bearer","expires_in":86400}. Invoking 0 token response hooks. Obtained token {'access_token': 'xxxxx_oauth2_access_token_xxxxx', 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer', 'expires_in': 86400, 'expires_at': 1714161100.9841394}. {'access_token': 'xxxxx_oauth2_access_token_xxxxx', 'expires_at': 1714161100.9841394, 'expires_in': 86400, 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer'} OK My info Invoking 0 protected resource request hooks. Adding token {'access_token': 'xxxxx_oauth2_access_token_xxxxx', 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer', 'expires_in': 86400, 'expires_at': 1714161100.9841394} to request. Requesting url https://api.upwork.com/graphql using method POST. Supplying headers {'Content-type': 'application/json', 'Authorization': 'Bearer xxxxx_oauth2_access_token_xxxxx'} and data None Passing through key word arguments {'json': {'query': '\n query {\n user {\n id\n nid\n rid\n }\n organization {\n id\n }\n }\n '}}. {'data': {'organization': {'id': 'xxxxxxxxxxxxxxxx'}, 'user': {'id': 'xxxxxxxxxxxxxxxx', 'nid': 'xxxxxxxxxxxxxxxx', 'rid': 'xxxxxxxxxxxxxxxx'}}}

// SECOND SESSION: ACCESS TOKEN IS KNOWN AND ACTIVE

Assume that the token is saved, retrive it and set as:

token = {'access_token': 'xxxxx_oauth2_access_token_xxxxx', 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer', 'expires_in': 86400, 'expires_at': 1640339543.5184577}
config = upwork.Config({'client_id': config.client_id, 'client_secret': config.client_secret, 'token': token})

Execute script:

Emulating desktop app
My info
Invoking 0 protected resource request hooks.
Adding token {'access_token': 'xxxxx_oauth2_access_token_xxxxx', 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer', 'expires_in': 86400, 'expires_at': 1714161100.9841394} to request.
Requesting url https://api.upwork.com/graphql using method POST.
Supplying headers {'Content-type': 'application/json', 'Authorization': 'Bearer xxxxx_oauth2_access_token_xxxxx'} and data None
Passing through key word arguments {'json': {'query': '\n    query {\n      user {\n        id\n        nid\n        rid\n      }\n      organization {\n        id\n      }\n    }\n        '}}.
{'data': {'organization': {'id': 'xxxxxxxxxxxxxxxx'},
'user': {'id': 'xxxxxxxxxxxxxxxx',
'nid': 'xxxxxxxxxxxxxxxx',
'rid': 'xxxxxxxxxxxxxxxx'}}}

// THIRD SESSION: AUTO-REFRESHING

Assume that the token is saved but expired - change 'expires_at' (which is 1714161100.9841394 at the moment) to some past time - 1640339543.5184577

token = {'access_token': 'xxxxx_oauth2_access_token_xxxxx', 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer', 'expires_in': 86400, 'expires_at': 1640339543.5184577}
config = upwork.Config({'client_id': config.client_id, 'client_secret': config.client_secret, 'token': token})

Execute script:

Emulating desktop app
My info
Invoking 0 protected resource request hooks.
Adding token {'access_token': 'xxxxx_oauth2_access_token_xxxxx', 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer', 'expires_in': 86400, 'expires_at': 1640339543.5184577} to request.
Auto refresh is set, attempting to refresh at https://www.upwork.com/api/v3/oauth2/token.
Adding auto refresh key word arguments {'client_id': 'xxxxx_your_key_xxxxx', 'client_secret': 'xxxxx_key_secret_xxxxx'}.
Prepared refresh token request body grant_type=refresh_token&client_id=xxxxx_your_key_xxxxx&client_secret=xxxxx_key_secret_xxxxx&refresh_token=xxxxx_oauth2_refresh_token_xxxxx&json=%7B%27query%27%3A+%27%5Cn++++query+%7B%5Cn++++++user+%7B%5Cn++++++++id%5Cn++++++++nid%5Cn++++++++rid%5Cn++++++%7D%5Cn++++++organization+%7B%5Cn++++++++id%5Cn++++++%7D%5Cn++++%7D%5Cn++++++++%27%7D
Requesting url https://www.upwork.com/api/v3/oauth2/token using method POST.
Supplying headers {'Accept': 'application/json', 'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'} and data {'grant_type': 'refresh_token', 'client_id': 'xxxxx_your_key_xxxxx', 'client_secret': 'xxxxx_key_secret_xxxxx', 'refresh_token': 'xxxxx_oauth2_refresh_token_xxxxx', 'json': "{'query': '\\n    query {\\n      user {\\n        id\\n        nid\\n        rid\\n      }\\n      organization {\\n        id\\n      }\\n    }\\n        '}"}
Passing through key word arguments {'json': None, 'auth': None, 'timeout': None, 'verify': True, 'proxies': None}.
Request to refresh token completed with status 200.
Response headers were {'Date': 'Thu, 25 Apr 2024 19:55:25 GMT', 'Content-Type': 'application/json', 'Transfer-Encoding': 'chunked', 'Connection': 'keep-alive', 'vnd.eo.request-id': 'ce462d86-3fbb-4b55-9627-f2d421c53bc0', 'vnd.odesk.request-id': 'ce462d86-3fbb-4b55-9627-f2d421c53bc0', 'vnd-eo-trace-id': '87a0e3fb3bea4bed-MXP, 87a0e3fb3bea4bed-MXP', 'vnd-eo-span-id': 'ae909178-cc56-4ea6-a9ec-1d9916fecfa8', 'vnd-eo-parent-span-id': '87a0e3fb3bea4bed-MXP', 'vnd.eo.trace-id': '87a0e3fb3bea4bed-MXP', 'vnd.eo.span-id': 'ae909178-cc56-4ea6-a9ec-1d9916fecfa8', 'vnd.eo.parent-span-id': '87a0e3fb3bea4bed-MXP', 'x-upwork-target-status': 'true', 'vnd-eo-server-sr': '2024-04-25T19:55:25.468Z', 'vnd-eo-server-ss': '2024-04-25T19:55:25.524Z', 'vnd-eo-prana-client-receive': '2024-04-25T19:55:25.523Z, 2024-04-25T19:55:25.524Z', 'strict-transport-security': 'max-age=31536000; includeSubDomains; preload', 'referrer-policy': 'origin-when-cross-origin', 'x-xss-protection': '1; mode=block', 'CF-Cache-Status': 'DYNAMIC', 'Set-Cookie': 'xxxxxxxxxxxxxxxxxxxxxxxx', 'X-Content-Type-Options': 'nosniff', 'Server': 'cloudflare', 'CF-RAY': '87a0e3fb3bea4bed-MXP', 'Content-Encoding': 'br', 'alt-svc': 'h3=":443"; ma=86400'} and content {"access_token":"NEW_xxxxx_oauth2_access_token_xxxxx","refresh_token":"NEW_xxxxx_oauth2_refresh_token_xxxxx","token_type":"Bearer","expires_in":86400}.
Invoking 0 token response hooks.
Updating token to {'access_token': 'NEW_xxxxx_oauth2_access_token_xxxxx', 'refresh_token': 'NEW_xxxxx_oauth2_refresh_token_xxxxx', 'token_type': 'Bearer', 'expires_in': 86400, 'expires_at': 1714161325.7456222} using <bound method Client.refresh_config_from_access_token of <upwork.client.Client object at 0x7ca3a9c5ee90>>.
Requesting url https://api.upwork.com/graphql using method POST.
Supplying headers {'Content-type': 'application/json', 'Authorization': 'Bearer NEW_xxxxx_oauth2_access_token_xxxxx'} and data None
Passing through key word arguments {'json': {'query': '\n    query {\n      user {\n        id\n        nid\n        rid\n      }\n      organization {\n        id\n      }\n    }\n        '}}.
{'data': {'organization': {'id': 'xxxxxxxxxxxxxxxx'},
'user': {'id': 'xxxxxxxxxxxxxxxx',
'nid': 'xxxxxxxxxxxxxxxx',
'rid': 'xxxxxxxxxxxxxxxx'}}}

PAY ATTENTION: you need to re-save token response every time the refresh happens and pass it to the config as "token" parameter, because it contains an updated refresh_token. This way you can have an infinite session.

I can't reproduce auto-refreshing issue mentioned here and I see that the library works as expected. My recomendation, would be enabling the debug as mentioned above and investigate the issue on your following the described steps and comparing the responses/debug.

Kick28 commented 5 months ago

@mnovozhylov Thanks for the response! Tried the debugging approach and got the same logs as yours - everything looks good, and even the refresh is executed properly. But based on this:

PAY ATTENTION: you need to re-save token response every time the refresh happens and pass it to the config as "token" parameter, because it contains an updated refresh_token. This way you can have an infinite session.

I have a question: do you mean we need to re-save the token response every time the refresh happens in client.refresh_config_from_access_token method? If yes, isn't it already configured?

def refresh_config_from_access_token(self, token):
    """Callback from OAuth2 client which will refresh config with actual data"""
    self.config.token = token
mnovozhylov commented 5 months ago

@Kick28 , it is saved as a property of an object. Thus, it's true, while your script is running. But if you run the script periodically, it means that the script probably calls:

config = upwork.Config({'client_id': config.client_id, 'client_secret': config.client_secret, 'token': token})

So, make sure that you pick the previously saved and fresh token pair here.

Kick28 commented 5 months ago

@mnovozhylov Hi again! Sorry to bother you, but I got more logs from my application, maybe this can help to find the issue. The next logs are right before the application crashed. Also, I placed print inside the refresh_config_from_access_token, and there is nothing was printed out of this function in my logs.

May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: Traceback (most recent call last):
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     httplib_response = self._make_request(
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     six.raise_from(e, None)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "<string>", line 3, in raise_from
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     httplib_response = conn.getresponse()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     response.begin()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/usr/lib/python3.10/http/client.py", line 318, in begin
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     version, status, reason = self._read_status()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     raise RemoteDisconnected("Remote end closed connection without"
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: http.client.RemoteDisconnected: Remote end closed connection without response
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: During handling of the above exception, another exception occurred:
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: Traceback (most recent call last):
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/adapters.py", line 489, in send
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     resp = conn.urlopen(
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 787, in urlopen
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     retries = retries.increment(
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     raise six.reraise(type(error), error, _stacktrace)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     raise value.with_traceback(tb)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 703, in urlopen
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     httplib_response = self._make_request(
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 449, in _make_request
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     six.raise_from(e, None)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "<string>", line 3, in raise_from
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/urllib3/connectionpool.py", line 444, in _make_request
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     httplib_response = conn.getresponse()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/usr/lib/python3.10/http/client.py", line 1375, in getresponse
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     response.begin()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/usr/lib/python3.10/http/client.py", line 318, in begin
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     version, status, reason = self._read_status()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/usr/lib/python3.10/http/client.py", line 287, in _read_status
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     raise RemoteDisconnected("Remote end closed connection without"
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: During handling of the above exception, another exception occurred:
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: Traceback (most recent call last):
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 355, in <module>
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     fetcher.run()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 166, in run
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     self.run_all_queries()
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 208, in run_all_queries
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     self.process_job(job_from_list, query)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 248, in process_job
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     self.recheck_job(job_detailed, job_from_list, query, is_renewed)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 61, in recheck_job
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     job_upwork = upworkAPI.get_job_graphql(job_detailed['id'])
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/upwork_api.py", line 215, in get_job_graphql
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     output = graphql.Api(self.client).execute({"query": query})
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/upwork/routers/graphql.py", line 27, in execute
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     return self.client.post("", params)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/upwork/client.py", line 115, in post
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     return self.send_request(uri, "post", params)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/leadgen_tool/upwork/client.py", line 158, in send_request
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     r = self.__oauth.post(url, json=params, headers=headers)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/sessions.py", line 635, in post
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     return self.request("POST", url, data=data, json=json, **kwargs)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/requests_oauthlib/oauth2_session.py", line 521, in request
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     return super(OAuth2Session, self).request(
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/sessions.py", line 587, in request
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     resp = self.send(prep, **send_kwargs)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/sessions.py", line 701, in send
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     r = adapter.send(request, **kwargs)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:   File "/home/ubuntu/.local/lib/python3.10/site-packages/requests/adapters.py", line 547, in send
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]:     raise ConnectionError(err, request=request)
May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: requests.exceptions.ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))
May 12 06:17:41 ip-xxx-xx-xx-xxx systemd[1]: fetcher.service: Main process exited, code=exited, status=1/FAILURE
May 12 06:17:41 ip-xxx-xx-xx-xxx systemd[1]: fetcher.service: Failed with result 'exit-code'.
May 12 06:17:41 ip-xxx-xx-xx-xxx systemd[1]: fetcher.service: Consumed 46min 37.447s CPU time.
May 12 06:17:46 ip-xxx-xx-xx-xxx systemd[1]: fetcher.service: Scheduled restart job, restart counter is at 9.
May 12 06:17:46 ip-xxx-xx-xx-xxx systemd[1]: Stopped Fetcher.
May 12 06:17:46 ip-xxx-xx-xx-xxx systemd[1]: fetcher.service: Consumed 46min 37.447s CPU time.
May 12 06:17:46 ip-xxx-xx-xx-xxx systemd[1]: Started Fetcher.
May 12 06:18:17 ip-xxx-xx-xx-xxx python3[469878]: new job age set to=24.0
May 12 06:18:17 ip-xxx-xx-xx-xxx python3[469878]: .  query= ios "augmented reality"
May 12 06:18:17 ip-xxx-xx-xx-xxx python3[469878]: Error while fetching the API
May 12 06:18:17 ip-xxx-xx-xx-xxx python3[469878]: (invalid_grant) Refresh token is expired or revoked or not registered
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]: Traceback (most recent call last):
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 355, in <module>
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]:     fetcher.run()
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 166, in run
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]:     self.run_all_queries()
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]:   File "/home/ubuntu/leadgen_tool/fetcher.py", line 204, in run_all_queries
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]:     result_jobs = result['data']['marketplaceJobPostings']['edges']
May 12 06:18:18 ip-xxx-xx-xx-xxx python3[469878]: KeyError: 'data'
mnovozhylov commented 5 months ago

A clear identification of a network issue: May 12 06:17:22 ip-xxx-xx-xx-xxx python3[469253]: urllib3.exceptions.ProtocolError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')). It's not related to the library itself. And unfortunately, I can't do much here.

Kick28 commented 5 months ago

Well, but what about @eliyaba, that have the same problem? Also, as for me, this connection abort happens in the moment when key should refresh.

mnovozhylov commented 5 months ago

@Kick28 , sorry for not being clear. The issue will be escalated and investigated further, but there is not much to do on the application (library) layer, when "Connection aborted" (not even a timeout error).

Kick28 commented 5 months ago

Oh, got it 😄. I misunderstood this as well. Maybe logs from eliyaba might help to resolve and investigate this issue. Anyway, thanks for your help and responses!

Kick28 commented 4 months ago

@mnovozhylov @eliyaba Hey! Just a friendly ping to check the status of the issue. Thanks in advance!

eliyaba commented 4 months ago

Hi @Kick28 I'm sorry for not responding earlier, I seemed to bypass this issue by refreshing the key using the api directly, without the python sdk (after every api call).

Here's the method I use:

        if token and token.get('refresh_token', None):
            payload = {
                'grant_type': 'refresh_token',
                'refresh_token': token['refresh_token'],
                'client_id': self.config.client_id,
                'client_secret': self.config.client_secret
            }
        elif code:
            payload = {
                'grant_type': 'authorization_code',
                'code': code,
                'redirect_uri': self.config.redirect_uri,
                'client_id': self.config.client_id
            }
        else:
            current_app.logger.error('Cannot get access token without refresh token or authorization code')
            return None

        headers={
            'Accept': 'application/json',
            'Content-Type': 'application/x-www-form-urlencoded;charset=UTF-8'
        }

        res = requests.post(
            url='https://www.upwork.com/api/v3/oauth2/token',
            data=payload,
            headers=headers
        )
        if res.status_code != 200:
            current_app.logger.error(f'Error in requesting upwork access token: {res.text}')
            return False

        token_json = res.json()

        # Api result does not include 'expires_at' key
        token = UpworkAuthToken.insert_or_update(
            token_json['access_token'],
            token_json['refresh_token'],
            token_json['token_type'],
            token_json['expires_in'],
            token_json.get('expires_at', datetime.utcnow().timestamp() + token_json['expires_in'])
        )
        db.session.commit()

        return token_json
Kick28 commented 4 months ago

@eliyaba thanks for the answer! I found another solution btw. @mnovozhylov FYI: I investigated logs more thoroughly, and found that token is actually refreshing, but for some reason the application was still crashing. Fixed it by saving new token manually in refresh_config_from_access_token(). It's been a week since I applied this fix and it works like a charm.