ERROR - Background refresh thread has negative value with remaining seconds

[maria-oleszkiewicz]

I am using ibm_cos_sdk in code. Today it hanged with such error:

socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/urllib3/connection.py", line 284, in connect
    conn = self._new_conn()
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/urllib3/connection.py", line 146, in _new_conn
    (self.host, self.timeout))
urllib3.exceptions.ConnectTimeoutError: (<urllib3.connection.VerifiedHTTPSConnection object at 0x10efe5710>, 'Connection to iam.bluemix.net timed out. (connect timeout=30)')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/urllib3/util/retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='iam.bluemix.net', port=443): Max retries exceeded with url: /oidc/token (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x10efe5710>, 'Connection to iam.bluemix.net timed out. (connect timeout=30)'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/ibm_botocore/credentials.py", line 475, in _protected_refresh
    metadata = self.auth_function()
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/ibm_botocore/credentials.py", line 377, in _default_auth_function
    headers=self._get_headers(), timeout=30)
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/requests/api.py", line 112, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/anaconda/envs/Python3.5/lib/python3.5/site-packages/requests/adapters.py", line 496, in send
    raise ConnectTimeout(e, request=request)
requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='iam.bluemix.net', port=443): Max retries exceeded with url: /oidc/token (Caused by ConnectTimeoutError(<urllib3.connection.VerifiedHTTPSConnection object at 0x10efe5710>, 'Connection to iam.bluemix.net timed out. (connect timeout=30)'))
2018-03-26 12:38:18,302 - ibm_botocore.credentials - ERROR - Background refresh thread has negative value with remaining seconds 869.979037
2018-03-26 12:38:18,302 - ibm_botocore.credentials - ERROR - Background refresh thread encountered error: sleep length must be non-negative

I think that may be problem during connection refresh caused by ibm_botocore.

Please investigate this issue.

[cpcunningham]

Hi Maria, Having looked at the log output it appears you are using an earlier version of the sdk In verson 2.0.1 (IAM Resiliency Support) the token fetching was made more robust.

Can you update to latest sdk versoion or wait a couple of days we are releasing sdk version 2.1.0 which will also contain IBM json credntials file Configurable call retries Configurable Verify SSL flag for IAM Token fetching

[seamus-mcgrath]

@maria-oleszkiewicz are you still having this issue with the latest SDK?