Closed HaipengSu closed 7 years ago
Could you provide more information when this happens?
Hi @nikhilk
From Datalab,
I have
notebooks = [ "notebook1.ipynb",
"notebook2.ipynb" ,
.
.
.
"notebook20.ipynb" ]
for notebook in notebooks:
%run $notebook
Never had any issues running this, but since updating to the newest version, I got the Runtime Error. Not still not sure about whether it is related.
Thx
Hi @nikhilk,
I am trying to get the error reproducible, so that it wouldn't confuse you and others.
Will let you know once I get more information.
thx @parthea for pointing out.
@HaipengSu shared a link to a notebook session which is currently in a failed state. The issue only occurs in this specific notebook session. If I open a new notebook and run the same code the issue disappears.
from datalab.bigquery import _api
import datalab
table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
api = _api.Api(datalab.context.Context.default())
api.tables_get(table._name_parts)
I'm still looking into this but at first glance it appears that the recursion is happening in oauth2client
.
RuntimeErrorTraceback (most recent call last)
<ipython-input-47-5908590773f2> in <module>()
2 table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
3 api = _api.Api(datalab.context.Context.default())
----> 4 api.tables_get(table._name_parts)
/usr/local/lib/python2.7/dist-packages/datalab/bigquery/_api.pyc in tables_get(self, table_name)
357 """
358 url = Api._ENDPOINT + (Api._TABLES_PATH % table_name)
--> 359 return datalab.utils.Http.request(url, credentials=self._credentials)
360
361 def tables_list(self, dataset_name, max_results=0, page_token=None):
/usr/local/lib/python2.7/dist-packages/datalab/utils/_http.pyc in request(url, args, data, headers, method, credentials, raw_response, stats)
132 method=method,
133 body=data,
--> 134 headers=headers)
135 if 200 <= response.status < 300:
136 if raw_response:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 logger.info('Attempting refresh to obtain '
615 'initial access_token')
--> 616 self._refresh(request_orig)
617
618 # Clone and modify the request headers to add the appropriate
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 """
128 response, content = http_request(
--> 129 META, headers={'Metadata-Flavor': 'Google'})
130 content = _from_bytes(content)
131 if response.status == http_client.OK:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
629 resp, content = request_orig(uri, method, body,
630 clean_headers(headers),
--> 631 redirections, connection_type)
632
633 # A stored token may expire between the time it is retrieved and
... last 1 frames repeated, from the frame below ...
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
629 resp, content = request_orig(uri, method, body,
630 clean_headers(headers),
--> 631 redirections, connection_type)
632
633 # A stored token may expire between the time it is retrieved and
RuntimeError: maximum recursion depth exceeded while calling a Python object
Here are the results after upgrading to oauth2client==4.0.0
Collecting oauth2client==4.0.0
Downloading oauth2client-4.0.0-py2.py3-none-any.whl (184kB)
100% |################################| 194kB 3.2MB/s
Requirement already satisfied: httplib2>=0.9.1 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: rsa>=3.1.4 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: pyasn1>=0.1.7 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Requirement already satisfied: six>=1.6.1 in /usr/local/lib/python2.7/dist-packages (from oauth2client==4.0.0)
Installing collected packages: oauth2client
Found existing installation: oauth2client 2.2.0
Uninstalling oauth2client-2.2.0:
Successfully uninstalled oauth2client-2.2.0
Successfully installed oauth2client-4.0.0
RuntimeErrorTraceback (most recent call last)
<ipython-input-50-5908590773f2> in <module>()
2 table = bq.Table('geotab-embedded:Test_Haipeng.sgs')
3 api = _api.Api(datalab.context.Context.default())
----> 4 api.tables_get(table._name_parts)
/usr/local/lib/python2.7/dist-packages/datalab/bigquery/_api.pyc in tables_get(self, table_name)
357 """
358 url = Api._ENDPOINT + (Api._TABLES_PATH % table_name)
--> 359 return datalab.utils.Http.request(url, credentials=self._credentials)
360
361 def tables_list(self, dataset_name, max_results=0, page_token=None):
/usr/local/lib/python2.7/dist-packages/datalab/utils/_http.pyc in request(url, args, data, headers, method, credentials, raw_response, stats)
132 method=method,
133 body=data,
--> 134 headers=headers)
135 if 200 <= response.status < 300:
136 if raw_response:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
614 retval = cls(
615 data['access_token'],
--> 616 data['client_id'],
617 data['client_secret'],
618 data['refresh_token'],
/usr/local/lib/python2.7/dist-packages/oauth2client/contrib/gce.pyc in _refresh(self, http_request)
127 try:
128 self._retrieve_info(http)
--> 129 self.access_token, self.token_expiry = _metadata.get_token(
130 http, service_account=self.service_account_email)
131 except http_client.HTTPException as err:
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
629
630 @property
--> 631 def access_token_expired(self):
632 """True if the credential is expired or invalid.
633
... last 1 frames repeated, from the frame below ...
/usr/local/lib/python2.7/dist-packages/oauth2client/client.pyc in new_request(uri, method, body, headers, redirections, connection_type)
629
630 @property
--> 631 def access_token_expired(self):
632 """True if the credential is expired or invalid.
633
RuntimeError: maximum recursion depth exceeded while calling a Python object
Thx @parthea for keeping track on this issue.
Here is some code that @HaipengSu and I used to create the issue locally:
import datalab.storage as gs
bucketName = "uspto-pair"
bucket = gs.Bucket(bucketName)
for i in range(0,1000):
for item in bucket.items():
pass
results in
RuntimeErrorTraceback (most recent call last)
<ipython-input-9-5daf0d8d76e0> in <module>()
3 bucket = gs.Bucket(bucketName)
4 for i in range(0,1000):
----> 5 for item in bucket.items():
6 pass
/usr/local/lib/python2.7/dist-packages/datalab/utils/_iterator.pyc in __iter__(self)
34 """Provides iterator functionality."""
35 while self._first_page or (self._page_token is not None):
---> 36 items, next_page_token = self._retriever(self._page_token, self._count)
37
38 self._page_token = next_page_token
/usr/local/lib/python2.7/dist-packages/datalab/storage/_item.pyc in _retrieve_items(self, page_token, _)
276 page_token=page_token)
277 except Exception as e:
--> 278 raise e
279
280 items = list_info.get('items', [])
RuntimeError: maximum recursion depth exceeded while calling a Python object
The issue appears to be resolved if we revert the changes made in googledatalab/pydatalab#195 .
cc @nikhilk cc @yebrahim
It looks like the source of the bug is that every time an HTTP request is issued, the call to credentials.authorize
adds another wrapper around the http
instance's request
method.
Then, when http.request(...)
gets invoked, all of those layers of wrappers have to be passed through before the final request actually gets performed.
In essence, we have a leak where the level of method wrapping grows unbounded, and eventually hits the recursive depth limit.
We can avoid this if the number of request
wrappers was fixed (e.g. at 1). That, in turn, might be doable by making sure that the request
method in the shared http instance cannot be overwritten.
I'll try making a copy of the shared instance before adding the credentials, and see if that prevents the bug.
Thank you all for looking into this.
Hi,
After updating the Datalab to the newest version, I got this Runtime Error.
RuntimeError: maximum recursion depth exceeded
Apparently, this is related to the depth of recursive calls.
For the same notebook, I didn't get the Runtime Error before, but only for the new Datalab. Is there any change related to this issue?
Thank you.
Or should I just use
import sys sys.setrecursionlimit(10000)
to increase the recursion limit?