Large page_job_number return error when list jobs

Describe the bug Large page_job_number return error

To Reproduce Steps to reproduce the behavior:

%reload_ext adlmagics
%adl login --tenant microsoft.onmicrosoft.com
adl_accounts = %adl listaccounts --page_index 0 --page_account_number 30
adl_accounts.head(10)
jobs = %adl listjobs --account devtooltelemetryadla --page_index 0 --page_job_number 100000000000

Expected behavior Give out warning or limit

Screenshots

Desktop (please complete the following information): OS: Win10 Browser: chrome Jupyter Version: Python 3.6.2, Jupyter 4.4.0

Additional context

Listing azure data lake jobs under account 'devtooltelemetryadla'...

MaxRetryError Traceback (most recent call last) c:\python36\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 439 retries=self.max_retries, --> 440 timeout=timeout 441 )

c:\python36\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, response_kw) 731 release_conn=release_conn, --> 732 body_pos=body_pos, response_kw) 733

c:\python36\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw) 711 try: --> 712 retries = retries.increment(method, url, response=response, _pool=self) 713 except MaxRetryError:

c:\python36\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace) 387 if new_retry.is_exhausted(): --> 388 raise MaxRetryError(_pool, url, error or ResponseError(cause)) 389

MaxRetryError: HTTPSConnectionPool(host='devtooltelemetryadla.azuredatalakeanalytics.net', port=443): Max retries exceeded with url: /jobs?$top=100000000000&api-version=2017-09-01-preview (Caused by ResponseError('too many 500 error responses',))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last) c:\python36\lib\site-packages\msrest\service_client.py in send(self, request, headers, content, config) 256 request.url, --> 257 kwargs) 258 return response

c:\python36\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 507 send_kwargs.update(settings) --> 508 resp = self.send(prep, **send_kwargs) 509

c:\python36\lib\site-packages\requests\sessions.py in send(self, request, kwargs) 617 # Send the request --> 618 r = adapter.send(request, kwargs) 619

c:\python36\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 498 if isinstance(e.reason, ResponseError): --> 499 raise RetryError(e, request=request) 500

RetryError: HTTPSConnectionPool(host='devtooltelemetryadla.azuredatalakeanalytics.net', port=443): Max retries exceeded with url: /jobs?$top=100000000000&api-version=2017-09-01-preview (Caused by ResponseError('too many 500 error responses',))

During handling of the above exception, another exception occurred:

ClientRequestError Traceback (most recent call last)

in () ----> 1 jobs = get_ipython().run_line_magic('adl', 'listjobs --account devtooltelemetryadla --page_index 0 --page_job_number 100000000000') c:\python36\lib\site-packages\IPython\core\interactiveshell.py in run_line_magic(self, magic_name, line, _stack_depth) 2129 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals 2130 with self.builtin_trap: -> 2131 result = fn(*args,**kwargs) 2132 return result 2133 in adl(self, line, cell) c:\python36\lib\site-packages\IPython\core\magic.py in (f, *a, **k) 185 # but it's overkill for just that one bit of state. 186 def magic_deco(arg): --> 187 call = lambda f, *a, **k: f(*a, **k) 188 189 if callable(arg): c:\python36\lib\site-packages\adlmagics\adlmagics_main.py in adl(self, line, cell) 65 66 magic = self.__magics[cmd] ---> 67 return magic.execute(arg_string, cell) 68 69 def __register_azure_magic(self, azure_magic_class): c:\python36\lib\site-packages\adlmagics\magics\adla\adla_jobs_listing_magic.py in execute(self, arg_string, content_string) 22 self._write_line("Listing azure data lake jobs under account '%s'..." % (args.account)) 23 ---> 24 jobs = self._adla_service.retrieve_jobs(args.account, job_filter, args.page_index, args.page_job_number) 25 26 html = "" c:\python36\lib\site-packages\adlmagics\services\adla_service_sdk_impl.py in retrieve_jobs(self, account, filter, page_index, page_job_number) 66 jobs = job_client.job.list(account, filter = filter, top = page_job_number, skip = skip) 67 ---> 68 return [AdlaJob(job.job_id, job.name, job.type.name, job.submitter, job.degree_of_parallelism, job.priority, job.submit_time, job.start_time, job.end_time, job.state.name, job.result.name) for job in jobs] 69 70 @property c:\python36\lib\site-packages\adlmagics\services\adla_service_sdk_impl.py in (.0) 66 jobs = job_client.job.list(account, filter = filter, top = page_job_number, skip = skip) 67 ---> 68 return [AdlaJob(job.job_id, job.name, job.type.name, job.submitter, job.degree_of_parallelism, job.priority, job.submit_time, job.start_time, job.end_time, job.state.name, job.result.name) for job in jobs] 69 70 @property c:\python36\lib\site-packages\msrest\paging.py in __next__(self) 119 return response 120 else: --> 121 self.advance_page() 122 return self.__next__() 123 c:\python36\lib\site-packages\msrest\paging.py in advance_page(self) 105 raise StopIteration("End of paging") 106 self._current_page_iter_index = 0 --> 107 self._response = self._get_next(self.next_link) 108 self._derserializer(self, self._response) 109 return self.current_page c:\python36\lib\site-packages\azure\mgmt\datalake\analytics\job\operations\job_operations.py in internal_paging(next_link, raw) 123 request = self._client.get(url, query_parameters) 124 response = self._client.send( --> 125 request, header_parameters, stream=False, **operation_config) 126 127 if response.status_code not in [200]: c:\python36\lib\site-packages\msrest\service_client.py in send(self, request, headers, content, **config) 288 oauth2.rfc6749.errors.OAuth2Error) as err: 289 msg = "Error occurred in request." --> 290 raise_with_traceback(ClientRequestError, msg, err) 291 finally: 292 self._close_local_session_if_necessary(response, session, kwargs['stream']) c:\python36\lib\site-packages\msrest\exceptions.py in raise_with_traceback(exception, message, *args, **kwargs) 43 error = exception(exc_msg, *args, **kwargs) 44 try: ---> 45 raise error.with_traceback(exc_traceback) 46 except AttributeError: 47 error.__traceback__ = exc_traceback c:\python36\lib\site-packages\msrest\service_client.py in send(self, request, headers, content, **config) 255 request.method, 256 request.url, --> 257 **kwargs) 258 return response 259 c:\python36\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json) 506 } 507 send_kwargs.update(settings) --> 508 resp = self.send(prep, **send_kwargs) 509 510 return resp c:\python36\lib\site-packages\requests\sessions.py in send(self, request, **kwargs) 616 617 # Send the request --> 618 r = adapter.send(request, **kwargs) 619 620 # Total elapsed time of the request (approximately) c:\python36\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies) 497 498 if isinstance(e.reason, ResponseError): --> 499 raise RetryError(e, request=request) 500 501 if isinstance(e.reason, _ProxyError): ClientRequestError: Error occurred in request., RetryError: HTTPSConnectionPool(host='devtooltelemetryadla.azuredatalakeanalytics.net', port=443): Max retries exceeded with url: /jobs?$top=100000000000&api-version=2017-09-01-preview (Caused by ResponseError('too many 500 error responses',))

jingyanjingyan commented 6 years ago

Repro at 07-25-2018

Azure / Azure-Data-Service-Notebook

Large page_job_number return error when list jobs #20

Listing azure data lake jobs under account 'devtooltelemetryadla'...