iter_services and list_services fail when a folder requires a token

[telnet23]

In [14]: server.list_services()
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-14-2664562dc6fe> in <module>
----> 1 server.list_services()

/usr/local/lib/python3.9/site-packages/restapi/common_types.py in list_services(self, filterer)
    819     def list_services(self, filterer=True):
    820         """Returns a list of all services."""
--> 821         return list(self.iter_services(filterer))
    822 
    823     def iter_services(self, token='', filterer=True):

/usr/local/lib/python3.9/site-packages/restapi/common_types.py in iter_services(self, token, filterer)
    835         for s in self.folders:
    836             new = '/'.join([self.url, s])
--> 837             resp = self.request(new)
    838             for serv in resp[SERVICES]:
    839                 full_service_url =  '/'.join([self.url, serv[NAME], serv[TYPE]])

/usr/local/lib/python3.9/site-packages/restapi/rest_utils.py in request(self, *args, **kwargs)
    987             kwargs['ret_json'] = False
    988         kwargs['client'] = self.client
--> 989         return do_request(*args, **kwargs)
    990 
    991     def refresh(self):

/usr/local/lib/python3.9/site-packages/restapi/rest_utils.py in do_request(service, params, ret_json, token, cookies, proxy, referer, client, **kwargs)
    367             except:
    368                 return r
--> 369             RequestError(_json)
    370             return munch.munchify(_json)
    371         else:

/usr/local/lib/python3.9/site-packages/restapi/rest_utils.py in __init__(self, err)
   1500     def __init__(self, err):
   1501         if 'error' in err:
-> 1502             raise RuntimeError(json.dumps(err, indent=2, ensure_ascii=False))
   1503 
   1504 class Folder(RESTEndpoint):

RuntimeError: {
  "error": {
    "code": 499,
    "message": "Token Required",
    "details": []
  }
}

There is nothing to catch an exception here: https://github.com/Bolton-and-Menk-GIS/restapi/blob/84c05624e937d68e741bc73c947c7cec7a582a1c/restapi/common_types.py#L1345-L1351

I've gotten around the issue by writing my own iter_services method that catches the exception and continues to the next folder. Perhaps an errors="ignore" keyword argument could be added to do the same?

[philnagel]

Thanks for submitting this issue!

Just to clarify - the situation as I understand is that you are working against a server that has some secured and some unsecured services. You want to ignore the token errors so that you can iterate over the unsecured services and skip the secured ones?

[telnet23]

Correct. I resolved the issue locally by defining the subclass:

from restapi import ArcServer, NAME, TYPE, SERVICES

class MyArcServer(ArcServer):
    def iter_services(self):
        self.service_cache = []

        for s in self.services:
            full_service_url = '/'.join([self.url, s[NAME], s[TYPE]])
            self.service_cache.append(full_service_url)
            yield full_service_url

        for s in self.folders:
            new = '/'.join([self.url, s])
            try:
                resp = self.request(new)
            except RuntimeError as exception:
                print(exception)
                continue
            for serv in resp[SERVICES]:
                full_service_url =  '/'.join([self.url, serv[NAME], serv[TYPE]])
                self.service_cache.append(full_service_url)
                yield full_service_url

[CalebM1987]

I think it would be best to create an explicit TokenRequired Exception and throw a warning when a folder requires a token and continue the iteration.

[philnagel]

@telnet23 we are unable to reproduce this issue against our development server. Would you be able to share the server URL and the folder that you are experiencing this issue with? If you'd rather not share publicly, let me know and I'll get you my e-mail.

[philnagel]

Since I'm not able to reproduce this, and there has not been a response from the issue creator, I will close this for now. Feel free to re-open and send the requested follow-up info. Thanks!

[michelmetran]

I have the same problem: on a server, some data is restricted and others are not, generating the token warning.

I'm testing the code below on the server:

• https://mapas.agenciapcj.org.br/arcgis/rest/services

import restapi

session = requests.Session()
client = restapi.RequestClient(session)
restapi.set_request_client(client)

url = 'https://mapas.agenciapcj.org.br/arcgis/rest/services'
ags = restapi.ArcServer(url)

for root, services in ags.walk():
     print(f'Folder: {root}')
     print(f'Services: {services}')

And it gives error:

RuntimeError: {
   "error": {
     "code": 499,
     "message": "Token Required",
     "details": []
   }
}

I would like to ignore private data, collecting only public data. Is there any way to tweak the code to ignore private data?

[philnagel]

This should be resolved now. A parameter ignore_folder_auth=True has been added to the walk() and iter_services() methods - it defaults to True, so no changes should be needed. This will issue warnings and continue the iteration for each folder that requires authentication.

By the way, I noticed that the server provided by @michelmetran is connected to an Enterprise Portal instance. This seems to behave differently from a standalone server. I was testing against our standalone server initially and was not able to reproduce this issue.