tenable / pyTenable

Python Library for interfacing into Tenable's platform APIs
https://pytenable.readthedocs.io
MIT License
349 stars 172 forks source link

ExploreIterator does not handle searches that are still processing #671

Closed ethansutcliffe closed 1 year ago

ethansutcliffe commented 1 year ago

Describe the bug When a long-running search job returns 202, the iterator tries to parse JSON from the empty response and crashes.

To Reproduce Steps to reproduce the behavior:

Call a v3 endpoint such as api/v3/findings/vulnerabilities/host/search using tio.v3.explore.findings.search_host with arguments that will create a long-running search job. In my environment this is

{
    "sort": [
        {
            "property": "first_observed",
            "order": "asc"
        }
    ],
    "fields": [...],
    "filter": {
        "and": [
            {
                "property": "last_seen",
                "value": [
                    "2023-01-27T00:00:00.000Z",
                    "2023-01-31T23:59:59.999Z"
                ],
                "operator": "between"
            },
            {
                "property": "state",
                "value": [
                    "RESURFACED"
                ],
                "operator": "eq"
            },
            {
                "property": "definition.vpr.score",
                "value": [
                    7
                ],
                "operator": "gt"
            }
        ]
    }
}

Once the search job triggers, the iterator requests it and tries to parse JSON which doesn't exist, resulting in the below error

Traceback (most recent call last):
  File "/opt/py37env/lib/python3.7/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/removed/main.py", line 134, in <module>
    main(args)
  File "/removed/main.py", line 101, in main
    [v for v in searcher(filter=models.create_filter('vpr', **filter_), fields=models.fields, sort=SORT)]
  File "/opt/py37env/lib/python3.7/site-packages/restfly/iterator.py", line 114, in __next__
    return self.next()  # noqa: PLE1102
  File "/opt/py37env/lib/python3.7/site-packages/restfly/iterator.py", line 140, in next
    self._get_page()
  File "/opt/py37env/lib/python3.7/site-packages/tenable/io/v3/base/iterators/explore_iterator.py", line 59, in _get_page
    self._process_response(resp)
  File "/opt/py37env/lib/python3.7/site-packages/tenable/io/v3/base/iterators/explore_iterator.py", line 70, in _process_response
    body = response.json()
  File "/opt/py37env/lib/python3.7/site-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Expecting value] : 0

Expected behavior The iterator should use the Retry-After header as specified in the docs and wait until data is available: https://developer.tenable.com/reference/io-v3-uw-search-job-details

System Information (please complete the following information):

aseemsavio commented 1 year ago

The v3 API is currently in beta. Tenable recommends that you not build production integrations with v3 endpoints since they’re subject to change during the beta period.