Search incidents pagination not working

[marcelkwaschny]

Hello together,

I'm trying to use the api client to fetch incidents with pagination from our cortex xsoar system. Therefore I'm using the below code snippet to do that:

import logging
import os

import demisto_client
from demisto_client.demisto_api import Incident, IncidentFilter, InlineResponse200
from demisto_client.demisto_api.rest import ApiException

api_key = os.getenv("DEMISTO_API_KEY")
base_url = os.getenv("DEMISTO_BASE_URL")
api_instance = demisto_client.configure(base_url=base_url, api_key=api_key, debug=True)

incidents: list[Incident] = []
total_incidents: int = 1

incidents_filter = IncidentFilter(
    query="occurred:>=\"2023-07-01T00:00:00\" occurred:<\"2023-08-01T00:00:00\"",
    size=100,
    page=0
)
incidents_filter.sort = [{"asc": True, "field": "occurred"}]

while total_incidents > len(incidents):
    try:
        query_filter = demisto_client.demisto_api.SearchIncidentsData(filter=incidents_filter)
        api_response: InlineResponse200 = api_instance.search_incidents(query_filter)
    except ApiException as error:
        logging.error(str(error))
        break

    if not api_response.data:
        break

    total_incidents = api_response.total
    incidents.extend(api_response.data)
    incidents_filter.page += 1

print(incidents)

The problem is that in every iteration the same 100 incidents are returned although I increment the page. Therefore I think maybe the pagination is not working as I have guessed or this is a bug. Can someone maybe help me with that?

[amshamah419]

Can you attach the results with the debug arg? Feel free to email the to me if they contain sensitive info instead of posting them here. ashamah@paloaltonetworks.com

[DeanArbel]

This ticket is managed internally at https://jira-hq.paloaltonetworks.local/browse/CIAC-8350. Any updates will also be updated here. @marcelkwaschny Please note we'll try to reproduce this issue, but it'll help us if you get to do as @amshamah419 suggested.

[marcelkwaschny]

Sorry, I tried to generate results with dummy tickets in our development system because unfortunatly I'm not allowed to send the results from our productive system. As I tried to generate these dummy results in our development system I figured out why the pagination isn't working with our productive system.

In the above code I'm using this query:

query="occurred:>=\"2023-07-01T00:00:00\" occurred:<\"2023-08-01T00:00:00\""

But this was not the full query that I've used. As there are multiple tenants in our productive system I used the following query:

query="account=\"customer1\" occurred:>=\"2023-07-01T00:00:00\" occurred:<\"2023-08-01T00:00:00\""

When the account field is added to the query the pagination isn't working. But if I add the account to the base url like this:

base_url = f"{os.getenv('DEMISTO_BASE_URL')}/acc_{os.getenv('DEMISTO_ACCOUNT')}"

Then the pagination is working properly. So in my opinion this can be closed because I don't know if it's valid to add the account to the query instead of the base url.

[DeanArbel]

I'm glad to hear you were able to work it out! Good job :)