The limit key isn't reducing the number of results

[MrPaulAR]

pynetbox version

v7.3.3

NetBox version

3.7.4

Python version

3.11

Steps to Reproduce

NETBOX_TOKEN = "xxx"
NETBOX_URL = "https://demo.netbox.dev"
nb_demo = pynetbox.api(url=NETBOX_URL, token=NETBOX_TOKEN)
changes = nb_demo.extras.object_changes.filter(limit=1)
len(changes)

1850

Expected Behavior

The output should be 1 instead of 1850

Observed Behavior

The result is the total number of object available, equivalent to .all().

[markkuleinio]

Note that the limit parameter is not a feature of pynetbox but NetBox:

https://docs.netbox.dev/en/stable/integrations/rest-api/

The default page is determined by the PAGINATE_COUNT configuration parameter, which defaults to 50. However, this can be overridden per request by specifying the desired offset and limit query parameters.

But, note that the .filter() method does not return a list of all items: it returns a generator object that only downloads the data from NetBox API when you consume it. Getting the length of the object returns the count that was returned by NetBox in the first response, it does not consume the generator at all.

Thus, if you are determined that you only care about the first item, get it from the generator and don't get anything else:

>>> changes = nb.extras.object_changes.filter(limit=1)
>>> first_item = next(changes)
>>> first_item
c03fc9c5-cc80-4e27-b9fc-d9fbfb9d3d24
>>>

This only sends out one API call:

[18/Mar/2024:19:11:58 +0200] "GET /api/extras/object-changes/?limit=1 HTTP/1.1" 200 1252 "-" "python-requests/2.31.0"

And calling len(changes) does not send any API call after that at all (because the object already knows the length of the data from the first API call response).

[markkuleinio]

To be clear: adding the limit parameter does not change the amount of total data you will get, if you consume all the generator. But it will change the amount of data that is retrieved in one NetBox API call in the background. Thus it can be used for optimizing the NetBox resource usage if you only care about small amount of data.

[MrPaulAR]

Thanks for the clarification, my expectation was that since the REST API returned only the item count referenced in the limit value this could be used as an additional filter to say, grab the latest X results. The RecordSet returned from the query below did work in prior netbox versions, likely as recent as 3.6.x.

    cables = nb.dcim.cables.filter(
        ordering="-created", user=username, limit=cable_count
    )

Unfortunately, this cable endpoint has been changed in the current version of netbox (3.7.4) so I can no longer filter on the user keyword. This is what I'm trying to refactor and it's not worth spinning up an older netbox instance to prove myself right or wrong.

Sound like the proper course of action is to iterate over the RecordSet and use a counter to break out of the iterator.

[markkuleinio]

Just an idea, instead of using a manual counter, you can use itertools.islice() to get specific number of items:

>>> changes = nb.extras.object_changes.filter()
>>> import itertools
>>> list(itertools.islice(changes, 4))
[c03fc9c5-cc80-4e27-b9fc-d9fbfb9d3d24, 488e257a-a236-4c58-a98e-15b1200da753,
bf38573e-1612-4e27-b25d-063f0ca8d7a1, 09de9b1e-fd40-4682-b5f4-7ee170e4006b]
>>>

[MrPaulAR]

Thanks for your feedback here. I'm gonna close this out as I obviously had/have a bad understanding.