CSV and JSON exports truncate at 250 records

[martinhannigan]

Describe the bug JSON and CSV are truncated at 250 records

To Reproduce

1. Go to search
2. Select facilities
3. Select a scope of region=north america
4. Search
5. Press either export function
6. Inspect the output file

Expected behavior All relevant records export as expected

Who is affected by the problem?

Users of PDB

What is the impact?

Poor data quality

Are there security concerns?

No

Are there privacy concerns?

No

What are the proposed actions?

Fix it.

What is the proposed priority?

Medium, impacts user ability to perform PDB related functions quickly and easily in and out of secure and insecure environments (the API is not relevant)

[arnoldnipper]

That's a feature, not a bug, try a search, e.g. https://www.peeringdb.com/advanced_search?info_scope__in=North%20America&reftag=net, then scroll down to the end and you read

...More than 250 entries matched your query, refine your request...

[martinhannigan]

I saw that. I wasn't certain if that was an error in itself since why would anyone accept less than what they searched for considering it's an export?

On Sat, Mar 9, 2024 at 4:27 PM Arnold Nipper @.***> wrote:

That's a feature, not a bug, try a search, e.g. https://www.peeringdb.com/advanced_search?info_scope__in=North%20America&reftag=net, then scroll down to the end and you read

...More than 250 entries matched your query, refine your request...

— Reply to this email directly, view it on GitHub https://github.com/peeringdb/peeringdb/issues/1563#issuecomment-1986984974, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFA2YQT2IEKYQ2G474E27ETYXN5CZAVCNFSM6AAAAABEOL3LEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBWHE4DIOJXGQ . You are receiving this because you authored the thread.Message ID: @.***>

[arnoldnipper]

Minimise the load and prevent from abuse/dos

[martinhannigan]

Wouldn't it be more secure/smarter/better to prevent abuse to instead require a login to use such functions?

On Sun, Mar 10, 2024 at 12:59 PM Arnold Nipper @.***> wrote:

Minimise the load and prevent from abuse/dos

— Reply to this email directly, view it on GitHub https://github.com/peeringdb/peeringdb/issues/1563#issuecomment-1987294261, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFA2YQUGZXNP43USDZOO34DYXSGNXAVCNFSM6AAAAABEOL3LEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBXGI4TIMRWGE . You are receiving this because you authored the thread.Message ID: @.***>

[arnoldnipper]

If you really search for something, you don't need 250+ results, IMO. YMMV

[leovegoda]

The least we could do is update that message to let people know that they can get all fac objects in one go from the .KMZ download

[martinhannigan]

CSV, JSON vs. KMZ isn't really close in terms of utility so I'm not sure what we would gain by adding any sort of banner.

On Mon, Mar 11, 2024 at 10:23 AM Leo Vegoda @.***> wrote:

The least we could do is update that message to let people know that they can get all fac objects in one go from the .KMZ download

— Reply to this email directly, view it on GitHub https://github.com/peeringdb/peeringdb/issues/1563#issuecomment-1988564153, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFA2YQXMEWJPTAKXIH45LH3YXXEADAVCNFSM6AAAAABEOL3LEKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOBYGU3DIMJVGM . You are receiving this because you authored the thread.Message ID: @.***>

[martinhannigan]

+1

[martinhannigan]

If you really search for something, you don't need 250+ results, IMO. YMMV

Sorry Arnold, but can you explain what you mean?

Tks!

[jackcarrozzo]

PC call: same api limits apply to JSON and CSV; 250 limit is for performance reasons

[jackcarrozzo]

Periodic CSV dumps of various types might sense

[jackcarrozzo]

API limits can be a little more intelligent too

[jackcarrozzo]

PC call: we would like to do Something to improve this- pagination + performance improvements to api is good, but need to improve json + csv export too

[leovegoda]

Changed status: I'll schedule some work on how we could implement this in a safe way

[leovegoda]

Also, see: https://github.com/peeringdb/peeringdb/issues/1604