How to get Large list data using getAll() with single API call?

[bhoomesh-spe]

What version of PnPjs library you are using

3.x

Minor Version Number

3.15.0

Target environment

SharePoint Framework

Additional environment details

SharePoint Online, React

Question/Request

Hello Everyone,

I have created a SharePoint list that contains 100000+ items. For that, I have implemented the below code

import { spfi } from "@pnp/sp";
import "@pnp/sp/webs";
import "@pnp/sp/lists";
import "@pnp/sp/items";
import "@pnp/sp/items/get-all";

const sp = spfi(...);

const allItems: any[] = await sp.web.lists.getByTitle("BigList").items.select("*").getAll(5000);
console.log(allItems.length);

But when I load the page it takes around 3 minutes to load all data from the SharePoint list. I have checked the above call in the network under the browser console and it gets 5000 items per call and the same API calls multiple times. So, it might take time to get data from the SharePoint list.

Is there any way to call all items with a single API call? Can anyone help me with the same?

Thanks

[juliemturner]

No, this is a longstanding limitation of the REST apis for accessing this type of data. Whenever I have a large amount of list data I usually try and take a step back and decide what I'm trying to do with that data and if I need it all what form do I need it in. Can I filter it before the response is returned for instance? Since no person is really going to consume all 100000+ (in your case), it feels like you must be only looking for a subset of the information or to manipulate or work with it in some way. I suggest that this is probably not best done client-side, and is there an opportunity for optimizing it with "server" based code first?

[juliemturner]

I'm going to close this issue as answered. If you have further issues, please feel free to create a new issue and reference this one.

[github-actions[bot]]

This issue is locked for inactivity or age. If you have a related issue please open a new issue and reference this one. Closed issues are not tracked.