pnp / powershell

PnP PowerShell
https://pnp.github.io/powershell
MIT License
658 stars 341 forks source link

[FEATURE] Add PageSize option to more cmdlets #4141

Open sg1888 opened 1 month ago

sg1888 commented 1 month ago

Is your feature request related to a problem? Please describe. I would like to have -PageSize as an option for the following cmdlets: Get-PnPFileInFolder, Get-PnPFolderItem, and Get-PnPFolderInFolder. Right now, PageSize is not supported.

I have a large Sharepoint library with > 4 million files. Right now I have to use Get-PnPListItem, which loads the entire site. This is incredibly slow and memory intensive. I tried using Get-PnPListItem -List "Documents" -FolderServerRelativeURL "/Sites/Url/", but it doesn't seem to work when PageSize is specified.

Describe the solution you'd like I would like to be able to specify PageSize for the aforementioned cmdlets so that they can be used on large folders and subfolders.

Describe alternatives you've considered I've considered buying 256GB of RAM and carving out 2-3 weeks to let the script work, but there has to be a better way.

jackpoz commented 1 month ago

To avoid loading the whole list/library in memory, use the -ScriptBlock parameter of Get-PnPListItem

The script block to run after every page request.

It accepts script block like { Param($items) } where $items contains as many items as specified in -PageSize . I was able to process 1.5 M files in a library this way without any RAM issues.

sg1888 commented 1 month ago

To avoid loading the whole list/library in memory, use the -ScriptBlock parameter of Get-PnPListItem

The script block to run after every page request.

It accepts script block like { Param($items) } where $items contains as many items as specified in -PageSize . I was able to process 1.5 M files in a library this way without any RAM issues.

Thanks. I didn't think of using ScriptBlock - I will try that. I do think that adding PageSize to more cmdlets would be helpful as scriptblocks are a bit kludgy, especially if you need to reference other variables.