search

pnp / PnP-PowerShell

SharePoint PnP PowerShell CmdLets
https://pnp.github.io/powershell
Other
988 stars 662 forks source link

Get-PnPRecycleBinItem exceeds the list view threshold #2315

Open DanielSanIT opened 5 years ago

DanielSanIT commented 5 years ago

Reporting an Issue or Missing Feature

"Get-PnPRecycleBinItem" command on the recyclebin with a lot of items fails with "the list view threshold" error.

I have tried to filter results by following commands such as "Get-PnPRecycleBinItem | select -last 10" and "Get-PnPRecycleBinItems | ? DeletedDate -gt $today" however, these commands did not help. I got the same error.

This worked for recycle bin with more than 40k items some time ago.

Is there any workaround solution? Is it possible to fix this trouble in PnP-PowerShell module?

Expected behavior

Restored files/files list

Actual behavior

Error message: "Get-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator. At line:1 char:2

Steps to reproduce behavior

Connect-PnPOnline -Url https://... (Get-PnPRecycleBinItem).count

Which version of the PnP-PowerShell Cmdlets are you using?

What is the version of the Cmdlet module you are running?

3.1.1809.0

How did you install the PnP-PowerShell Cmdlets?

KoenZomers commented 4 years ago

I don't think anyone is trying to block this release. I'm just struggling with why it doesn't run into thresholds on my environment which makes it hard for me to fix what you guys are running into. Can't fix what's not broken. Can any of you by chance file a PR where it is fixed in the way you envision it needs fixing?

bierlyt commented 4 years ago

I am running into the exact same issue with Restore-PnPRecycleBinItem, just like PrzemyslawKlys.

We have a SharePoint Online customer with over 107,000 items in the first stage recycle bin, and we need to restore over 24,000 of them that were deleted yesterday. I cannot run Get-PnPRecycleBinItem without the -RowLimit parameter or I get the error:

Get-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.

I can use the -RowLimit parameter to successfully perform the Get command. Here's some simple commands that illustrate this.

$restoreSet = Get-PnPRecycleBinItem -RowLimit 150000 $restoreSet.Count ## 107540 results $restoreDavid = $restoreSet | ?{ $_.DeletedByEmail -eq "Davidm@<domain>.com" -and $_.DeletedDate -gt "7/14/2020" }
$restoreDavid.count ## 24325 results

However, I cannot find any way to get the Restore-PnPRecycleBinItem to work. Either I get the "list view threshold" error, or I get a pipeline input error if I try to use -RowLimit at the same time. Here's 3 attempts, 2 with the pipeline and one passing in Identity.

$restoreDavid | Restore-PnPRecycleBinItem -Force ##Results in "list view threshold" error

$restoreDavid | Restore-PnPRecycleBinItem -Force -RowLimit 500 ##Results in a "pipeline input" error

$foreach($r in $restoreDavid){ Restore-PnPRecycleBinItem -Identity $r.Id.Guid -Force} ## Results in "list view threshold" error

I don't understand why the Restore-PnPRecycleBinItem cmdlet runs into the "list view threshold" error when simply trying to restore a single file that was passed in by Identity. But it appears that if Get-PnPRecycleBinItem without -RowLimit gets an error, then Restore-PnPRecycleBinItem without -RowLimit does too, and if there's enough items in the recycle bin, there's no way to restore a specific list of files passed in from the Get-PnPRecycleBinItem cmdlet.

Is there a fix since the discussion stopped 11 days ago?

bierlyt commented 4 years ago

I found an alternative way to restore the recycle bin items in a case like I'm running into where there are over 100K items in the recycle bin. The author in this blog post figured out the API calls that SharePoint makes when restoring a file from the web and found a way to replicate it using the Invoke-PnPSPRestMethod cmdlet in the SharePointPnPPowerShellOnline module.

https://cann0nf0dder.wordpress.com/2020/02/13/viewing-restoring-and-removing-items-from-the-sharepoint-recycle-bin-the-attempted-operation-is-prohibited-because-it-exceeds-the-list-view-threshold-enforced-by-the-administrator/

image

I was able to pass in the Id's from my list of files to restore that I had retrieved using the Get-PnPRecycleBinItem' into theRestore-RecycleBinItem` function in the screenshot above, and it restored the files successfully.

pmatthews05 commented 4 years ago

@bierlyt glad you found my blog post. You can also make a single call and have the body contain a collection of Ids. The blog post is just an example, but in the production environment where I'm running a similar piece of code to this, I am passing in 200 Ids at a time.

So the body would just have the collection of Deleted Object Id's. Example of 5 shown below.

$body = "{""ids"":[""Id1"",""Id2"",""Id3"",""Id4"",""Id5""]}"
bierlyt commented 4 years ago

pmatthews05, thanks for the information! You're a life-saver. I didn't even realize the blog post was linked from this thread or was written by you. I found it after hours of Googling and trying other CSOM attempts that ran into the same error message the Restore-PnPRecycleBinItem cmdlet runs into. Is it a lot faster passing 200 Ids instead of passing 1 Id at a time? It seems to take 2-3 seconds per file to restore 1 Id at a time. I have two PowerShell processes running restores from my sorted list, one from the top down and the other from the bottom up, but it will still probably take about 10 hours to finish. If it's faster, do you have a bit of code that demonstrates passing in an array of ID's and processing 200 Ids at a time? I could write some code to do it, but if you have a snippet available that'd be awesome. Thanks!

pmatthews05 commented 4 years ago

@bierlyt It is a lot quicker when batching.

I've put together the following Gist. https://gist.github.com/pmatthews05/83e3fb4471a1e82187d4f8d648068190 Please note, I haven't tested it, (although it looks ok) as I've taken bits of production code and my Restore-RecycleBinItems.ps1 code. The original production code I have was to clear the recycle bin, leaving x amount of days left in the recycle bin.

bierlyt commented 4 years ago

Wow, it's a LOT faster when batching 200 at a time. Here's the code I whipped up on the fly to do it.

$siteUrl = "https://<tenant>.sharepoint.com"
Connect-PnPOnline -Url $siteUrl
$restoreSet = Get-PnPRecycleBinItem -RowLimit 150000
$restoreSet.Count ## 107540 results
$restoreDavid = $restoreSet | ?{ $_.DeletedByEmail -eq "Davidm@<domain>.com" -and $_.DeletedDate -gt "7/14/2020" }
$restoreDavid.count ## 24325 results
# I restored all matching deleted folders first, and then restored files as shown below
$restoreDavidFileSorted = $restoreDavid | ?{$_.ItemType -eq "File"} | sort DirName, LeafName
$restoreDavidFileSorted.Count  ## 20761 results

# Batch restore up to 200 at a time
$restoreList = $restoreDavidFileSorted | select Id, ItemType, LeafName, DirName
$apiCall = $siteUrl + "/_api/site/RecycleBin/RestoreByIds"
$restoreListCount = $restoreList.count
$start = 0
$leftToProcess = $restoreListCount - $start
while($leftToProcess -gt 0){
    If($leftToProcess -lt 200){$numToProcess = $leftToProcess} Else {$numToProcess = 200}
    Write-Host -ForegroundColor Yellow "Building statement to restore the following $numToProcess files"
    $body = "{""ids"":["
    for($i=0; $i -lt $numToProcess; $i++){
        $cur = $start + $i
        $curItem = $restoreList[$cur]
        $Id = $curItem.Id
        Write-Host -ForegroundColor Green "Adding ", $curItem.ItemType, ": ", $curItem.DirName, "//", $curItem.LeafName
        $body += """" + $Id + """"
        If($i -ne $numToProcess - 1){ $body += "," }
    }
    $body += "]}"
    Write-Host -ForegroundColor Yellow $body
    Write-Host -ForegroundColor Yellow "Performing API Call to Restore items from RecycleBin..."
    try {
        Invoke-PnPSPRestMethod -Method Post -Url $apiCall -Content $body | Out-Null
    }
    catch {
        Write-Error "Unable to Restore"     
    }
    $start += 200
    $leftToProcess = $restoreListCount - $start
}
PrzemyslawKlys commented 4 years ago

You know you can convert hashtable to JSON and skip this ugly bodybuilding right?

$Ids = @(youridsgohere)
$MyCode = @{
    ids = @($Ids)
} 
$Body = $MyCode | ConvertTo-Json -Depth 5
Conda86 commented 4 years ago

Any fix to this yet? Still struggeling (with the clear-pnpR) here.

PS C:\WINDOWS\system32> Get-PnPRecycleBinItem -RowLimit 10 | where {$_.DirName -like "l157"} | Clear-PnPRecycleBinItem

Confirm Permanently delete file '170102_52alene_UTM32_NN2000_20191106' from the recycle bin? [Y] Yes [N] No [S] Suspend [?] Help (default is "Y"): Y Clear-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator. At line:1 char:74