Open DanielSanIT opened 5 years ago
Thank you for reporting this issue. We will be triaging your incoming issue as soon as possible.
Is this being looked at yet? It's been a month and a half and this module is simply 100% broken and can no longer even so much as get a COUNT of recycle bin items etc.
Hello @CollinChaffin , as part of the January release I pushed an update to this cmdlet that should resolve the issue you were facing. I, too, was facing this problem on my own tenant. Unfortunately, if the recycle bin is too full, the service begins to misbehave a bit, so there is no simple fix to retrieve all items.
That said, i added a parameter to allow you to limit the amount of returned results, which you should be able to use to retrieve items without the service timing out for you. Just use the -RowLimit parameter for this function. I recommend 5000 to 10000 items to return, that seems to be the sweet spot - depending on how the tenant is doing you may be able to retrieve more (but, again, performance will vary tenant by tenant).
Hope it helps!
Hi @MrDoNotBreak , Thanks for your fix.
Two questions for you.
Do you know when we will get the update to see this fix?
As I understand, we just will get top 5000-10000 results. Is there any pagination, to get all result by 5-10k elements on page?
@MrDoNotBreak this problem also seems to apply to Restore-PnPRecycleBinItem, for which there is no limiting parameter. For example, if I try:
Get-PnPRecycleBinItem -SecondStage -RowLimit 4500 | ? {($_.DeletedByEmail -eq 'user@domain.com')} | Restore-PnPRecycledBinItem -Force
Then errors are thrown: Restore-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.
Hello @CollinChaffin , as part of the January release I pushed an update to this cmdlet that should resolve the issue you were facing. I, too, was facing this problem on my own tenant. Unfortunately, if the recycle bin is too full, the service begins to misbehave a bit, so there is no simple fix to retrieve all items.
@MrDoNotBreak thank you for putting the restriction on the get-pnprecyclebinitem. Unfortunately I am having the same issue as @6079-Winston-Smith where a client has accidentally deleted a massive amount of files and the restore function fails because of the 5000 limit, regardless of how few files are piped to it. Even if I have to restore items 5000 at a time it would be a big help to have -RowLimit on the restore function. The only other alternative is to do a full library restore and lose the day's work.
@Brendenawit I have come across the same limitation recently. It seems to be that the Restore-PnpRecycleBinItem does a call to get-pnpRecycleBinItem in the code, but this code doesn't have a Row Limit on it. Until someone at PNP fixes this issue, (not meaning to promote myself, but it should help) I wrote a blog on this yesterday., with code to restore.
Clear-PnPRecycleBinItem has the same problem, so I've added support for the RowLimit parameter to it. Would this be useful to make a PR for?
All recycle bin cmdlets should have the RowLimit parameter in the July 2020 release.
And when can we expect that release? I am just trying to restore files and hitting this error.
@PrzemyslawKlys, here's a custom build which includes this fix already, if you want to test it out already:
SharePointPnPPowerShellOnline.zip
Official July 2020 release will be somewhere between next Friday and the end of next week. Exact date yet to be defined.
July 13th will be the release date
Thank you!
Perhaps if I have to ask this question I shouldn't be testing your custom build, but if I've installed the module from the gallery, how do I incorporate the changes from your zip file above? I've tried copying the folder over to C:\Program Files\WindowsPowerShell\Modules but I still don't see the -RowLimit option on Restore-PnPRecycleBinItem. TIA
Questions are always welcome @Vindicator007 . As I think more people will have the same question, I have just written a wiki page outlining how you can make use of this or other custom builds which we share in the issue topics:
https://github.com/pnp/PnP-PowerShell/wiki/Using-custom-PnP-PowerShell-builds
@KoenZomers Thanks for the wiki, I was off by one directory, but after uninstall and following your post it still doesn't seem to be working for me. It acts like an untrusted script whenever I attempt to use tab completion, prompting me to allow it to run even though I tried setting execution policy to unrestricted. Maybe it's me, but I thought I would let you know.
@Vindicator007 You may need to right-click on the file in explorer, properties, and check if you need to unblock them. It's often like that when downloaded from internet directly.
@KoenZomers any reason why pipeline doesn't work
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Files = $Stage | Where-Object { $_.DirName -like "$Test*"} #| Out-HtmlView -ScrollX -DisablePaging
#foreach ($File in $Files) {
# $File | Restore-PnPRecycleBinItem -RowLimit 150000
#}
$Files | Restore-PnPRecycleBinItem -RowLimit 150000
Restore-PnPRecycleBinItem : The input object cannot be bound to any parameters for the command either because the command does not take pipeline input or the input and its properties do not match any of the parameters that take pipeline input.
At line:1 char:10
+ $Files | Restore-PnPRecycleBinItem -RowLimit 150000
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : InvalidArgument: (Microsoft.Share....RecycleBinItem:PSObject) [Restore-PnPRecycleBinItem], ParameterBindingException
+ FullyQualifiedErrorId : InputObjectNotBound,SharePointPnP.PowerShell.Commands.RecycleBin.RestoreRecycleBinItem
It doesn't say it accepts pipeline like in other cmdlets, but examples show it's normally used that way.
get-help Restore-PnPRecycleBinItem -Full
NAME
Restore-PnPRecycleBinItem
SYNOPSIS
Restores the provided recycle bin item to its original location
SYNTAX
Restore-PnPRecycleBinItem [-Force <SwitchParameter>] [-RowLimit <Int32>] [-Connection <PnPConnection>] [<CommonParameters>]
Restore-PnPRecycleBinItem -Identity <RecycleBinItemPipeBind> [-Force <SwitchParameter>] [-Connection <PnPConnection>] [<CommonParameters>]
DESCRIPTION
PARAMETERS
-Force [<SwitchParameter>]
If provided, no confirmation will be asked to restore the recycle bin item
Required? false
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?
-Identity <RecycleBinItemPipeBind>
Id of the recycle bin item or the recycle bin item object itself to restore
Required? true
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?
-RowLimit [<Int32>]
Limits restoration to specified number of items
Required? false
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?
-Connection [<PnPConnection>]
Optional connection to be used by the cmdlet. Retrieve the value for this parameter by either specifying -ReturnConnection on Connect-PnPOnline or by executing Get-PnPConnection.
Required? false
Position? named
Default value
Accept pipeline input?
Accept wildcard characters?
<CommonParameters>
This cmdlet supports the common parameters: Verbose, Debug,
ErrorAction, ErrorVariable, WarningAction, WarningVariable,
OutBuffer, PipelineVariable, and OutVariable. For more information, see
about_CommonParameters (https:/go.microsoft.com/fwlink/?LinkID=113216).
INPUTS
OUTPUTS
------------------EXAMPLE 1---------------------
PS:> Restore-PnpRecycleBinItem -Identity 72e4d749-d750-4989-b727-523d6726e442
Restores the recycle bin item with Id 72e4d749-d750-4989-b727-523d6726e442 to its original location
------------------EXAMPLE 2---------------------
PS:> Get-PnPRecycleBinItem | ? -Property LeafName -like "*.docx" | Restore-PnpRecycleBinItem
Restores all the items in the first and second stage recycle bins to their original location of which the filename ends with the .docx extension
------------------EXAMPLE 3---------------------
PS:> Restore-PnPRecycleBinItem -All -RowLimit 10000
Permanently restores up to 10,000 items in the recycle bin
RELATED LINKS
SharePoint Developer Patterns and Practices: http://aka.ms/sppnp
Did something change?
Ok, so the reason for the above is, RowLimit was added, but only if you don't use Identity. This makes no sense :-(
@PrzemyslawKlys Unblock did the trick, thanks. Now I'm getting the same thing you are, that it can't take pipeline input.
Good feedback guys. I've updated the wiki with the instruction to unblock the files. Regarding it not working as expected, I guess that shows the value of sharing custom builds before it gets released into dev :)
I'm going to write a simple script to upload 5050 items to a doclib and delete them so I can try to reproduce the scenario.
I am now trying to restore items between 22939-24209. I am now scrolling in web interface every 200 objects. I am slowly dying inside :-)
I can tell now that you simply need to update parameter sets. You have 2 parameter sets, where Identity is in a different parameter sets then RowLimit. It should be as simple as fix that there is a single parameter set name, not two.
Restore-PnPRecycleBinItem [-Force <SwitchParameter>] [-RowLimit <Int32>] [-Connection <PnPConnection>] [<CommonParameters>]
Restore-PnPRecycleBinItem -Identity <RecycleBinItemPipeBind> [-Force <SwitchParameter>] [-Connection <PnPConnection>] [<CommonParameters>]
@KoenZomers Thanks for your efforts. Are there any updates forthcoming, before I click the 1 week restore button in the portal? 🤦♂️
Haven't made progess on this one yet. I let a script run to upload 5050 files and then deleted them, but forgot to include the -Recycle argument in Remove-PnPListItem, so they bypassed the recycle bin. Going to upload the 5050 again now :)
I can't seem to reproduce the issue here. I uploaded 5050 files, deleted them to the first stage recycle bin and then ran:
Restore-PnPRecycleBinItem
This worked fine and restored all 5050 items again.
Is the problem specifically to piping recyclebin items into Restore-PnPRecycleBinItem
? In that case it would take -Identity
as the parameter and that indeed does not allow a -RowLimit
as it would only retrieve that single item anyway. Also that seems to be working fine here.
What am I missing?
I apologise to anyone who's here looking for my previous comment that I deleted because I realised it was not helpful. The problem from the start has been that even when you pipe a single file to the restore function, the function would retrieve the entire list anyway (running into the 5000+ limit error). So the version of the function that accepts pipe input needs to limit its own retrieval function, or be rewritten in a way that does not retrieve the entire list before filtering it down to the ones you want to delete. The problem then is, if the retrieval function within restore-pnprecyclebinitem limits itself to 5000, what if the file it needs to delete is #5001?
So the test is (was):
In the broken scenario at the time of my original reply, this fails with {exceeds the list view threshold} because step 3 loaded the entire list from step 1 into a view, regardless as to what you piped to it. If the problem has been resolved, then @PrzemyslawKlys should be able to pipe a single file at a time to Restore-PnPRecycleBinItem without the error occurring as it did before, OR -RowLimit should be applicable to the version that accepts piped input so that limit can be passed through to the internal get-pnpRecycleBinItem.
OR -RowLimit should be applicable to the version that accepts piped input so that limit can be passed through to the internal get-pnpRecycleBinItem.
But of course, then you run into the issue of "What if the file I'm looking for is #5001" but that's for smarter minds to figure out.
To me, Restore/Remove commands needs work because:
If you go into the web to restore something, you click 5 checkboxes and press restore and it restores them in 5 seconds. or less.
In my tenant, I've over 150k items deleted (I've not checked further) and I needed to restore 70 of them based on a path.
# Get all items from recycle bin - although it would be cool if there would be -Filter option to not store 150k items in variable as AD cmdlets have
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
# Here I'm limiting the outputt and test it with PSWriteHTML function Out-HtmlView to see if what I want to restore is there
$Files = $Stage | Where-Object { $_.DirName -like "$Test*"} #| Out-HtmlView -ScrollX -DisablePaging
# Here I simply go item by item and restore it, i shouldn't have to use pipeline or rowlimit
foreach ($File in $Files) {
$File | Restore-PnPRecycleBinItem
}
# Here's of course pipeline version where $Files could have 1 item, 5 items 50k items and it should restore them (without rowlimit)
$Files | Restore-PnPRecycleBinItem
As you see above the RowLimit is a bit special - why would it be needed for Restore/Remove commands if you already have the data using Get command? You should simply pass whole item as pipeline or as parameter or alternatively accept ID as pipeline/parameter - without executing in background another get of 150k (or more elements - just to get 5 items).
I can't seem to reproduce the issue here. I uploaded 5050 files, deleted them to the first stage recycle bin and then ran:
Restore-PnPRecycleBinItem
This worked fine and restored all 5050 items again.
Is the problem specifically to piping recyclebin items into
Restore-PnPRecycleBinItem
? In that case it would take-Identity
as the parameter and that indeed does not allow a-RowLimit
as it would only retrieve that single item anyway. Also that seems to be working fine here.What am I missing?
You're doing it wrong ;-) Using Restore-PnPRecycleBinItem
without any parameters or Restore-PnPRecycleBinItem -RowLimit 15000
will restore all 15000 items. We're trying to restore 1, 5, 1000 items that we want. So you need a way to pass Items/IDs to Restore and not even need RowLimit for that, as you already have them on you, and you know what you want to delete.
On the web interface when you try to restore 5 items that are on 7 different pages it's a nightmare. I had to switch to classic view because modern view refreshes automatically loosing track. On classic I was able to go into proper place but very time I pressed restore on an item it went to the beginning of the list. And then I had to click 200+ (wait 5 seconds to change page) times to get to items on a list 22000 choose 5 items and restore again, and repeat.
@PrzemyslawKlys that sounds exactly like what I said:
Is the problem specifically to piping recyclebin items into Restore-PnPRecycleBinItem? In that case it would take -Identity as the parameter and that indeed does not allow a -RowLimit as it would only retrieve that single item anyway. Also that seems to be working fine here.
Can you share an exact PnP PowerShell line which returns an error for you?
I now notice there have been more replies. The sample you're sharing is the one I used and worked for me. I'll give it another try. Waiting now for my script to complete which deletes the 5050 items into the recycle bin so I can try it again.
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
foreach ($File in $Stage) {
$File | Restore-PnPRecycleBinItem -RowLimit 150000
}
Or
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
foreach ($File in $Stage) {
Restore-PnPRecycleBinItem -RowLimit 150000 -Identity $File
}
But you can easily see the problem by simply typing the command and trying to find -Identity parameter while RowLimit is set.
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Stage | Select-Object -Last 5 | Restore-PnPRecycleBinItem
I shouldn't need RowLimit for Restore-PnPRecycleBin at all in this case, as I am restoring just 5 elements, that I already know/have, as Get command provided them to me.
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000 $Stage | Select-Object -Last 5 | Restore-PnPRecycleBinItem
I shouldn't need RowLimit for Restore-PnPRecycleBin at all in this case, as I am restoring just 5 elements, that I already know/have, as Get command provided them to me.
You would need it on the Get, but not on the Restore, so this script is exactly as you would like to see it, correct? I agree that this script should work. Still waiting for the deletes to be done to try it out.
None of the options I provided worked without RowLimit. With old version there's RowLimit on Get, and there's no RowLimit on Restore and I wasn't able to restore 70 items that sit between 22000 and 24000. In new version RowLimit is there but it removes the Identity parameter once it's used.
Make sure it works this way as well, as it's valid use case to first limit output, verify it and then pass to restore.
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Limited = $Stage | Select-Object -Last 5
$Limited | Restore-PnPRecycleBinItem
Optionally if possible that would be good as well as I often skip pipelines
$Stage = Get-PnPRecycleBinItem -SecondStage -RowLimit 150000
$Limited = $Stage | Select-Object -Last 5
Restore-PnPRecycleBinItem -Identity $Limited
@PrzemyslawKlys Got a filled up recycle bin again :) Just curious, does this throw an exception for you already?
Yes
Your test is also working fine here:
This will not be possible:
Restore-PnPRecycleBinItem -Identity $Limited
As that would mean it would need to accept an array, which would be different from all other cmdlets. So we will have to stick with the pipe which basically for-eaches through the collection.
Strange that it doesn't work for you. Are you sure you're using my custom build?
Some thresholds are only starting to kick in at 20,000 items. I'm wondering if that may be making the difference here as you mentioned to have a huge recycle bin?
Yes, your custom build for sure.
PS C:\Support\GitHub\O365Essentials> get-module SharePointPnPPowerShellOnline | fl
Name : SharePointPnPPowerShellOnline
Path : C:\Support\GitHub\O365Essentials\SharePointPnPPowerShellOnline\SharePointPnP.PowerShell.Online.Commands.dll
Description : SharePoint Patterns and Practices PowerShell Cmdlets for SharePoint Online
ModuleType : Binary
Version : 3.22.2006.2
NestedModules : {}
ExportedFunctions :
ExportedCmdlets : {Add-PnPAlert, Add-PnPApp, Add-PnPApplicationCustomizer, Add-PnPClientSidePage...}
ExportedVariables : WebRequestCounter
ExportedAliases : {Connect-PnPHubSite, Disconnect-PnPHubSite, Ensure-PnPFolder, Execute-PnPQuery...}
Most likely your tenant has higher thresholds limit then mine, that's why you don't get the error. I am not sure how I can change the view. I don't even know what size of recycle bin I have, but at 150k items in 2nd stage i was still able to get more.
What if you would try doing something like this:
$Stage = Get-PnPRecycleBinItem -RowLimit 15000
$Stage | Select-Object -Last 5 | Restore-PnPRecycleBinItem -RowLimit 150
Surely it should give you an error because of RowLimit and Identity at the same time.
In my case, the first stage recycle bin has about 120000 items and I'm attempting to restore about about 107000 of those. I'm able to filter and retrieve the items, but when I try to pass that to the restore command it fails.
$BinItems = Get-PnPRecycleBinItem -RowLimit 120000 -FirstStage | Where-Object -Property Leafname -Like -Value "." | Where-Object -Property DeletedDateLocalFormatted -Like -Value "6/26/2020" | Where-Object -Property Dirname -Like -Value "sites/MY-Site/Shared Documents/" | Where-Object -Property DeletedByEmail -EQ –Value John.Doe@contoso.com $BinItems | Restore-PnPRecycleBinItem -Force
I've tried several different variations at this point for the restore with different for, foreach, etc. I'm losing track. lol For example, I tried: foreach ($Item in $BinItems) {Restore-PnPRecycleBinItem -Identity $Item.Id -Force}
With your new version I tried adding the -RowLimit to the Restore-PnPRecycleBinItem in a few different scenarios, but still no luck.
Confirming that the o365 sharepoint view list threshold is 5000. The difference in results we're seeing here would be neatly explained if @KoenZomers is using a sharepoint server with a higher limit.
I'm also testing against a normal multi tenant, so there should be no difference in threshold levels. I've just generated another 16.000 documents in my library and am running a script now to throw all of them into the recycle bin. I did already get an error running just Get-PnPListItem
now regarding the threshold, so perhaps that explains why it didn't give me any issues before. Will run the before mentioned tests on the recycle bin here once it's done throwing everything into the bin.
Passed the 20.000 items in my recycle bin and it still works fine
I'm starting to think my custom build perhaps doesn't contain the proper updates. I have compiled a new build. Can you guys try this version and see if the results are different for you with this build?
Nope
The RowLimit implementation seems to have paging built in, so Get-PnPRecycleBinItem -RowLimit 99999999
would theoretically work where Get-PnPRecycleBinItem
would not. I don't know what magic happens in the RowLimit code to get single-item operations to work within its paged loop, but that magic is definitely not in place in these specific cmdlets - when the Identity parameter is specified, they just use the standard sharepoint single-item commands which do not work in environments over the list view threshold.
I think the code is important to include in the release even as-is, because without the ability to do bulk operations via RowLimit there's literally no way to bring a recycle bin back under that 5,000-item threshold for O365 tenants - not through the web interface, not through powershell.
The ability to work with individual items in an environment over that threshold is important, but less hair-on-fire IMO. Whoever understands how RowLimit does its thing (not me, I could barely get this far) will need to take that strategy and apply it to the single-item commands so that Identity will work too.
The RowLimit implementation seems to have paging built in, so
Get-PnPRecycleBinItem -RowLimit 99999999
would theoretically work whereGet-PnPRecycleBinItem
would not. I don't know what magic happens in the RowLimit code to get single-item operations to work within its paged loop, but that magic is definitely not in place in these specific cmdlets - when the Identity parameter is specified, they just use the standard sharepoint single-item commands which do not work in environments over the list view threshold.I think the code is important to include in the release even as-is, because without the ability to do bulk operations via RowLimit there's literally no way to bring a recycle bin back under that 5,000-item threshold for O365 tenants - not through the web interface, not through powershell.
The ability to work with individual items in an environment over that threshold is important, but less hair-on-fire IMO. Whoever understands how RowLimit does its thing (not me, I could barely get this far) will need to take that strategy and apply it to the single-item commands so that Identity will work too.
But Get command is already released with RoWLimit. Restore is broken and releasing it as is makes no difference right? You can't specify RowLimit and Identity at same time. This means you can restore all of it or none of it, unless you are under 5000 limit. I dont understand why would you create 2 parameter sets where you need it in all cases.
The release updates Clear-PnPRecycleBinItem
, which would not work at all. That's crucial to letting people clear recycle bins over 5000 items. It also adds RowLimit to the other related cmdlets like Restore-PnPRecycleBinItem
even though it doesn't support single-item operations yet, so before this release you couldn't do anything with a large bin except list the items in it and after this release you can delete or restore items from large bins wholesale.
Sure, there's more we need to do re: single-item operations, but that's no reason to not release an update that fixes a blocking bug with a core O365 user story :)
By all means, I am not blocking. Release away. You can release daily as far as I am concerned (that's what I do with my PowerShell modules).
Reporting an Issue or Missing Feature
"Get-PnPRecycleBinItem" command on the recyclebin with a lot of items fails with "the list view threshold" error.
I have tried to filter results by following commands such as "Get-PnPRecycleBinItem | select -last 10" and "Get-PnPRecycleBinItems | ? DeletedDate -gt $today" however, these commands did not help. I got the same error.
This worked for recycle bin with more than 40k items some time ago.
Is there any workaround solution? Is it possible to fix this trouble in PnP-PowerShell module?
Expected behavior
Restored files/files list
Actual behavior
Error message: "Get-PnPRecycleBinItem : The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator. At line:1 char:2
Steps to reproduce behavior
Connect-PnPOnline -Url https://... (Get-PnPRecycleBinItem).count
Which version of the PnP-PowerShell Cmdlets are you using?
What is the version of the Cmdlet module you are running?
3.1.1809.0
How did you install the PnP-PowerShell Cmdlets?