search

12Knocksinna / Office365itpros

Office 365 for IT Pros PowerShell examples
MIT License
1.33k stars 577 forks source link

ReportFilesSPOSite.PS1 - Creating duplicate rows of data #111

Closed IWSITJS closed 4 months ago

IWSITJS commented 4 months ago

Primary issue: The script is exporting duplicate data and ballooning CSV Export to the point where the file cannot be opened.

I am working through this script in an environment that has over 300k items in a single document library. While this script functionally works very well, it is exporting duplicate data. I was able to work with the data a few months ago when there were closer to 250k items, but now the CSV has over 3 million lines and I am unable to open or load the file with power query.

I have not been able to resolve the code, though believe the issue may be related to the logic in the while loop with "@odata.nextLink" and it may be re-processing files and folders.

I have been unsuccessfully attempting a workaround of adding a unique sort from a temporary variable and I am running into several errors. 

# Add unique files to the final collection
foreach ($item in $SiteFilesTemp | Sort-Object Url -Unique) {
    $SiteFiles.Value.Add($item)
}

I would provide more information though all of my edits have not succeeded.

12Knocksinna commented 4 months ago

That script was one of the first that I tried to write for SharePoint Online. The most recent version (described in https://practical365.com/sharepoint-online-files-report/) is https://github.com/12Knocksinna/Office365itpros/blob/master/Report-SPOFilesDocumentLibrary.PS1. This version uses the Microsoft Grah PowerShell SDK and it seems a bit more robust. I have not tried to run it on a library with 300K items though...

IWSITJS commented 4 months ago

The other script works as intended, thank you.