get-searchresult with type record returns only the last page of results if more than 100 rows

Get-SearchResult should return all the result rows for the query not just the last page of results. This is what it used to do before the recent re-write and publication on powershell gallery.

The new version has a bug that each loop it overwrites the entire result array and does not append. On the previous version we had a query that typically returned over 3000 rows now it never returns more than 100.

I have created a test script to demonstrate the erroneous behaviour. You can see three problems here. The first one renders the command unusuable for any query with ore than 100 results.

the worst problem here is that the results are overwritten not appended to an array of results at each loop. So the function returns only the last incomplete page if there are ever more than 100.

This is clearest in the last lot of test reults where I get either 100 or 73 results even though there are 500 rows in the results query.

This happens because of this: https://github.com/SumoLogic/sumo-powershell-sdk/blob/master/src/main/SumoLogic-Core/Lib/Utils.ps1#L170

I expect the same behaviour will happen for both result types. You should be appending to a $results array not overwriting it.

the limit is not clear what it does and doens't really work either. My test shows that -limit argument: a) does not mean the function returns a number of rows as with say limit 1 you don't get 1 back. b) bizarre things happen with the size of last page based on what you pick with limit
If you look at teh first group the .Count property is null. That is because you return $results not a typed object so powershell defaults to changing the type if there is 0, 1 or more than 1 (where it returns an object with .Count property.). This is annoying but the old command did this too. You could fix that by changing the function to here so it defined a [array]$results=@() object and always returned that object with a return statement.

https://github.com/SumoLogic/sumo-powershell-sdk/blob/master/src/main/SumoLogic-Core/Lib/Utils.ps1#L180

$record_limits= @(1,5,20,120,500);

$getresult_limit=@(1,5,20,120,200,300);

foreach ($l in $record_limits) {
    write host "`n`n ######## test query '| limit $l'`n`n"
        $query = "_sourcecategory=test/* | count by _collector,_source,_sourcename,_sourcehost | limit $l"
        write-host $query
        write-host "lets run a series or queries with different values for get-seaarchjob -limit x"
        foreach ($g in $getresult_limit){
        # this should generate us a good number of rows. With l we can cap the rows returned
        write-host "... with Start-SearchJob -Limit $g"
        $j=Start-SearchJob -Query $query -Last '00:15:00' | Get-SearchResult -Type Record -limit $g
        write-host "rows: $($j.Count) objecttype: $($j.GetType())"
        #exit
    }

}

test output:


 ######## test query '| limit 1'

_sourcecategory=test/* | count by _collector,_source,_sourcename,_sourcehost | limit 1
lets run a series or queries with different values for get-seaarchjob -limit x
... with Start-SearchJob -Limit 1
rows:  objecttype: System.Management.Automation.PSCustomObject
... with Start-SearchJob -Limit 5
rows:  objecttype: System.Management.Automation.PSCustomObject
... with Start-SearchJob -Limit 20
rows:  objecttype: System.Management.Automation.PSCustomObject
... with Start-SearchJob -Limit 120
rows:  objecttype: System.Management.Automation.PSCustomObject
... with Start-SearchJob -Limit 200
rows:  objecttype: System.Management.Automation.PSCustomObject
... with Start-SearchJob -Limit 300
rows:  objecttype: System.Management.Automation.PSCustomObject
host

 ######## test query '| limit 5'

_sourcecategory=test/* | count by _collector,_source,_sourcename,_sourcehost | limit 5
lets run a series or queries with different values for get-seaarchjob -limit x
... with Start-SearchJob -Limit 1
rows: 5 objecttype: System.Object[]
... with Start-SearchJob -Limit 5
rows: 5 objecttype: System.Object[]
... with Start-SearchJob -Limit 20
rows: 5 objecttype: System.Object[]
... with Start-SearchJob -Limit 120
rows: 5 objecttype: System.Object[]
... with Start-SearchJob -Limit 200
rows: 5 objecttype: System.Object[]
... with Start-SearchJob -Limit 300
rows: 5 objecttype: System.Object[]
host

 ######## test query '| limit 20'

_sourcecategory=test/* | count by _collector,_source,_sourcename,_sourcehost | limit 20
lets run a series or queries with different values for get-seaarchjob -limit x
... with Start-SearchJob -Limit 1
rows: 20 objecttype: System.Object[]
... with Start-SearchJob -Limit 5
rows: 20 objecttype: System.Object[]
... with Start-SearchJob -Limit 20
rows: 20 objecttype: System.Object[]
... with Start-SearchJob -Limit 120
rows: 20 objecttype: System.Object[]
... with Start-SearchJob -Limit 200
rows: 20 objecttype: System.Object[]
... with Start-SearchJob -Limit 300
rows: 20 objecttype: System.Object[]
host

 ######## test query '| limit 120'

_sourcecategory=test/* | count by _collector,_source,_sourcename,_sourcehost | limit 120
lets run a series or queries with different values for get-seaarchjob -limit x
... with Start-SearchJob -Limit 1
rows: 100 objecttype: System.Object[]
... with Start-SearchJob -Limit 5
rows: 100 objecttype: System.Object[]
... with Start-SearchJob -Limit 20
rows: 100 objecttype: System.Object[]
... with Start-SearchJob -Limit 120
rows: 20 objecttype: System.Object[]
... with Start-SearchJob -Limit 200
rows: 20 objecttype: System.Object[]
... with Start-SearchJob -Limit 300
rows: 20 objecttype: System.Object[]
host

 ######## test query '| limit 500'

_sourcecategory=test/* | count by _collector,_source,_sourcename,_sourcehost | limit 500
lets run a series or queries with different values for get-seaarchjob -limit x
... with Start-SearchJob -Limit 1
rows: 100 objecttype: System.Object[]
... with Start-SearchJob -Limit 5
rows: 100 objecttype: System.Object[]
... with Start-SearchJob -Limit 20
rows: 100 objecttype: System.Object[]
... with Start-SearchJob -Limit 120
rows: 100 objecttype: System.Object[]
... with Start-SearchJob -Limit 200
rows: 73 objecttype: System.Object[]
... with Start-SearchJob -Limit 300
rows: 73 objecttype: System.Object[]

@bin3377 FYI.

SumoLogic-Labs / sumo-powershell-sdk

get-searchresult with type record returns only the last page of results if more than 100 rows #20