search

CrowdStrike / psfalcon

PowerShell for CrowdStrike's OAuth2 APIs
The Unlicense
345 stars 64 forks source link

[ BUG ] `Get-FalconScan` and `Get-FalconScanFile` limited to 100 results #412

Open HanFastolfe7 opened 1 month ago

HanFastolfe7 commented 1 month ago

Describe the bug Get-FalconScan -Detailed -All and Get-FalconScanFile -Detailed -All result in error {"code":404,"message":"404: Page Not Found"}. 100 results are returned, presumed issue in requesting the next page of results.

To Reproduce

Get-FalconScan -Detailed -All
Get-FalconScanFile -Detailed -All

Expected behavior All FalconScan and FalconScanFile objects are returned, in line with Get-FalconHost and Get-FalconDetection.

Environment (please complete the following information):

Transcript content Cut from full transcript, same error for both commands at GET https://api.crowdstrike.com/ods/queries/scans/v1&offset=100

VERBOSE: 07:25:50 [ApiClient.Invoke] Accept=application/json
VERBOSE: 07:25:50 [ApiClient.Invoke] 200: OK
VERBOSE: 07:25:50 [ApiClient.Invoke] Server=nginx, Date=Tue, 02 Jul 2024 11:25:50 GMT, Transfer-Encoding=chunked, Connection=keep-alive, Strict-Transport-Security=max-age=15724800; includeSubDomains, max-age=31536000; includeSubDomains, X-Cs-Region=us-1, X-Cs-Traceid=b0e99352-df6c-47f6-ac2a-c9d40f982527, X-Ratelimit-Limit=6000, X-Ratelimit-Remaining=5998
VERBOSE: 07:25:50 [Write-Result] query_time=0.142178182, writes=, powered_by=svc-odsapi, trace_id=b0e99352-df6c-47f6-ac2a-c9d40f982527
VERBOSE: 07:25:50 [Get-FalconScan] Retrieved 100 of 10000
VERBOSE: 07:25:50 [ApiClient.Invoke] GET https://api.crowdstrike.com/ods/queries/scans/v1&offset=100
VERBOSE: 07:25:50 [ApiClient.Invoke] Accept=application/json
VERBOSE: 07:25:50 [ApiClient.Invoke] 404: NotFound**
VERBOSE: 07:25:50 [ApiClient.Invoke] Server=nginx, Date=Tue, 02 Jul 2024 11:25:50 GMT, Connection=keep-alive, X-Content-Type-Options=nosniff, X-Cs-Traceid=16999236-4b3d-4ebd-a586-ea9be78948e7, Strict-Transport-Security=max-age=31536000; includeSubDomains
VERBOSE: 07:25:51 [Write-Result] query_time=2.24E-07, powered_by=crowdstrike-api-gateway, trace_id=16999236-4b3d-4ebd-a586-ea9be78948e7
Write-Result: C:\Users\[...]\Documents\PowerShell\Modules\psfalcon\2.2.6\private\Private.ps1:660
Line |
 660 |          $Output = Write-Result $Object
     |                    ~~~~~~~~~~~~~~~~~~~~
     | {"code":404,"message":"404: Page Not Found"}
bk-cs commented 1 month ago

In your transcript, I don't see any reference to meta.pagination. Was that cut out?

Have you tried reinstalling?

Uninstall-Module -Name PSFalcon -AllVersions
Install-Module -Name PSFalcon -Scope CurrentUser
HanFastolfe7 commented 1 month ago

I did the reinstall and ran through the testing steps again, here's the full transcript for Get-FalconScan

**********************
PowerShell transcript start
Start time: 20240702121819
Username: [...]
RunAs User: [...]
Configuration Name: 
Machine: [...] (Microsoft Windows NT 10.0.22631.0)
Host Application: C:\Program Files\PowerShell\7\pwsh.dll
Process ID: 37052
PSVersion: 7.4.3
PSEdition: Core
GitCommitId: 7.4.3
OS: Microsoft Windows 10.0.22631
Platform: Win32NT
PSCompatibleVersions: 1.0, 2.0, 3.0, 4.0, 5.0, 5.1, 6.0, 7.0
PSRemotingProtocolVersion: 2.3
SerializationVersion: 1.1.0.1
WSManStackVersion: 3.0
**********************
Transcript started, output file is C:\Users\[...]\Documents\PowerShell_transcript.[...].zHDtZYFJ.20240702121819.txt
PS C:\Users\[...]> show-FalconModule

PSVersion      : Core [7.4.3]
ModuleVersion  : v2.2.6 {d893eb9f-f6bb-4a40-9caf-aaff0e42acd1}
ModulePath     : C:\Users\[...]\Documents\PowerShell\Modules\PSFalcon\2.2.6
UserModulePath : C:\Users\[...]\Documents\PowerShell\Modules;C:\Program
                 Files\PowerShell\Modules;c:\program files\powershell\7\Modules;C:\Program Files
                 (x86)\WindowsPowerShell\Modules;C:\Windows\system32\WindowsPowerShell\v1.0\Modules;C:\Program Files
                 (x86)\ConfigMgr Console\bin
UserHome       : C:\Users\[...]
UserAgent      : crowdstrike-psfalcon/2.2.6

PS C:\Users\[...]> $scanList = Get-FalconScan -Detailed -All
VERBOSE: 12:18:31 [Get-FalconScan] /ods/queries/scans/v1:get
VERBOSE: 12:18:32 [ApiClient.Invoke] GET https://api.crowdstrike.com/ods/queries/scans/v1
VERBOSE: 12:18:32 [ApiClient.Invoke] Accept=application/json
VERBOSE: 12:18:32 [ApiClient.Invoke] 200: OK
VERBOSE: 12:18:32 [ApiClient.Invoke] Server=nginx, Date=Tue, 02 Jul 2024 16:18:31 GMT, Transfer-Encoding=chunked, Connection=keep-alive, Strict-Transport-Security=max-age=15724800; includeSubDomains, max-age=31536000; includeSubDomains, X-Cs-Region=us-1, X-Cs-Traceid=a4dd9261-d514-4b97-8575-472ecea65a1b, X-Ratelimit-Limit=6000, X-Ratelimit-Remaining=5998
VERBOSE: 12:18:32 [Write-Result] query_time=0.012675743, pagination.offset=0, pagination.limit=100, pagination.total=10000, powered_by=svc-odsapi, trace_id=a4dd9261-d514-4b97-8575-472ecea65a1b
VERBOSE: 12:18:32 [Get-FalconScan] /ods/entities/scans/v2:get
VERBOSE: 12:18:32 [ApiClient.Invoke] GET https://api.crowdstrike.com/ods/entities/scans/v2?ids=a0322d78eb9f48cfaa8d687ca13f2908&ids=683046bdfdba4b288284034c03f8cda9&ids=8d84db8865a64199b9db619e5f793d7d&ids=20f269e063e0403f85f1ee20cde406a5&ids=2067ea47259745b6b984d0f2cb8783f5&ids=1f4ddd6780234c7daa01f28ef1cafbea&ids=2ef382de60674da588d3db2775f476f5&ids=7098a6ca9bdd42a98fb292404ff7f044&ids=b5a06f42e4d64c2cbc9e6c0c4e417246&ids=4383ecbf346f45598c3d55243dfec5fb&ids=4abbda8d99f24665a9ed2ca96e8a6f7c&ids=839010a39e5145cd9c7d2ebc2eded699&ids=da299e822a30478c893bcbc304c96bac&ids=a8576cc00f5b49288974bfd8be342364&ids=65a4cfa195f1430cb389ace23825ea20&ids=42cf17853ad242e89f40c33aed92de2b&ids=b48b761d01254192af8e834128c03a07&ids=444140450b424982b9992863ee459505&ids=bf9d3292014a4a5ba1eca73ef8682a1c&ids=ad0c964db02d4834a4b02ecdfe81c2b2&ids=9b9ac0364d104202a9652a443b3abecb&ids=579bfeb2a4184d9ab30e22fd3c9964ed&ids=7996e18a7d9a48f7b9a0fdba8740c146&ids=70a35ac777a443b19d72d7f2ae2c4924&ids=8185ec77b39a486cab7ac6dc94afc17a&ids=ec80af97b2aa423aa95ba8365e1642ec&ids=76c978c223fe4343949c25bdc059656b&ids=1a90a1a023144e7faac393cc2d86c7f6&ids=d250d6ac3f7a4b2ebe3e08ca1e4d4bd1&ids=c0a40942917e4b67be9f1831190490cb&ids=0db7e73b30eb4ce5b80bd7ad19c33a88&ids=100cddd7811d47a283b8d14a6be8ffc8&ids=ff06a170e48e48929b82004811345711&ids=18901a521b6f41788a8a6e54b6c80457&ids=3c8c0280d63c4ca08407ca616f4cad41&ids=9260862c9fee430183b97006faa36b8b&ids=e278661795274b2b9161eea35771f28d&ids=ea85142073184bf69a292051b5a9b3ff&ids=ca95b8e086c64a32b15ed23e6626df35&ids=e088706be2db4451995b2ae1e2d5f5b2&ids=f01556a44e824106abbea40895295486&ids=e703ea037a534640be2f15c2d697720b&ids=12b180d1dc374101b2070aff742eb765&ids=f4d3e4ff9e50478cadf1de078d7602b8&ids=36bc8347dce740bf8febc86b007d6990&ids=a31025de24674dfba9b2d9fd8c3a5379&ids=b63bd26dffbe4c25ad209e37c9bec5fd&ids=657d6952266e49969b1e660912d60cf8&ids=962b82bc9d794cbcb1d2ca35f05b4d52&ids=6937972d14e340418fc87a63964df565&ids=95d4c7b5a5724dcb9802405eb56cdf86&ids=7c34b4170a9e4243b24769db30879643&ids=b5e224ab1be74b14a6465d041a366fbf&ids=7d6daa34516e4d3ea7cc0f7c41fe63b7&ids=5f015443092f4960b33b4b63f300c021&ids=17b9047f63c247229f2aae2c538f17b0&ids=00280bdd9b304f79a14debc69361e6bf&ids=7422fdb10fba4767879bfb7f027c27b2&ids=f11b1d37a7ef4d1394e6060472efe539&ids=07233464724e4aa8b68596893b170ee3&ids=5ad466382a1b4516b1b5d8395e165596&ids=09d100b4a4c44feb9cad956b7ada3631&ids=b27687c2fc574fdc8a122c112dbc4a03&ids=6dcd345591614df69ed6b8b01d9820e3&ids=cbe8c52747bf49ab808c23c0bb310233&ids=065f63b85a9249d49a3c13ae5b207463&ids=31a6be491a9f407da4d086b86e1a4add&ids=582eed06ff944c68b6c2e6e30747cb36&ids=cb21813e9d394f1fb73e05ab09d6b178&ids=cfe962206f114d188a958f90be181307&ids=46c7c1cee5174e72a58bc81b4ab8f8ab&ids=587876d6df0847059721c79dc16ccf2f&ids=96326ccd90fa429498e014a8876b31fa&ids=d6481d1d6499490890fb34992c8976b9&ids=3231826e8cad4ffbb6aa36f2e275c59f&ids=f634c98d947a4450b10ee4dcf0024dc3&ids=e462accc68384969a23b0c85cae48d2a&ids=ebbad7318bcb43a899178268a6d0c2d0&ids=ab49d654864a4933b83a97bad4e17c35&ids=416c1ee21397427480de2501b5935555&ids=d63b8c378875483abd63b3edf3c9ed63&ids=02959c73e4554cfc8967ad9f697428f5&ids=a49afb26d775423e889dd6593b0a874a&ids=77dd6a56b0aa4cb5852a6dd2bcd2c9f1&ids=7ba2227ee2ee4df399e9c78e60ac35a9&ids=5b220182e1284c37a873535b3a052d13&ids=7f10933edaf04aa6bf36fb5c883accea&ids=37f3fcc331b14bd99e38a7d5ebb07558&ids=5cb8396f48554f79a883ceeeeee2fa18&ids=57a2e47a7e5b46e98d72cafb20467493&ids=5dd121e0572b463b8d464225ef349786&ids=3424dce9eca244d1b094a28f46a5569e&ids=b2cc511bf37f45de84055b23702aaa1f&ids=9d1848ef7c0848038bcd1e0e333f9c9b&ids=a006b31832784c339166df46f4c771b9&ids=9001f58ae5cc4da9ad597c1f5b4d6e09&ids=f849a74e304b44c88827d51bace8e232&ids=5ca2940678464d229575506815c946ee&ids=81067098e6b44b66a6e820625d930585&ids=ea734bc25688424e88ff9c653357a5da
VERBOSE: 12:18:32 [ApiClient.Invoke] Accept=application/json
VERBOSE: 12:18:32 [ApiClient.Invoke] 200: OK
VERBOSE: 12:18:32 [ApiClient.Invoke] Server=nginx, Date=Tue, 02 Jul 2024 16:18:32 GMT, Transfer-Encoding=chunked, Connection=keep-alive, Strict-Transport-Security=max-age=15724800; includeSubDomains, max-age=31536000; includeSubDomains, X-Cs-Region=us-1, X-Cs-Traceid=eef205ae-63c5-480c-b010-68e06c3efb5d, X-Ratelimit-Limit=6000, X-Ratelimit-Remaining=5997
VERBOSE: 12:18:32 [Write-Result] query_time=0.095379703, writes=, powered_by=svc-odsapi, trace_id=eef205ae-63c5-480c-b010-68e06c3efb5d
VERBOSE: 12:18:32 [Get-FalconScan] Retrieved 100 of 10000
VERBOSE: 12:18:32 [ApiClient.Invoke] GET https://api.crowdstrike.com/ods/queries/scans/v1&offset=100
VERBOSE: 12:18:32 [ApiClient.Invoke] Accept=application/json
VERBOSE: 12:18:32 [ApiClient.Invoke] 404: NotFound
VERBOSE: 12:18:32 [ApiClient.Invoke] Server=nginx, Date=Tue, 02 Jul 2024 16:18:32 GMT, Connection=keep-alive, X-Content-Type-Options=nosniff, X-Cs-Traceid=48d08820-0b6f-432a-8ea3-b9475bf07b44, Strict-Transport-Security=max-age=31536000; includeSubDomains
VERBOSE: 12:18:32 [Write-Result] query_time=2.19E-07, powered_by=crowdstrike-api-gateway, trace_id=48d08820-0b6f-432a-8ea3-b9475bf07b44
Write-Result: C:\Users\[...]\Documents\PowerShell\Modules\PSFalcon\2.2.6\private\Private.ps1:660
Line |
 660 |          $Output = Write-Result $Object
     |                    ~~~~~~~~~~~~~~~~~~~~
     | {"code":404,"message":"404: Page Not Found"}
Write-Result: C:\Users\[...]\Documents\PowerShell\Modules\PSFalcon\2.2.6\private\Private.ps1:660
Line |
 660 |          $Output = Write-Result $Object
     |                    ~~~~~~~~~~~~~~~~~~~~
     | {"code":404,"message":"404: Page Not Found"}

PS C:\Users\[...]> stop-transcript
**********************
PowerShell transcript end
End time: 20240702121838
**********************
bk-cs commented 1 month ago

PSFalcon is doing what it's supposed to do here (reading meta.pagination and supplying the appropriate offset in response). Can you try one more test to see if you can successfully paginate in smaller numbers?

Get-FalconScan -Limit 10 -All

This seems to be an error with the API service that's outside of PSFalcon, but I'll need to troubleshoot further to figure out where the problem is.

bk-cs commented 1 month ago

I see at least one problem. The & character should be ? here:

VERBOSE: 12:18:32 [ApiClient.Invoke] GET https://api.crowdstrike.com/ods/queries/scans/v1&offset=100

I'm wondering if retrying with a limit will force it to properly insert a ?. Are you using non-US English language packs or anything that might impact how text is interpreted when attempting to build the URL string?

HanFastolfe7 commented 1 month ago

Only have English (US) language pack installed. Adding the -limit command does correct the URI, which is a workaround I can work with.

VERBOSE: 06:31:40 [Write-Result] query_time=0.042683545, pagination.offset=1100, pagination.limit=100, pagination.total=10000, powered_by=svc-odsapi, trace_id=18cfa045-3a9f-437e-bf79-dea6dcdb0be5 VERBOSE: 06:31:40 [Get-FalconScan] Retrieved 1200 of 10000 VERBOSE: 06:31:40 [ApiClient.Invoke] GET https://api.crowdstrike.com/ods/queries/scans/v1?limit=100&offset=1200 VERBOSE: 06:31:40 [ApiClient.Invoke] Accept=application/json VERBOSE: 06:31:40 [ApiClient.Invoke] 200: OK

bk-cs commented 1 month ago

Thank you for confirming! I'll troubleshoot why it's not properly inserting a ?.

bk-cs commented 1 month ago

Adding the -limit command does correct the URI, which is a workaround I can work with.

You can use -Limit 500 as the maximum amount of results per request.

I've added a ValidateRange attribute to Limit for Get-FalconScan and Get-FalconScanFile to avoid this problem in the next release, which will automatically be added when using -All (ensuring that the ? issue is avoided).

If you'd like to fix your local module before release, you can replace public\ods.ps1 using the steps outlined below.

Import-Module -Name PSFalcon
$ModulePath = (Show-FalconModule).ModulePath
(Invoke-WebRequest -Uri https://raw.githubusercontent.com/CrowdStrike/psfalcon/e24f96cc66bfba5839cd90f01fd9f692760625e5/public/ods.ps1 -UseBasicParsing).Content > (Join-Path (Join-Path $ModulePath public) ods.ps1)

I'm still trying to track down where you're getting an & instead of an ?, but this fix will ensure that it stops happening for those specific commands for now.