search

zh54321 / SnafflerParser

Parses Snaffler output file and generate beautified outputs.
37 stars 1 forks source link

Parsing of large files #1

Open pinetest opened 1 month ago

pinetest commented 1 month ago

In the beginning this project states "Especially in large environments, the Snaffler output gets very large and time-consuming to analyze." But in the code you use 'Get-Content' to load the entire file into PowerShell memory. I have an output of 1.2GB and this script takes 15 minutes to load the file (and using all of the ram availible) to output the first line "Input file had 4123123 lines" before taking way longer to parse the rest.

Is it possible to parse the file line-by-line preventing loading it all in memory and only storing the least necessary data in variables and the rest direct to a file?

zh54321 commented 1 month ago

Thank you for the suggestion. 4 Millions identified files / a 1.2 GB output file is indeed huge. I never had such a case. I will look into it. However, it will take me a couple of weeks until I can dedicate some time for it.