ReadingLines() function fails with 'bufio.Scanner: token too long' error on large line data

[c0r1]

Hi,

When I run Rustscan for port scanning, some IPs return a large number of open ports. As a result, the generated raw-open-ports.txt file looks like the following:

However, when I run CleanRustScan() to process the above raw-open-ports.txt file, I noticed that the function throws a "bufio.Scanner: token too long" error when processing some very large line data.

Therefore, I suggest setting the maximum token size in the scanner.Buffer() function's second parameter to support reading files with very large line data. For example, you can use the following code:

scanner := bufio.NewScanner(file)
scanner.Buffer(make([]byte, 0, 64*1024), 1024*1024)
for scanner.Scan() {
    val := strings.TrimSpace(scanner.Text())
    if val == "" {
        continue
    }
    result = append(result, val)
}

[j3ssie]

Thanks for raising the issue. I've pushed an updated version that addresses the problem you mentioned.