For large (1gb+) files, we probably don't want to read the whole thing at once if possible. Instead we should break scanning and
performing tasks on the file up into chunks. Ideally the program will do this automatically for large files, and support command-line override of this behavior. E.g. chunk-size=50mb (valid values are number of mb to chunk, or -1 for never chunk)
For large (1gb+) files, we probably don't want to read the whole thing at once if possible. Instead we should break scanning and performing tasks on the file up into chunks. Ideally the program will do this automatically for large files, and support command-line override of this behavior. E.g. chunk-size=50mb (valid values are number of mb to chunk, or -1 for never chunk)