At the moment iterating through the BigLines.iterator is running on O(n!), which will take extremely long to finish on a real big lines. App need to apply a special idiom to do the iteration to avoid the performance issue:
int bufSize = 20000;
int offset = 1;
do {
List<String> buf = bigLines.fetch(offset, bufSize);
for (String line : buf) {
// consume the line
}
offset += bufSize;
} while (offset < lines);
The enhancement here is to encapsulate the above logic inside the iterator and simply application usage.
At the moment iterating through the
BigLines.iteratoris running onO(n!), which will take extremely long to finish on a real big lines. App need to apply a special idiom to do the iteration to avoid the performance issue:The enhancement here is to encapsulate the above logic inside the iterator and simply application usage.