search

osglworks / java-tool

Some simple common Java utilities
Apache License 2.0
52 stars 18 forks source link

Osgl performance issue - too many temporary objects generated with functional call chain. #236

Open greenlaw110 opened 4 years ago

greenlaw110 commented 4 years ago

Context

In a technical chat room, it was talking about use a Java program to simulate the following shell command:

cat *.log | grep 'Login' | sort | uniq -c | sort -nr

I have come up with a project to implement the function. After I get the function working logically, I start check the performance of the app and found issue with osgl-tool library.

Issue

I have generated 100 log files with each file got 2M lines. Then I run my code. What I found is when I change this line:

C.seq(bigLines).filter(s -> s.contains(keyword)).accept(lineCollector);

to 

for (String line: bigLines) {
    if (line.contains(keyword) {
        lineCollector.accept(line);
    }
}

The performance has been improved from 12s to 10s. I use VisualVM to inspect what happened, and found there are lots of temporary objects created when we are using the functional programming style:

image