search

satyrius / gonx

Nginx access log parser for Golang
MIT License
356 stars 86 forks source link

Multicore performance #30

Open przmv opened 8 years ago

przmv commented 8 years ago

Hello,

I've noticed recently that running gonx-based applications on more CPUs doesn't improve the performance. So, I've created a benchmark processing 1 million records and measured the results on four multi-core VM instances on Google Cloud Platform — https://cloud.google.com/compute/docs/machine-types#highcpu

Also I've compared 'em with another benchmark results utilizing glow library.

Here are the resulting charts: image

image 1

satyrius commented 8 years ago

Thanks for benchmarks! The library has no multicore support or any kink of autoscaling, it uses hardcoded number of workers https://github.com/satyrius/gonx/blob/master/mapreduce.go#L25. I think we can calculate topLoad depending on CPU count or accept desired count as a parameter. And we should play with GOMAXPROCS