search

splunk / eventgen

Splunk Event Generator: Eventgen
Apache License 2.0
380 stars 179 forks source link

[BUG] ITSI bundle perdayvolume throughput ~1/3 compared to eventgen 5 #342

Closed YifengMao closed 5 years ago

YifengMao commented 5 years ago

Describe the bug Throughput of ITSI data generation configure bundle (attached) on an aws i3.4xl host reduces from 350G(eventgen5 multiprocess) to 150G(eventgen latest 7.3 version, standalone server mode + multiprocess)

To Reproduce apply the bundle, set payload_size=756K, interval=20, process_num = 20. for eventgen latest, run "/usr/bin/python3.7 /usr/bin/splunk_eventgen service --role standalone&"

issue.tar.gz

arctan5x commented 5 years ago

How are you measuring the throughput? Are you measuring after one day? How are you setting up the receiving splunk instance? Also, perDayVolume sum in the conf is not 350. Are you setting the total volume to 350? Please provide the chain of api calls you invoked.

Yangxulight commented 5 years ago

I have try two kind of builds in one container, here is the result on that bundle with the same settings, I have set 500GB/day totally for eventgen.:

  1. eventgen latest build on python3.7(7.0.0): 800KB/s, CPU: 13% usr 86% sys 0% nic 0% idle
  2. eventgen build on python2-stable branch(6.5.2): 4.9MB/s, CPU: 91% usr 7% sys 0% nic 0% idle

Looks like there are too many CPU usage on system level about eventgen python3 version.

li-wu commented 5 years ago

This issue has been fixed. Close it.