splunk / eventgen

Splunk Event Generator: Eventgen
Apache License 2.0
380 stars 179 forks source link

[BUG] Memory increases in extendIndexes feature. #259

Closed Yangxulight closed 5 years ago

Yangxulight commented 5 years ago

Describe the bug Using eventgenx feature "extendIndexes" to run with 900 indexes for one sample, I saw memory increase. (in 16 hours it increased from 300MB to 4GB)

To Reproduce Use the following conf to generate data.


httpeventMaxPayloadSize = 1048576
index = main
httpeventWaitResponse = true
outputCounter = true
disabled = false
debug = false
verbose = false
spoolDir = $SPLUNK_HOME/var/spool/splunk
spoolFile = <SAMPLE>
breaker = [^\r\n\s]+
mode = sample
sampletype = raw
interval = 60
delay = 0
timeMultiple = 1
earliest = now
latest = now
randomizeEvents = false
outputMode = httpevent
fileMaxBytes = 10485760
fileBackupFiles = 5
splunkPort = 8089
splunkMethod = https
index = main
source = eventgen
sourcetype = eventgen
host = 127.0.0.1
generator = default
rater = config
generatorWorkers = 1
outputWorkers = 1
timeField = _raw
threading = thread
profiler = true
maxIntervalsBeforeFlush = 3
maxQueueLength = 0
useOutputQueue = false
autotimestamps = [["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}", "%Y-%m-%d %H:%M:%S"], ["\\d{1,2}\\/\\w{3}\\/\\d{4}\\s\\d{2}:\\d{2}:\\d{2}:\\d{1,3}", "%d/%b/%Y %H:%M:%S:%f"], ["\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}\\.\\d{3}", "%Y-%m-%dT%H:%M:%S.%f"], ["\\d{1,2}/\\w{3}/\\d{4}\\s\\d{2}:\\d{2}:\\d{2}:\\d{1,3}", "%d/%b/%Y %H:%M:%S:%f"], ["\\d{1,2}/\\d{2}/\\d{2}\\s\\d{1,2}:\\d{2}:\\d{2}", "%m/%d/%y %H:%M:%S"], ["\\d{2}-\\d{2}-\\d{4} \\d{2}:\\d{2}:\\d{2}", "%m-%d-%Y %H:%M:%S"], ["\\w{3} \\w{3} +\\d{1,2} \\d{2}:\\d{2}:\\d{2}", "%a %b %d %H:%M:%S"], ["\\w{3} \\w{3} \\d{2} \\d{4} \\d{2}:\\d{2}:\\d{2}", "%a %b %d %Y %H:%M:%S"], ["^(\\w{3}\\s+\\d{1,2}\\s\\d{2}:\\d{2}:\\d{2})", "%b %d %H:%M:%S"], ["(\\w{3}\\s+\\d{1,2}\\s\\d{1,2}:\\d{1,2}:\\d{1,2})", "%b %d %H:%M:%S"], ["(\\w{3}\\s\\d{1,2}\\s\\d{1,4}\\s\\d{1,2}:\\d{1,2}:\\d{1,2})", "%b %d %Y %H:%M:%S"], ["\\d{4}-\\d{2}-\\d{2} \\d{2}:\\d{2}:\\d{2}\\.\\d{3}", "%Y-%m-%d %H:%M:%S.%f"], ["\\,\\d{2}\\/\\d{2}\\/\\d{2,4}\\s+\\d{2}:\\d{2}:\\d{2}\\s+[AaPp][Mm]\\,", ",%m/%d/%Y %I:%M:%S %p,"], ["^\\w{3}\\s+\\d{2}\\s+\\d{2}:\\d{2}:\\d{2}", "%b %d %H:%M:%S"], ["\\d{2}/\\d{2}/\\d{4} \\d{2}:\\d{2}:\\d{2}", "%m/%d/%Y %H:%M:%S"], ["^\\d{2}\\/\\d{2}\\/\\d{2,4}\\s+\\d{2}:\\d{2}:\\d{2}\\s+[AaPp][Mm]", "%m/%d/%Y %I:%M:%S %p"], ["\\d{2}\\/\\d{2}\\/\\d{4}\\s\\d{2}:\\d{2}:\\d{2}", "%m-%d-%Y %H:%M:%S"], ["\\\"timestamp\\\":\\s\\\"(\\d+)", "%s"], ["\\d{2}\\/\\w+\\/\\d{4}\\s\\d{2}:\\d{2}:\\d{2}:\\d{3}", "%d-%b-%Y %H:%M:%S:%f"], ["\\\"created\\\":\\s(\\d+)", "%s"], ["\\d{4}-\\d{2}-\\d{2}T\\d{2}:\\d{2}:\\d{2}", "%Y-%m-%dT%H:%M:%S"], ["\\d{1,2}/\\w{3}/\\d{4}:\\d{2}:\\d{2}:\\d{2}:\\d{1,3}", "%d/%b/%Y:%H:%M:%S:%f"], ["\\d{1,2}/\\w{3}/\\d{4}:\\d{2}:\\d{2}:\\d{2}", "%d/%b/%Y:%H:%M:%S"]]
autotimestamp = false
httpeventWaitResponse = true
httpeventServers = {"servers": [{"protocol": "https", "port": "8088", "key": "02fb6992-00e2-49f0-b170-36df8a308682", "address": "127.0.0.1"}]}
httpeventOutputMode = roundrobin

[network_samplelog_tcp_connection_failure.cisco.asa]
token.0.replacement = integer[10000:50000]
token.3.token = @@port2@@
token.5.replacement = integer[0:100]
token.4.replacementType = random
token.0.token = @@randint@@
token.2.token = @@port1@@
token.1.replacement = /Users/xyang/Project/github/eventgen/tests/samples/100k_user_comp_ip.sample
bundlelines = False
token.3.replacement = /Users/xyang/Project/github/eventgen/tests/samples/network_port2.sample
index = main
token.2.replacementType = random
perDayVolume = 60
sourcetype = cisco:asa
source = samplelog_tcp_connection_failure
token.0.replacementType = random
token.5.replacementType = random
token.2.replacement = integer[1:8000]
token.3.replacementType = mvfile
token.1.token = @@external_ip@@
token.5.token = @@rand_bytes@@
token.4.token = @@rand_duration@@
token.1.replacementType = mvfile
interval = 3
token.4.replacement = integer[0:9]
extendIndexes = test_:900```

**Expected behavior**
Memory usage should be stable.

**Actual behavior**
Memory usage increase from 300MB to 4GB in 16 hours. And then my container exited by OOM problem.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Sample files and eventgen.conf file**
Please attach your sample files and eventgen conf file

**Do you run eventgen with SA-eventgen?**
No

**If you are using eventgen with pip module mode (please complete the following information):**
 - python version: [e.g. 2.6]
 - OS: [e.g. Windows10]
 - Virtual Env is used: Yes/No
 - Eventgen Version [e.g. 6.3.2]

**Additional context**
Add any other context about the problem here.
This was caused by bugs in extendIndexes feature. Every time the generator calls "loadSamples" function will trigger eventgen to append indexer list to each samples, which makes memory increase.