[BUG]replay issues

[lsnow11]

Description Replay mode does not seem to be working as expected. I am having problems with my own sample, and I'm also seeing issues with the data generated by eventgen.conf.tutorial1. The issue with the tutorial described below may just mean that the tutorial needs updating, but since I'm having issues of my own with replay, it would be nice to have something working to refer to.

To Reproduce Steps to reproduce the behavior:

1. Run tutorial1 python -m splunk_eventgen generate SA-Eventgen/lib/splunk_eventgen/README/eventgen.conf.tutorial1
2. Examine the data
3. Note that the timestamp is not changing as expected.

Expected behavior Timestamp on event should be changing.

Actual behavior Timestamp is not changing.

Screenshots Command line output shown below With v6.5.0:

$ python -m splunk_eventgen generate SA-Eventgen/lib/splunk_eventgen/README/eventgen.conf.tutorial1 11-13-2019 14:09:46.364461 INFO Metrics - group=mpool, max_used_interval=11259, max_used=95646, avg_rsv=251, capacity=268435456, used=0 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=fschangemanager, processor=fschangemanager, cpu_seconds=0.000000, executes=1, cumulative_hits=506 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=fschangemanager, processor=sendindex, cpu_seconds=0.000000, executes=1, cumulative_hits=506 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=http-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0.000000, executes=76, cumulative_hits=44392 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexer, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=signing, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=syslog-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=indexerpipe, processor=tcp-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=merging, processor=aggregator, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=merging, processor=readerin, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=merging, processor=sendout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=parsing, processor=header, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=parsing, processor=linebreaker, cpu_seconds=0.000000, executes=49, cumulative_hits=32921 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0.000000, executes=4, cumulative_hits=4585 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=parsing, processor=sendout, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0.000000, executes=49, cumulative_hits=32921 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=typing, processor=annotator, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=typing, processor=previewout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=typing, processor=readerin, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=typing, processor=regexreplacement, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:09:46.364461 INFO Metrics - group=pipeline, name=typing, processor=sendout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:09:46.364461 INFO Metrics - group=searchscheduler, dispatched=0, skipped=0, total_lag=0, max_ready=0, max_pending=0, max_lag=0, max_running=0, actions_triggered=0, completed=0, total_runtime=0.000, max_runtime=0.000

With v5.something:

$ /opt/splunk/bin/splunk cmd python bin/eventgen.py README/eventgen.conf.tutorial1 11-13-2019 14:31:36.005277 INFO Metrics - group=mpool, max_used_interval=11259, max_used=95646, avg_rsv=251, capacity=268435456, used=0 11-13-2019 14:31:36.020770 INFO Metrics - group=pipeline, name=fschangemanager, processor=fschangemanager, cpu_seconds=0.000000, executes=1, cumulative_hits=506 11-13-2019 14:31:36.040657 INFO Metrics - group=pipeline, name=fschangemanager, processor=sendindex, cpu_seconds=0.000000, executes=1, cumulative_hits=506 11-13-2019 14:31:36.044679 INFO Metrics - group=pipeline, name=indexerpipe, processor=http-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:31:36.103668 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:31:36.043548 INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0.000000, executes=76, cumulative_hits=44392 11-13-2019 14:31:36.096265 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:31:36.254312 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexer, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:31:36.059045 INFO Metrics - group=pipeline, name=indexerpipe, processor=signing, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:31:36.090987 INFO Metrics - group=pipeline, name=indexerpipe, processor=syslog-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:31:36.011032 INFO Metrics - group=pipeline, name=indexerpipe, processor=tcp-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-13-2019 14:31:36.121628 INFO Metrics - group=pipeline, name=merging, processor=aggregator, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:31:36.251554 INFO Metrics - group=pipeline, name=merging, processor=readerin, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:31:36.083130 INFO Metrics - group=pipeline, name=merging, processor=sendout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:31:36.017637 INFO Metrics - group=pipeline, name=parsing, processor=header, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:31:36.150238 INFO Metrics - group=pipeline, name=parsing, processor=linebreaker, cpu_seconds=0.000000, executes=49, cumulative_hits=32921 11-13-2019 14:31:36.040203 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0.000000, executes=4, cumulative_hits=4585 11-13-2019 14:31:36.174218 INFO Metrics - group=pipeline, name=parsing, processor=sendout, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-13-2019 14:31:36.138765 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0.000000, executes=49, cumulative_hits=32921 11-13-2019 14:31:36.254731 INFO Metrics - group=pipeline, name=typing, processor=annotator, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:31:36.147876 INFO Metrics - group=pipeline, name=typing, processor=previewout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:31:36.067219 INFO Metrics - group=pipeline, name=typing, processor=readerin, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:31:36.104720 INFO Metrics - group=pipeline, name=typing, processor=regexreplacement, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:31:36.043445 INFO Metrics - group=pipeline, name=typing, processor=sendout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-13-2019 14:31:36.142154 INFO Metrics - group=searchscheduler, dispatched=0, skipped=0, total_lag=0, max_ready=0, max_pending=0, max_lag=0, max_running=0, actions_triggered=0, completed=0, total_runtime=0.000, max_runtime=0.000

Sample files and eventgen.conf file the samples for the tutorial referenced above are included as part of eventgen

Do you run eventgen with SA-eventgen? yes, but this is faster to see at command line as described above. when i've tried with SA-Eventgen, it's on orca.

If you are using SA-Eventgen with Splunk (please complete the following information):

• OS: Mac 10.14.6
• Browser: firefox
• Eventgen Version: 6.5.0
• Splunk Version: 7.3.2, 8.0

If you are using eventgen with pip module mode (please complete the following information):

• python version: 2.7
• OS: Mac 10.14.6
• Virtual Env is used: No
• Eventgen Version: 6.5.0

[jmeixensperger]

The tutorial seems out of date because the conf file is using the "timestamp" token replacement type instead of "replaytimestamp". Can you change this setting for each token replacement and confirm you are getting the desired behavior?

[lsnow11]

sorry for the delay in testing this. no change in output after changing "timestamp" to "replaytimestamp":

$ python -m splunk_eventgen generate SA-Eventgen/lib/splunk_eventgen/README/eventgen.conf.tutorial1 11-20-2019 08:59:15.000000 INFO Metrics - group=mpool, max_used_interval=11259, max_used=95646, avg_rsv=251, capacity=268435456, used=0 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=fschangemanager, processor=fschangemanager, cpu_seconds=0.000000, executes=1, cumulative_hits=506 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=fschangemanager, processor=sendindex, cpu_seconds=0.000000, executes=1, cumulative_hits=506 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=http-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexin, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=index_thruput, cpu_seconds=0.000000, executes=76, cumulative_hits=44392 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexandforward, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=indexer, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=signing, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=syslog-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=indexerpipe, processor=tcp-output-generic-processor, cpu_seconds=0.000000, executes=78, cumulative_hits=46081 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=merging, processor=aggregator, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=merging, processor=readerin, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=merging, processor=sendout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=parsing, processor=header, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=parsing, processor=linebreaker, cpu_seconds=0.000000, executes=49, cumulative_hits=32921 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=parsing, processor=readerin, cpu_seconds=0.000000, executes=4, cumulative_hits=4585 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=parsing, processor=sendout, cpu_seconds=0.000000, executes=48, cumulative_hits=31355 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=parsing, processor=utf8, cpu_seconds=0.000000, executes=49, cumulative_hits=32921 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=typing, processor=annotator, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=typing, processor=previewout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=typing, processor=readerin, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=typing, processor=regexreplacement, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-20-2019 08:59:15.000000 INFO Metrics - group=pipeline, name=typing, processor=sendout, cpu_seconds=0.000000, executes=47, cumulative_hits=30025 11-20-2019 08:59:15.000000 INFO Metrics - group=searchscheduler, dispatched=0, skipped=0, total_lag=0, max_ready=0, max_pending=0, max_lag=0, max_running=0, actions_triggered=0, completed=0, total_runtime=0.000, max_runtime=0.000

[jmeixensperger]

Hmm, I'm not seeing that behavior anymore. I would let it run for a minute or 2 so that it dumps more events on a new interval. You should see new events continue to generate with the current timestamp. It's hard to tell what's going on based on your output, since it's entirely possible that 20-25 events could have the same timestamp. If you still have issues, please output to a file and attach here.

[li-wu]

Was this issue fixed? I saw a pr is already merged: https://github.com/splunk/eventgen/pull/351

[jmeixensperger]

The issue should be fixed, but I was awaiting follow-up. @lsnow11 please re-open and attach your output file if you still have issues.