Closed sthen closed 3 years ago
Hmm, I've had a report of something that kinda sounds similar on OSX. Maybe they are related? Could you please try the latest git version? That adds a bit of extra debugging, which may help me correlate the two issues.
Also, something seems to be off with that line number in the last trace (there is no line 808 in aggregators.py) - any chance you can figure out the real place it's spinning?
How much memory is it allocating before it crashes / before you kill it, and how long does it spin for?
It spins for as long as I let it run or until it bumps into ulimit, RES grows at about 16M per 10 seconds.
There's not much difference in output with git head, apart from outputting Executing lines when gathering information at startup the only differences in output are that it doesn't print "[New thread 395121]" after "NetperfDemoRunner TCP download::4 finished" and timers expire in a different order, here's the diff
NetperfDemoRunner TCP download::1 finished
NetperfDemoRunner TCP download::3 finished
NetperfDemoRunner TCP download::4 finished
-[New thread 395121]
PingRunner Ping (ms) ICMP finished
Runner aggregation finished
TimerRunner Watchdog [Ping (ms) ICMP]: timer expired
TimerRunner Watchdog [Ping (ms) ICMP] finished
TimerRunner Watchdog [TCP download::2]: timer expired
TimerRunner Watchdog [TCP download::2] finished
-TimerRunner Watchdog [TCP download::1]: timer expired
-TimerRunner Watchdog [TCP download::4]: timer expired
-TimerRunner Watchdog [TCP download::1] finished
TimerRunner Watchdog [TCP download::3]: timer expired
-TimerRunner Watchdog [TCP download::4] finished
TimerRunner Watchdog [TCP download::3] finished
+TimerRunner Watchdog [TCP download::4]: timer expired
+TimerRunner Watchdog [TCP download::4] finished
+TimerRunner Watchdog [TCP download::1]: timer expired
+TimerRunner Watchdog [TCP download::1] finished
I'll add some prints in aggregators.py to try to figure out where it's spinning and get back to you.
Oh, well the python 2 trace gives a pretty good idea where it's spinning actually. Printing t_0/t_max/steps before the loop, something is clearly wrong with t_0...
t_0 581513.01849 t_max 1615131003.179 steps 8072747451
"the loop" = "for s in range(steps)" in TimeseriesAggregator aggregate().
...so, fping is buggy. If SO_TIMESTAMPNS isn't available on the OS then it tries to fetch time from CLOCK_MONOTONIC and use it as realtime for -D.
$ fping -D -c1 127.0.0.1 2>/dev/null
[594271.59597] 127.0.0.1 : [0], 64 bytes, 0.079 ms (0.079 avg, 0% loss)
Having repaired that, flent is happy again. Hopefully this is the same issue on OSX.
Reported here: https://github.com/schweikert/fping/issues/216
Ah, nice find, thanks! Still, Flent should probably detect this and bail - going to reopen this issue to track that...
Hi. flent test runs aren't completing on OpenBSD. I've tried with 1.2.2 on python 2.7.18 and 3.8.8, and with 2.0.0 on 3.8.8 and a few test types. netperf and fping are spawned, tests run, then at the end of the run one of the following happens:
python 2.7.x:
python 3.x: flent process loops consuming cpu and continually allocating memory. If I run under gdb and point py-bt at it I get a similar trace to 2.7's after the MemoryError.
Do you have any ideas as to what might be happening please? The netperf and fping processes definitely exit before the "TimerRunner Watchdog [xxx]: timer expired" messages are displayed (confirmed a few times while running pstree at various points). I'm not all that proficient in python but maybe able to poke at some things if you have any ideas of what to try.
Flent has worked on OpenBSD in the past but it's been a couple of years since I previously tried it, so couldn't say when this started. I'm running recent OpenBSD -current on amd64, netperf is v2.7.0 and fping is v5.0.