Python exception thrown when running bench

[dotwaffle]

I've just installed bgperf, as described in your instructions.

21:18:49 [dotwaffle@hasselhoff:~/Code/bgperf] master ± sudo ./bgperf.py bench
creating Docker network "bgperf-br" with subnet 10.10.0.0/16
run monitor
run gobgp
waiting bgp connection between gobgp and monitor
run tester tester type normal
tester booting.. (100/100)
elapsed: 0sec, cpu: 0.00%, mem: 0.00B, recved: 9900
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/srv/home/dotwaffle/Code/bgperf/base.py", line 179, in stats
    queue.put({'who': self.name, 'cpu': cpu_percentage, 'mem': stat['memory_stats']['usage']})
KeyError: 'usage'
elapsed: 2sec, cpu: 0.00%, mem: 0.00B, recved: 10000
21:20:20 [dotwaffle@hasselhoff:~/Code/bgperf] master ±

Any ideas? I made sure to run pip --upgrade -r pip-requirements.txt in case that helps... It didn't. Also happens using -t bird.

System is Debian Stable.

[ishidawataru]

could you show the output of ./bgperf.py doctor ?

[dotwaffle]

Sure!

21:18:11 [dotwaffle@hasselhoff:~/Code/bgperf] master ± ./bgperf.py doctor
docker version ... ok (17.05.0-ce)
bgperf image ... ok
gobgp image ... ok
bird image ... ok
quagga image ... ok
/proc/sys/net/ipv4/neigh/default/gc_thresh3 ... 1024

[ishidawataru]

thanks, I'll try to reproduce the issue with the same docker version you are using.

[dotwaffle]

Appreciated, thank you!

[ishidawataru]

I tried with docker 17.05.0-ce but couldn't reproduce the issue. Does this happen everytime or occasionally?

[dotwaffle]

Every time. I'm assuming this is more likely a python issue?

[ishidawataru]

yes, it seems so. Could you add print stat like below and see what the stat contains?

# base.py line 179
print stat
queue.put({'who': self.name, 'cpu': cpu_percentage, 'mem': stat['memory_stats']['usage']})

[dotwaffle]

15:42:59 [dotwaffle@hasselhoff:~/Code/bgperf] master(+1/-0) 1 ± sudo ./bgperf.py bench
creating Docker network "bgperf-br" with subnet 10.10.0.0/16
run monitor
run gobgp
waiting bgp connection between gobgp and monitor
run tester tester type normal
tester booting.. (100/100)
elapsed: 0sec, cpu: 0.00%, mem: 0.00B, recved: 9900
{u'blkio_stats': {u'io_service_time_recursive': [], u'sectors_recursive': [], u'io_service_bytes_recursive': [], u'io_serviced_recursive': [], u'io_time_recursive': [], u'io_queue_recursive': [], u'io_merged_recursive': [], u'io_wait_time_recursive': []}, u'precpu_stats': {u'cpu_usage': {u'total_usage': 0, u'usage_in_kernelmode': 0, u'usage_in_usermode': 0}, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}, u'name': u'/bgperf_gobgp_target', u'read': u'2017-06-15T15:44:00.155074129Z', u'storage_stats': {}, u'networks': {u'eth1': {u'tx_dropped': 0, u'rx_packets': 27, u'rx_bytes': 2285, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 0, u'rx_dropped': 0, u'tx_packets': 0}, u'eth0': {u'tx_dropped': 0, u'rx_packets': 8467, u'rx_bytes': 632884, u'tx_errors': 0, u'rx_errors': 0, u'tx_bytes': 6148219, u'rx_dropped': 0, u'tx_packets': 10928}}, u'num_procs': 0, u'preread': u'0001-01-01T00:00:00Z', u'memory_stats': {}, u'pids_stats': {}, u'id': u'22a9c2cfb0a9a224c465a7a913774e46036e942366ce767cecb07d218156e516', u'cpu_stats': {u'cpu_usage': {u'usage_in_usermode': 6770000000, u'total_usage': 7265281488, u'percpu_usage': [994928057, 945538940, 1159216139, 968660265, 857438607, 781201737, 795383303, 762914440], u'usage_in_kernelmode': 300000000}, u'system_cpu_usage': 2071616930000000, u'online_cpus': 8, u'throttling_data': {u'throttled_time': 0, u'periods': 0, u'throttled_periods': 0}}}
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/srv/home/dotwaffle/Code/bgperf/base.py", line 180, in stats
    queue.put({'who': self.name, 'cpu': cpu_percentage, 'mem': stat['memory_stats']['usage']})
KeyError: 'usage'
elapsed: 2sec, cpu: 0.00%, mem: 0.00B, recved: 10000
15:44:02 [dotwaffle@hasselhoff:~/Code/bgperf] master(+1/-0) ±

[ishidawataru]

thanks.

Could you try the latest master branch?

Should be fixed by https://github.com/osrg/bgperf/pull/21

[dotwaffle]

Well, it proceeded further than last time, but doesn't seem to be giving good results!

21:20:58 [dotwaffle@hasselhoff:~/Code/bgperf] master ± sudo ./bgperf.py bench
removing target container bgperf_bird_target
removing monitor container bgperf_monitor
removing tester container bgperf_exabgp_tester_tester
Docker network "bgperf-br" already exists
run monitor
run gobgp
waiting bgp connection between gobgp and monitor
run tester tester type normal
tester booting.. (100/100)
elapsed: 2sec, cpu: 0.00%, mem: 0.00B, recved: 10000
21:22:02 [dotwaffle@hasselhoff:~/Code/bgperf] master ± sudo ./bgperf.py bench -t quagga
removing target container bgperf_gobgp_target
removing monitor container bgperf_monitor
removing tester container bgperf_exabgp_tester_tester
Docker network "bgperf-br" already exists
run monitor
run quagga
waiting bgp connection between quagga and monitor
run tester tester type normal
tester booting.. (100/100)
elapsed: 3sec, cpu: 0.19%, mem: 0.00B, recved: 10000
21:23:06 [dotwaffle@hasselhoff:~/Code/bgperf] master ±

[ishidawataru]

After the benchmark, try docker stats. Does it show memory info?

[dotwaffle]

CONTAINER           CPU %               MEM USAGE / LIMIT   MEM %               NET I/O             BLOCK I/O           PIDS
7f2771319bfb        22.65%              0B / 0B             0.00%               6.11MB / 639kB      0B / 0B             0
d30c3f482ae1        0.02%               0B / 0B             0.00%               658kB / 6.18MB      0B / 0B             0
8d0cd73f5b38        0.00%               0B / 0B             0.00%               99.2kB / 7.35kB     0B / 0B             0
^C%                                                                                                                                                                                                                                23:21:40 ⌂114% [dotwaffle@hasselhoff:~/Code/bgperf] master 130 ± docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
7f2771319bfb        bgperf/exabgp       "bash"              About a minute ago   Up About a minute                       bgperf_exabgp_tester_tester
d30c3f482ae1        bgperf/gobgp        "bash"              About a minute ago   Up About a minute                       bgperf_gobgp_target
8d0cd73f5b38        bgperf/gobgp        "bash"              About a minute ago   Up About a minute                       bgperf_monitor
23:21:44 ⌂114% [dotwaffle@hasselhoff:~/Code/bgperf] master ±

Well, that's curious!

[dotwaffle]

I've just followed http://awhitehatter.me/debian-jessie-wdocker/ and re-run:

23:27:42 [dotwaffle@hasselhoff:~/Code/bgperf] master ± ./bgperf.py bench
creating Docker network "bgperf-br" with subnet 10.10.0.0/16
run monitor
run gobgp
waiting bgp connection between gobgp and monitor
run tester tester type normal
tester booting.. (100/100)
elapsed: 2sec, cpu: 0.00%, mem: 30.16MB, recved: 10000
23:28:54 ⌂69% [dotwaffle@hasselhoff:~/Code/bgperf] master ± ./bgperf.py bench -t quagga
removing target container bgperf_gobgp_target
removing monitor container bgperf_monitor
removing tester container bgperf_exabgp_tester_tester
Docker network "bgperf-br" already exists
run monitor
run quagga
waiting bgp connection between quagga and monitor
run tester tester type normal
tester booting.. (100/100)
elapsed: 4sec, cpu: 0.13%, mem: 422.40MB, recved: 10000
23:30:09 ⌂141% [dotwaffle@hasselhoff:~/Code/bgperf] master ± ./bgperf.py bench -t bird
removing target container bgperf_quagga_target
removing monitor container bgperf_monitor
removing tester container bgperf_exabgp_tester_tester
Docker network "bgperf-br" already exists
run monitor
run bird
waiting bgp connection between bird and monitor
run tester tester type normal
tester booting.. (100/100)
elapsed: 2sec, cpu: 0.00%, mem: 134.98MB, recved: 10000
23:31:10 ⌂130% [dotwaffle@hasselhoff:~/Code/bgperf] master ±

[ishidawataru]

nice! Thanks for the report. I'm closing this one.

[dotwaffle]

Thank you, @ishidawataru -- it means a lot that you'd help me diagnose this to that extent!

Best of luck with the project!