search

CAIDA / pybgpstream

Python bindings for BGPStream
https://bgpstream.caida.org
BSD 2-Clause "Simplified" License
28 stars 22 forks source link

How can I use v2-beta branch with group option for load-balancing? #7

Closed a16 closed 3 years ago

a16 commented 5 years ago

Hello,

In order to test load-balancing using v2-beta, I've run my test code below(run.py) with 2 processes(let's say, p1 and p2) .

from pybgpstream.pybgpstream import BGPStream, BGPRecord, BGPElem

stream = BGPStream(data_interface="beta-bmp-stream")
stream.set_data_interface_option("beta-bmp-stream", "group", "test_hoge")
for rec in stream.records():
    for elem in rec:
        print(rec.type, rec.time, rec.collector, elem.type, elem.peer_asn, elem.peer_address)

Then, 

# start to consume and store response to test_hoge_p1.log on p1
pybgpstream-test% ./run.py > test_hoge_p1.log &
[1] 18075
# start to consume and store response to test_hoge_p2.log on p2
pybgpstream-test% ./run.py > test_hoge_p2.log &
[2] 18162
# It seems to increase the data size on p1, but receive none on p2 
pybgpstream-test% ls -al test_hoge*
-rw-r--r-- 1 watanabe watanabe 16094308 Aug 28 13:25 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe        0 Aug 28 13:25 test_hoge_p2.log
pybgpstream-test% ls -al test_hoge*
-rw-r--r-- 1 watanabe watanabe 16291169 Aug 28 13:26 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe        0 Aug 28 13:25 test_hoge_p2.log
pybgpstream-test% ls -al test_hoge*
-rw-r--r-- 1 watanabe watanabe 16323984 Aug 28 13:26 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe        0 Aug 28 13:25 test_hoge_p2.log
# Kill p1 process
pybgpstream-test% kill 18075
[1]  - terminated  ./run.py > test_hoge_p1.log
# p2 started to receive the data
pybgpstream-test% ls -al test_hoge*
-rw-r--r-- 1 watanabe watanabe 16323984 Aug 28 13:26 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe  2534570 Aug 28 13:26 test_hoge_p2.log
pybgpstream-test%

It looks to work fine as kafka group feature, but I guess it isn't load-balancing. Am I missing on my code?

If it can be load-balancing on multi session, Could you tell me how to do it.

alistairking commented 5 years ago

it can take some time (perhaps a minute or so) for the members of the group to rebalance. how long did you run the test for?

a16 commented 5 years ago

Umm... I'm running them for about 10 min. Never rebalanced...

(bgpstream) bugyo-stg /home/watanabe/pybgpstream-test% ./run.py > test_hoge_p1.log &
[1] 981
(bgpstream) bugyo-stg /home/watanabe/pybgpstream-test% ./run.py > test_hoge_p2.log &
[2] 1067
(bgpstream) bugyo-stg /home/watanabe/pybgpstream-test% date; ls -al test_hoge*
Tue Aug 28 23:51:14 JST 2018
-rw-r--r-- 1 watanabe watanabe        0 Aug 28 23:50 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe 38838828 Aug 28 23:51 test_hoge_p2.log
(bgpstream) bugyo-stg /home/watanabe/pybgpstream-test% date; ls -al test_hoge*
Tue Aug 28 23:51:16 JST 2018
-rw-r--r-- 1 watanabe watanabe        0 Aug 28 23:50 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe 46122302 Aug 28 23:51 test_hoge_p2.log
-- snip --
(bgpstream) bugyo-stg /home/watanabe/pybgpstream-test% date; ls -al test_hoge*
Wed Aug 29 00:00:02 JST 2018
-rw-r--r-- 1 watanabe watanabe          0 Aug 28 23:50 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe 2845210068 Aug 29 00:00 test_hoge_p2.log
(bgpstream) bugyo-stg /home/watanabe/pybgpstream-test% date; ls -al test_hoge*
Wed Aug 29 00:01:40 JST 2018
-rw-r--r-- 1 watanabe watanabe          0 Aug 28 23:50 test_hoge_p1.log
-rw-r--r-- 1 watanabe watanabe 3388305631 Aug 29 00:01 test_hoge_p2.log
(bgpstream) bugyo-stg /home/watanabe/pybgpstream-test%
alistairking commented 5 years ago

ok, we'll try and reproduce the problem and get back to you

a16 commented 5 years ago

FYI, I'm running it on: Debian/GNU Linux 9.2 librdkafka-dev:amd64 0.11.5-1 libbgpstream-2.0.0-beta-3.tar.gz(Install from tarball) python 3.6.6 pybgpstream(commit 259f10a5b4b8230009dfbf71be0087720f6f1795)

Thanks in advance.

alistairking commented 3 years ago

This has been fixed by #209 Feel free to re-open if you're still having problems.