CAIDA / pybgpstream

Python bindings for BGPStream
https://bgpstream.caida.org
BSD 2-Clause "Simplified" License
28 stars 22 forks source link

Update bgpstream_filter_parser to support terms appearing more than once. #28

Open csw1995 opened 4 years ago

csw1995 commented 4 years ago

Hi CAIDA team,

Recently, I start to study on the BGP and find that BGPStream is just what I need. Because retrieving data from collectors is time-consuming, I was going to filter all the prefixes I need in one pass using multiple conditions in the filter string. But some errors were raised in my first attempt.

After reading the documents and source code, I find multiple conditions are supported for those terms with * in the BGPReader document. And stream.add_filter() in pybgpstream achieves similar results.

So may I ask is there any plan to update bgpstream_filter_parser to support terms appearing more than once? Do I miss somewhere in the documents that indicate this inconsistency? Thank you very much!

-Siwei


Here is how I reproduce the issue. macOS 10.15.4 Python 3.7.4 libbgpstream 2.0.0-rc4 pybgpstream 2.0.0

Using the same term mutiple times in the filter string will raise error when parsing.

stream = pybgpstream.BGPStream(
   from_time="2017-07-07 00:00:00", until_time="2017-07-07 00:10:00 UTC",
   collectors=["route-views.sg", "route-views.eqix"],
   record_type="updates",
   filter="prefix more 210.118.0.0/16 and prefix more 210.180.0.0/16"
)

for elem in stream:
    print(elem)
2020-04-28 19:17:30 2005: bgpstream_filter_parser.c:165: ERROR: Term 'prefix' used more than once
...
ValueError: Invalid filter string: prefix more 210.118.0.0/16 and prefix more 210.180.0.0/16

But add_filter() does support multiple filters on the same term.

stream = pybgpstream.BGPStream(
   from_time="2017-07-07 00:00:00", until_time="2017-07-07 00:10:00 UTC",
   collectors=["route-views.sg", "route-views.eqix"],
   record_type="updates",
)

stream.add_filter("prefix-more", "210.118.0.0/16")
stream.add_filter("prefix-more", "210.180.0.0/16")

for elem in stream:
    print(elem)
...
update|A|1499385796.000000|routeviews|route-views.eqix|None|None|6830|206.126.236.104|210.180.96.0/19|206.126.236.12|6830 2914 9318|6830:16000 6830:35304 2914:2000 2914:1005 2914:3275 2914:3000 6830:16232 2914:3075 2914:410|None|None
update|A|1499385821.000000|routeviews|route-views.eqix|None|None|4589|206.126.236.10|210.118.3.0/24|206.126.236.10|4589 3786 4663 4663|4589:680 4589:685 4589:61 4589:10140 4589:2 4589:420 3786:11|None|None
...