search

CAIDA / pybgpstream

Python bindings for BGPStream
https://bgpstream.caida.org
BSD 2-Clause "Simplified" License
29 stars 23 forks source link

Reading BMP messages from Kafka stream #48

Closed ACodingfreak closed 1 year ago

ACodingfreak commented 1 year ago

Hi All,

As per my understanding bgpreader provides data_interface option to read from a particular topic in kafka stream and dump output.

While using pybgpstream, I can see data_interface option where I can provide Kafka, but I am not sure how I can share kafka topic and filter for the same? Any ideas ?

alistairking commented 1 year ago

Hey there, Yeah, you should be able to use set_data_interface_option (https://bgpstream.caida.org/docs/api/pybgpstream/_pybgpstream.html#_pybgpstream.BGPStream.set_data_interface_option) And pass the topic option (https://github.com/CAIDA/libbgpstream/blob/master/lib/datainterfaces/bsdi_kafka.c#L69-L119)

ACodingfreak commented 1 year ago

Thanks for the reply alistairking that worked.

I have one more question. When working with MRT dumps for BGP updates I can see all the updates from a collector in time sequential order but when I process it on Kafka stream, only bgp updates from a particular peer is Time sequential order but not from a single collector. Is this expected behavior ?

alistairking commented 1 year ago

Yeah. The topics in the RouteViews BMP stream are partitioned based on peer ASN.

ACodingfreak commented 1 year ago

Thanks for the info @alistairking

Where can I get more details regarding the routeviews bmp kafka stream partitions and topics supported Currently I am only aware of the kafka stream as "stream.routeviews.org:9092" and topics as "routeviews.linx.6830.bmp_raw" taken from an example in BGPKIT.

If there are any other BMP KAFKA streams which can be used for live processing of BGP updates from RRC or routeviews please do share the same.

alistairking commented 1 year ago

the RV topics are all structured as routeviews.<collector>.<peer-asn>.bmp_raw. you can get a sense of the collectors and peer ASNs here: http://www.routeviews.org/peers/peering-status.html (most though not all are available via kafka).

afaik there is not currently a kafka stream for RIS data.

ACodingfreak commented 1 year ago

Thanks for the info @alistairking