search

CAIDA / pybgpstream

Python bindings for BGPStream
https://bgpstream.caida.org
BSD 2-Clause "Simplified" License
29 stars 23 forks source link

MRT Update Dump Vs Kafka Live Stream #49

Open ACodingfreak opened 1 year ago

ACodingfreak commented 1 year ago

Hi All,

Thanks to @alistairking i am able to bringup pybgpstream over kafka stream "stream.routeviews.org:9092" and "topic": "routeviews.linx.6830.bmp_raw".

I observed one strange behavior where none of the BGP-UPDATES in KAFKA-STREAM contain multiple prefixes under ANNOUNCEMENTS and WITHDRAWLS messages. This is quite strange as BGP can send single WITHDRAWL message with multiple prefixes.

U|W|1674087346.913545|routeviews-stream|is-ah-bmp1|fortaleza|189.90.173.248|268011|2001:12f8:0:9::145:237|2a0e:b105:6af::/48||||||
U|W|1674087346.913753|routeviews-stream|is-ah-bmp1|fortaleza|189.90.173.248|268011|2001:12f8:0:9::145:237|2405:3740::/32||||||

As you can see in above example, i can see separate WITHDRAWLs containing different prefixes. I am not sure if this is an expected behavior ? If YES, then there is a difference between MRT update dumps Vs accessing the live stream over Kafka Stream.

alistairking commented 1 year ago

Can you provide an example of what these updates look like in the corresponding MRT file?

The timestamps may be different between BMP and MRT but the content of the updates should be identical.

ACodingfreak commented 1 year ago

@alistairking .. let me work on it and share results ...

Since I am only picking BGP updates with ASN 6830 when compared to MRT files it is taking some time to share the comparison.

Is there a way to dump live stream into a MRT file in a sequential order based on timestamp ? Currently looks like if I try topic as "routeviews.linx.*.bmp_raw" it tries to get all BGP updates for LINX but as routeview adds BGP updates across different Kafka partitions these entries are getting scattered across and creating non-sequential flow.

alistairking commented 1 year ago

Oh maybe I misunderstood then. How do you know that these updates are different to MRT if you don't have the corresponding updates from the MRT yet?