Add the ability to define the topics that OpenBMP produces to.
Specific consumers may only need to consume messages from specific peers, routers, address families. Kafka consumers scale horizontally by topic/partition. Normally the number of partitions in Kafka is small. In order to scale consumers efficiently, additional topics need to be used. This will ensure FIFO message processing for messages that should be processed in order they were received by the collector (openbmpd/producer).
This enhancement is to add YAML configuration file support with the ability to define/change the existing topic names and to add grouping by router and peer. Later we will add filtering and other enhanced groupings, such as date/time, communities, as path, etc.
Below is an example configuration file. In order to ensure that the topic name is consistent we use grouping with a group name. All entries that match the group will appear in the topic by group.name substitution. This can be used to classify/group routers or peers by region/location, role (public, private, transit, internal, ...), etc.
topics:
grouping:
# {router_group} is the variable that you use for topic substitution
router_group:
# name defines the value that is substituted for the variable. This provides a consistent
# mapping for different IP's and hostnames
- name: "phil"
# You can define a list of regexp's that match for hostname to group mapping
regexp_hostname:
- .*\.phil\..* # Match
# You can also define a list of prefixes that match for ip to group mapping
prefix_range:
- 10.100.100.0/24
- 10.100.104.0/24
# {router_group} is the variable that you use for topic substitution
peer_group:
# name defines the value that is substituted for the variable. This provides a consistent
# mapping for different IP's and hostnames
- name: "phil"
# You can define a list of regexp's that match for hostname to group mapping
regexp_hostname:
- .*\.phil\..* # Match
# You can also define a list of prefixes that match for ip to group mapping
prefix_range:
- 10.100.100.0/24
- 10.100.104.0/24
mapping:
root: "openbmp" # Defines the root topic (e.g. openbmp.*)
raw: "bmp_raw" # Defines the topic for bmp raw messages (e.g. openbmp.bmp_raw.*)
parsed: "parsed" # Defines the topic for parsed messages (e.g. openbmp.parsed.*)
peer: "{root}.{parsed}.peer"
unicast_prefix: "{root}.{parsed}.unicast_prefix.{router_group}.{peer_group}.{family}"
<etc...>
TimEvens
commented
8 years ago
Add the ability to define the topics that OpenBMP produces to.
Specific consumers may only need to consume messages from specific peers, routers, address families. Kafka consumers scale horizontally by topic/partition. Normally the number of partitions in Kafka is small. In order to scale consumers efficiently, additional topics need to be used. This will ensure FIFO message processing for messages that should be processed in order they were received by the collector (openbmpd/producer).
This enhancement is to add YAML configuration file support with the ability to define/change the existing topic names and to add grouping by router and peer. Later we will add filtering and other enhanced groupings, such as date/time, communities, as path, etc.
Below is an example configuration file. In order to ensure that the topic name is consistent we use grouping with a group name. All entries that match the group will appear in the topic by group.name substitution. This can be used to classify/group routers or peers by region/location, role (public, private, transit, internal, ...), etc.