BGPKIT Broker Backend is the backend API and data collection implementation for the [[https://bgpkit.com/broker][BGPKIT Broker project]].
Users can use this repository to deploy a BGPKIT Broker instance on their own equipment, and it is free under MIT license! See more in [[#deployment][Deployment]] and [[#license][LICENSE]] sections.
Table of Content :TOC:
[[#bgpkit-broker-api][BGPKIT Broker API]]
[[#data-sources][Data Sources]]
[[#deployment][Deployment]]
[[#license][LICENSE]]
BGPKIT Broker API :PROPERTIES: :ID: 279d0e5a-8de8-4906-9b45-71669a00469c :END:
At a high level, the BGPKIT Broker is a REST API service that provides search capability for BGP MRT archive files from various data collection projects. An user can search for matching archive files with simple REST API calls.
For example, we want to learn about exactly what BGP updates were involved that caused Facebook’s outage for multiple hours on Oct. 4th, 2021. We can use the following REST call to the BGPKIT Broker API to gather the full list of data files that may contain the relevant updates for the event:
{ "count": 5, "page": 1, "page_size": 5, "error": null, "data": [ { "ts_start": "2021-10-04T15:30:00+00:00", "ts_end": "2021-10-04T15:45:00+00:00", "collector_id": "route-views2", "data_type": "update", "url": "http://archive.routeviews.org/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2", "rough_size": 1153433, "exact_size": 0 }, { "ts_start": "2021-10-04T15:30:00+00:00", "ts_end": "2021-10-04T15:45:00+00:00", "collector_id": "route-views3", "data_type": "update", "url": "http://archive.routeviews.org/route-views3/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2", "rough_size": 8388608, "exact_size": 0 }, { "ts_start": "2021-10-04T15:30:00+00:00", "ts_end": "2021-10-04T15:45:00+00:00", "collector_id": "route-views5", "data_type": "update", "url": "http://archive.routeviews.org/route-views5/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2", "rough_size": 22528, "exact_size": 0 }, { "ts_start": "2021-10-04T15:30:00+00:00", "ts_end": "2021-10-04T15:45:00+00:00", "collector_id": "route-views.amsix", "data_type": "update", "url": "http://archive.routeviews.org/route-views.amsix/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2", "rough_size": 7864320, "exact_size": 0 }, { "ts_start": "2021-10-04T15:30:00+00:00", "ts_end": "2021-10-04T15:45:00+00:00", "collector_id": "route-views.chicago", "data_type": "update", "url": "http://archive.routeviews.org/route-views.chicago/bgpdata/2021.10/UPDATES/updates.20211004.0830.bz2", "rough_size": 1887436, "exact_size": 0 } ] } // GET https://api.broker.bgpkit.com/v2/search?ts_start=1633362000&ts_end=1633363200&data_type=update&page_size=5&page=1 // HTTP/1.1 200 OK // Content-Length: 1395 // Content-Type: application/json // Date: Mon, 25 Apr 2022 21:54:29 GMT // Server: Caddy // Server: uvicorn // Request duration: 1.434210s
Here is a break-down of the call parameters:
For more examples and API spec, please checkout our [[https://api.broker.bgpkit.com/v2/docs][documentation site]].
Broker backend currently support two data collection projects: [[http://archive.routeviews.org][RouteViews]] and [[https://www.ripe.net/analyse/internet-measurements/routing-information-service-ris/ris-raw-data][RIPE RIS]]. Specifically, we have designed tailored data collection procedures to efficiently crawl the data publication sites from the two projects and collect basic information of the published BGP MRT files.
The information we currently collect for each data file includes:
Note that the data dump frequencies for RouteViews and RIPE RIS are different. Here is a table summarizing their frequencies:
|------------+--------------------+----------------------| | Project | Update frequency | Table dump frequency | |------------+--------------------+----------------------| | RouteViews | one per 15 minutes | one per 2 hours | | RIPE RIS | one per 5 minutes | one per 8 hours | |------------+--------------------+----------------------|
In essence, you would see more frequent BGP updates dumps from RIPE RIS and more frequent full table dumps from RouteViews.
Here is a step-by-step guide for deploying BGPKIT Broker with Docker
The initial database bootstrap phase would take about 30 minutes to one hour depending on your deployment environment. It requires at least 30GB of free storage for the container and build processes.
After the initial bootstrap phase is done, the API service should be up and running, currently hosted at port ~18888~. You can modify the port in ~docker-compose.yml~ file. In the mean time, a cronjob service also started, crawling collectors for recent data every 5 minutes. The frequency can be configured in the ~update.cron~. It is not recommended to go more frequent than one crawl per 5 minutes.
You can check out if the API is running by running:
curl 0.0.0.0:18888/latest curl "0.0.0.0:18888/search?ts_start=1643673600&ts_end=2022-02-02&collector_id=rrc00"
BGPKIT Broker backend code is under [[file:LICENSE][MIT LICENSE]].
BGPKIT is a small-team focuses on building the best open-source tooling for BGP data processing and analysis in Rust. Learn more about what we are offering at https://github.com/bgpkit.
[[https://spaces.bgpkit.org/assets/logos/wide-solid-200px.png]]