jeffsw / rpkilog

rpkilog.com web site and tools
MIT License
6 stars 0 forks source link

Additional RPKI data providers and related alterations to data pipeline and schema #6

Open jeffsw opened 2 years ago

jeffsw commented 2 years ago

We're going to want older RPKI data sometime soon. Our data pipeline will need re-factoring to keep track of separate data sources which are independent of each-other when calculating diffs.

I think RIPE's daily roas.csv files make the most sense to consume next. They're separated by TAL. We'll want to do diff comparisons on a per-TAL basis when working with these files. Ultimately, what this means is, we need to have multiple different JSON Summary File Series which feed into the diff process -- one for each RIR (think about the APNIC multiple TALs situation prior to early-2018 later).

Our database schema already references the TAL. It might be nice if individual records indicated which summary file supplied the data, for both old and new ROAs.

This is probably less important & urgent than making current data available to the public / networking community in the short-term.