bernhardreiter
commented
6 years ago Implementation considerations
- Right now filtering by country is done early, e.g. in https://github.com/Intevation/intelmq/blob/certbund-contact/intelmq/bots/experts/certbund_contact/ripe_data.py#L112 to reduce the amount of data early to speed up processing. This would need to be changed because we will need the information for all countries to search for less specific CIDRs.
- Next we need an efficient way to find out the next wider CIDR, first idea is to build a tree data structure so that more specific CIDRs are below the wider one and we can go up the tree. This would only be the effort of building the tree, otherwise we need to run over all CIDRs to find out which is the next wider one. Needs research because probably index data structures for this already exist, probably in python modules.
- After finding the INETNUM and INET6NUM we want to import additionally to the one found by country, we need to remove the other ones.
- Code after this and diff should be fine, but we need to test this at least with a few samples.
- There are speed reasons to consider, we probably have to create a small test set of ripe_files to be able to do fast developement runs.
Problem: Some CIDRs with a specific country code do not have an abuse email address via abuse-c or the org, but a wider CIDR that covers the range has and is from a different country.
Right now the wider CIDR is not imported because it is from a different country. So we do not find an abuse contact for the CIDR.
Technical solution idea
If no abuse mail-address is found in the CIDR for a country, look into wider CIDRs covering the range until one with an abuse address is found. The first one that is found, will be imported.