search

sbng / mrt2mmdb

Convert a mrt file into a maxmind database file
1 stars 0 forks source link

Add additional ASN enrichment source - flat file #5

Open johnhtodd opened 3 months ago

johnhtodd commented 3 months ago

The MaxMind tool is very good for adding ASN data, but I'd like to make it a bit more complete, and give people the ability to customize their own ASN data easily. By specifying an additional file with data, it should be possible for the user to generate a full map of all BGP prefixes that appear in the MRT file and create custom names assigned to an ASN without having to re-create the MMDB format of the MaxMind hints file.

New ASN data source model:

1) tab-separated OR comma-separated - it should take both (filename can be the hint - csv, tsv)

2) the MaxMind data can be used in addition to the CSV/TSV file, but the CSV/TSV takes precedence if there are overlaps

3) The file should be optional with a command line flag of some type that specifies the name. It should be the case that the MaxMind file can be omitted if there is a CSV/TSV specified, as some users will opt only for their own map data.

4) Overlaps with hint MMDB file should be printed in a debug output

5) Perhaps the debug should say which of the two sources of AS data were used for creation?

Columns: asn,rir,org_name

RIR can be ignored.

I will send you the file via email to start.

sbng commented 3 months ago

feature added. commit 87e1620 under flat_file branch. Please verify

johnhtodd commented 3 months ago

Haven't yet verified - sorry, out of time. Will do soon.

sbng commented 3 months ago

FYI. I added to new feature difference.py to check the differences between mmdb or mmdb/csv/tsv files. All output in json so u can parse it via jq. Currently code is under differences branch. If you test and ok with it, I will then merge it to main.

On Thu, Mar 14, 2024 at 7:32 AM John Todd @.***> wrote:

The MaxMind tool is very good for adding ASN data, but I'd like to make it a bit more complete, and give people the ability to customize their own ASN data easily. By specifying an additional file with data, it should be possible for the user to generate a full map of all BGP prefixes that appear in the MRT file and create custom names assigned to an ASN without having to re-create the MMDB format of the MaxMind hints file.

New ASN data source model:

1.

tab-separated OR comma-separated - it should take both (filename can be the hint - csv, tsv) 2.

the MaxMind data can be used in addition to the CSV/TSV file, but the CSV/TSV takes precedence if there are overlaps 3.

The file should be optional with a command line flag of some type that specifies the name. It should be the case that the MaxMind file can be omitted if there is a CSV/TSV specified, as some users will opt only for their own map data. 4.

Overlaps with hint MMDB file should be printed in a debug output 5.

Perhaps the debug should say which of the two sources of AS data were used for creation?

Columns: asn,rir,org_name

RIR can be ignored.

I will send you the file via email to start.

— Reply to this email directly, view it on GitHub https://github.com/sbng/mrt2mmdb/issues/5, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB25MFZCC7PVDS6SABVPRF3YYDOZ5AVCNFSM6AAAAABEVCOISWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE4DKMBZGQZDIMQ . You are receiving this because you are subscribed to this thread.Message ID: @.***>