jszwedko
commented
7 months ago Thanks for the detailed request @johnhtodd ! I'm not terribly familiar with the mmdb format, but my quick investigation of the format seems to validate what you've written: records can have arbitrary fields. I think the difficulty would be that it appears that the names of the fields aren't embedded the database structure itself, from what I can see, so I think this would need to be provided to Vector. All that the database has to indicate the structure of each record is the the database_type field. I agree that this is different enough from the geoip lookups that it could be a different enrichment table type (mmdb).
The mmdb reader we use has the geoip database record structures "hardcoded" for each GeoIP database type (city, ISP, etc.): https://github.com/oschwald/maxminddb-rust/blob/9045bfe1ac1fa3e7a29258692a03ea6ab2e01069/src/maxminddb/geoip2.rs#L5-L90. That's how we can pull out the fields with names like autonomous_system_number.
Additionally: there was discussion of the use of "rayon" to increase performance in the original issue (https://github.com/vectordotdev/vector/issues/847) - was that used? It was more than 3x improvement in speed. Our reason for looking at custom MMDB files is to summarize multiple lookups into one, or at most two lookups for the sake of speed, and anything else to make MMDB lookups faster would be appreciated.
The benchmark mentioned in that issue was just demonstrating that executing lookups in parallel with rayon is faster than serial lookups. This actually should already be happening by virtue of Vector executing remap transforms concurrently.
johnhtodd
esensar
A note for the community
Use Cases
The current methodology around using GeoIP lookups with MMDB files seems to be quite limited in capability, and does not allow for custom files which contain field names that are not hard-coded in Vector source code. We perform several MMDB lookups on our pipeline (three in some cases) and having this decreased to just one lookup event would be ideal. To do that, we would need to "merge" the two MMDB files with each other, and then Vector would need to be flexible enough to return an arbitrary set of fields with the single lookup. Also, we find that the existing MMDB lookups do not return a full set of data when that would be sometimes useful.
The existing MMDB lookup method is quite inflexible, and forces specific choices of what fields are returned, and even looks at file names to make assumptions about what kind of fields should be sent back in the response. This does not allow us to extend existing MMDB tables with more information without really ugly overloading of field names, and even that is not workable past a certain point.
Attempted Solutions
Tried overloading field names; this is awful and has a limited ramp as more complexity encroaches. Also, there is no other way to get other fields without code changes.
Proposal
We would propose a much more generic method of doing MMDB lookups and data returns, which uses the format of the MMDB file to define the data structure handed back. This seems like it would be less work than the existing method, so I'm not sure what was the intent of the original effort that resulted in the way it works today. There are field names implicit within the MMDB file, and this could simply be exported as a Vector "default" object set into memory. This would be much more flexible by allowing arbitrary database fields to be named, and then the management and parsing of those fields would be done within Vector as all fields are currently managed.
Here would be an example future configuration idea:
So a lookup for (as an example) 128.151.224.17 would result in an object from the MMDB lookup in the GeoIP2-City.mmdb file inserted into the .GeoData object that resembles this:
(Note: a developer of ours wrote some python code to produce this output directly from MMDB files - https://github.com/sbng/mrt2mmdb/ - use the "lookup.py" script like this: "python3 lookup.py --mmdb /etc/vector/maxmind/GeoIP2-City_20231117/GeoIP2-City.mmdb --ipaddress 128.151.224.17") - you may need to alter single quotes to double quotes. You an also use mmdbctl (https://github.com/ipinfo/mmdbctl) for queries.)
This would be a breaking change, and is different enough to the old method that it would seem to warrant a new enrichment method entirely. I proposed "geoip2" but perhaps "mmdb" needs to start appearing in the name to be less ambiguous.
References
Additionally: there was discussion of the use of "rayon" to increase performance in the original issue (https://github.com/vectordotdev/vector/issues/847) - was that used? It was more than 3x improvement in speed. Our reason for looking at custom MMDB files is to summarize multiple lookups into one, or at most two lookups for the sake of speed, and anything else to make MMDB lookups faster would be appreciated.
Historical: Original PR: https://github.com/vectordotdev/vector/pull/1015 Further work: https://github.com/vectordotdev/vector/issues/1372
Version
vector 0.37.0 (x86_64-unknown-linux-gnu e2d8ad4 2024-03-02 04:01:47.120115067)