search

hep-gc / shoal

A squid cache publishing and advertising tool designed to work in fast changing environments
Apache License 2.0
4 stars 8 forks source link

City DB format #143

Closed MarcusEbert closed 3 years ago

MarcusEbert commented 5 years ago

For the city db, the old .dat format is still used while for the domain db the new mmdb one is used. Could the implementation be changed to use the new mmdb format for both databases?

The old .dat format also doesn't have all data in there (DESY squids are not correctly mapped).

DrDaveD commented 5 years ago

And in fact the old .dat format is no longer supported and can't be downloaded anymore. An update to the GeoLite2 .mmdb format is required.

The cvmfs-server package has code for downloading and keeping up to date a copy of GeoLite2-City.mmdb. For the wlcg-wpad package I just depended on that.

DrDaveD commented 5 years ago

And this is now blocking me so unless you say you're working on this or have already done it, I'll develop a PR for it.

MarcusEbert commented 5 years ago

We found that some sites where not correctly mapped to a location, e.g DESY. It was the same in the mmdb as well as dat files we tested. Solution for us so far was to use the CVS version which can be edited easily (e.g. automated), and then convert to dat, which also allows to keep the old interface so far. @colsond would know what the status of the native mmdb implementation is.

DrDaveD commented 5 years ago

I have not noticed any differences between the GeoLite2-City .csv and .mmdb files. If you can give me a specific example I would like to know about it. As far as I understand it they generate the two formats from the exact same data.

I have had very good luck with submitting change requests to maxmind through the standard web form. They never reply and say they're going to make the change, but when I check a few weeks later the changes are there.

MarcusEbert commented 5 years ago

That is correct that all formats that can be downloaded have the same content. However, adding/fixing entries using the cs files is easy and can be automated too. And if using the csv anyway, we can convert to dat and still use that format. But I agree that shoal currently uses a mix of mmdb and dat and that should be consistently be mmdb.

When DESY registered with shoal, their IP range was missing in the db and no location mapped; and it seems it still seems to be the case (a year later).That was the initial reason to switch to csv and converting to a usable format after editing.

DrDaveD commented 5 years ago

If you know what the correction for DESY is, please submit it to maxmind. I have so far been able to avoid having any mechanism for applying local corrections because of that.

MarcusEbert commented 5 years ago

The new mmdb format is used now. We test the new version, updated to CentOS7, and will make it available then.

MarcusEbert commented 3 years ago

issue solved with the new shoal version