search

rucio / rucio

Rucio - Scientific Data Management
http://rucio.cern.ch
Apache License 2.0
247 stars 314 forks source link

GEOIP Maxmindb daily download limit reached #5077

Closed bari12 closed 2 years ago

bari12 commented 2 years ago

Motivation

On the ATLAS Rucio cluster(s) we reached the daily download limit of the maxmind geoip database. (2000 downloads / day)

In principle https://github.com/rucio/rucio/blob/631e5d5daa81ab86f310fd046350e8212b91e1eb/lib/rucio/core/replica_sorter.py#L82-L91 is already quite conservative (1 refresh every 30 days) so it seems like we reach this limit simply based on the number of pods + restarts. (Still a bit surprised that we are hitting 2000 in a day)

Modification

Best is probably to cache this, somewhere internally or in the Kube cluster. The db needs to be refreshed from time to time though.

bari12 commented 2 years ago

Perhaps this is just an artefact due to playing around with cluster3?

bari12 commented 2 years ago

After hotfixing this on the cluster, we decided to add a new config parameter to just use an url instead of the hardcoded url + licence_key https://github.com/rucio/rucio/blob/12b56095d7b0763a3a3ac257bd5ae9ff9b8e2d40/lib/rucio/core/replica_sorter.py#L58

[replicas]
geoip_database_url =

and remove the [core] licence_key entry. People who still want to use maxmindb directly can pass that url + key there as well.