huge memory usage with big geoip map

[aderumier]

Hi, I have an acl with a geoip map

http-request set-header X-Country-Code %[src,map_ip(/etc/haproxy/geoip/geoip.lst)]

ls -lah /etc/haproxy/geoip/geoip.lst -rw-r--r-- 1 root root 29M juil. 23 14:55 /etc/haproxy/geoip/geoip.lst

cat /etc/haproxy/geoip/geoip.lst|wc -l 1413857

sample of content from geoip.list

::200:0/108 FR
::210:0/118 FR
::210:400/119 FR
::210:75b5/128 FR
...

memory usage is around 2x400mb

root     29819 32.2  5.0 417148 409336 ?       Ss   12:56   0:01 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock
haproxy  29825  0.0  4.6 1523260 381192 ?      Sl   12:56   0:00 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -S /run/haproxy-master.sock

Not sure if it's a bug, or if it could be reduce ?

[wtarreau]

The maps take a significant amount of memory because they are hot-updatable. They keep a reference to the original format so that when you update one entry, it's immediately propagated through all usages (even if used under different formats).

We thought about ways to significantly reduce the maps' memory usage in the past (typically divide in half) but this would imply the loss of ability to hot-update them. We figured that it was not really an acceptable tradeoff (even if configurable per map) because practically speaking those eating tons of memory are precisely the ones that need to be updated at run time (like geoloc).

In your case what is sad is that it eats twice the memory because the map is also loaded in the master process used to speak with systemd. We initially wanted it to switch to wait-mode after startup so that it doesn't hold all the config in RAM but William figured some corner cases which would be hard to address (I don't remember which ones and maybe it's not the case anymore). I'm marking this for "future fix" so that we don't forget it.

[aderumier]

Thanks for the explain !