leev / ngx_http_geoip2_module

Nginx GeoIP2 module
BSD 2-Clause "Simplified" License
958 stars 182 forks source link

Nginx returns 502 'worker process exited on signal 7' #98

Open ysgard opened 3 years ago

ysgard commented 3 years ago

This is an odd one and I can't reproduce it reliably, but it happens often enough in our test/staging environments (but not production) that we've had to stop using the module. Signal 7 is SIGBUS and usually a module problem when it occurs in nginx.

What happens is that occasionally nginx will wedge on a request, returning a 502. The log contains this cryptic error:

worker process 8519 exited on signal 7

The only reason I know it's GeoIP2 is because when the nginx is recompiled without the geoip2 module, it goes away. I've tried both static and dynamic compilations and the result is the same.

I've put the nginx configure script and nginx.conf in a gist: https://gist.github.com/ysgard/bd8305af0689af5484c21230a2cbf0fc

The problem is odd because it only manifests on our environments which don't get a lot of load/traffic. On our production systems it runs without issue.

dmitry-j-mikhin commented 3 years ago

We had the same similar issue. Actually the problem was in process how mmdb (database) file was updated. We first download file in /tmp folder and then move it to /var (nginx use it from /var in our config). If /tmp and /var is placed on one file system then all works OK. Move is atomic in this case. But if /var and /tmp are placed on different file systems, move isn't atomic. Old mmdb file is already mmapped into nginx process virtual memory, and instead of inode change, move overwrites process memory leading to SIGBUS.

ysgard commented 3 years ago

Interesting - we host the mmdb file on AWS's EFS filesystem, so this could be the issue. I'll try verifying.

nanght commented 3 years ago

I have the same issue. When i using geoipupdate to update our mmdb, nginx have some errors in error log, the nginx workers force recreate.

ysgard commented 2 years ago

Verified that moving the mmdb files off of EFS and onto the root filesystem solved the issue for us.

asmaek91 commented 1 year ago

i have the same error over web download ... Any progress on this ?