search

51Degrees / Device-Detection

THE Fastest and most Accurate device detection for C / PHP / Perl / Python and Node.js - professionally maintained device data
https://51degrees.com/device-detection
Other
112 stars 46 forks source link

Nginx module problem #17

Closed stgleb closed 7 years ago

stgleb commented 7 years ago

I am using 51 degrees built as nginx loadable module. But each time I turn on it I've got and error

2017/08/11 12:01:41 [alert] 27799#27799: *37513 open socket #3121 left in connection 3108
2017/08/11 12:01:41 [alert] 27799#27799: *38522 open socket #1001 left in connection 3123
2017/08/11 12:01:41 [alert] 27799#27799: *37847 open socket #3138 left in connection 3125
2017/08/11 12:01:41 [alert] 27799#27799: *38171 open socket #3173 left in connection 3160
2017/08/11 12:01:41 [alert] 27799#27799: *38211 open socket #3186 left in connection 3173

nginx.conf

load_module "/etc/nginx/modules/ngx_http_51D_module.so";
user www-data;
worker_processes auto;
pid /var/run/nginx.pid;
worker_rlimit_nofile 65536;
events {
        use epoll;
        worker_connections 32768;
        multi_accept on;
}
http {
    51D_filePath /etc/nginx/data/51Degrees-PremiumV3_2.dat;
        51D_cache 10000;
        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 30;
        keepalive_requests 100;
        types_hash_max_size 2048;
        client_body_timeout 10;
        send_timeout 2;
        reset_timedout_connection on;

        include /etc/nginx/mime.types;
    server_tokens off;

        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";
        gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

This is location

location / {
        proxy_pass         http://mysite.com;
        proxy_http_version 1.1;
        proxy_pass_header       Server;
        proxy_set_header   X-Real-IP          $remote_addr;
        proxy_set_header   X-Forwarded-For    $proxy_add_x_forwarded_for;
        proxy_set_header   X_FORWARDED_PROTO  $scheme;
        proxy_set_header   Host               $host;
        proxy_set_header   Connection         "";
    ##51D
    51D_match_single x-hardware-name HardwareName;
#        51D_match_single x-hardware-family HardwareFamily;
#        51D_match_single x-hardware-model HardwareModel;
#        51D_match_single x-hardware-vendor HardwareVendor;
#
#        51D_match_single x-device-type DeviceType;
#        51D_match_single x-browser-name BrowserName;
#        51D_match_single x-browser-version BrowserVersion;
#
#        51D_match_single x-platform-name PlatformName;
#        51D_match_single x-platform-version PlatformVersion;
#        51D_match_single x-platform-vendor PlatformVendor;
#
#        ## Do a multiple HTTP header match for IsMobile, IsTablet and IsSmartphone ##
#        51D_match_all x-mobile IsMobile;
#        51D_match_all x-tablet IsTablet;
#        51D_match_all x-smartphone IsSmartPhone;
#
#        51D_match_all x-crawler IsCrawler;
#        51D_match_all x-html5 HTML5;
#        51D_match_all x-layout-engine LayoutEnginee;
#
#        ## Do a multiple HTTP header match for match metrics ##
#        51D_match_all x-device-id DeviceId;

        access_log off;
    error_log off;
}
stgleb commented 7 years ago

Also there is such kind of message in logs

2017/08/11 12:04:48 [alert] 28758#28758: shared memory zone "51Degrees Shared Cache" was locked by 28766
fan51degrees commented 7 years ago

http://codegists.com/code/preproc%20java/

51D_cache 0; # crucial - otherwise crashes might occur

Joseph51D commented 7 years ago

Hi @stgleb

Could you provide the following information in order to help us diagnose the issue:

Regards, Joe

DeamonMV commented 7 years ago

Hi We use those software

built with OpenSSL 1.0.2g  1 Mar 2016              
TLS SNI support enabled                            
configure arguments: --with-cc-opt='-g -O2 -fPIE -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2' --with-ld-opt='-Wl,-Bsymbolic-functions -fPIE -pie -Wl,-z,relro -Wl,-z,now' --prefix=/usr/share/nginx --conf-path=/etc/nginx/nginx.conf --http-log-path=/var/log/nginx/access.log --error-log-path=/var/log/nginx/error.log --lock-path=/var/lock/nginx.lock --pid-path=/run/nginx.pid --http-client-body-temp-path=/var/lib/nginx/body --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --http-proxy-temp-path=/var/lib/nginx/proxy --http-scgi-temp-path=/var/lib/nginx/scgi --http-uwsgi-temp-path=/var/lib/nginx/uwsgi --with-debug --with-pcre-jit --with-ipv6 --with-http_ssl_module --with-http_stub_status_module --with-http_realip_module --with-http_auth_request_module --with-http_addition_module --with-http_dav_module --with-http_geoip_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_image_filter_module --with-http_v2_module --with-http_sub_module --with-http_xslt_module --with-stream --with-stream_ssl_module --with-mail --with-mail_ssl_module --with-threads

OS on which we run nginx

DISTRIB_ID=Ubuntu                                  
DISTRIB_RELEASE=16.04                              
DISTRIB_CODENAME=xenial                            
DISTRIB_DESCRIPTION="Ubuntu 16.04.2 LTS"           
NAME="Ubuntu"                                      
VERSION="16.04.2 LTS (Xenial Xerus)"               
ID=ubuntu                                          
ID_LIKE=debian                                     
PRETTY_NAME="Ubuntu 16.04.2 LTS"                   
VERSION_ID="16.04"                                 
HOME_URL="http://www.ubuntu.com/"                  
SUPPORT_URL="http://help.ubuntu.com/"              
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/" 
VERSION_CODENAME=xenial                            
UBUNTU_CODENAME=xenial

we are building 51D from source and have gcc 

ii  gcc                                                   4:4.8.2-1ubuntu6                                      amd64        GNU C compiler                                                                  
ii  gcc-4.8                                               4.8.4-2ubuntu1~14.04.3                                amd64        GNU C compiler                                                                  
ii  gcc-4.8-base:amd64                                    4.8.4-2ubuntu1~14.04.3                                amd64        GCC, the GNU Compiler Collection (base package)                                 
ii  gcc-4.8-base:i386                                     4.8.4-2ubuntu1~14.04.3                                i386         GCC, the GNU Compiler Collection (base package)                                 
ii  gcc-4.9-base:amd64                                    4.9.3-0ubuntu4                                        amd64        GCC, the GNU Compiler Collection (base package)                                 
ii  gcc-4.9-base:i386                                     4.9.3-0ubuntu4                                        i386         GCC, the GNU Compiler Collection (base package)                                 
ii  lib32gcc1                                             1:4.9.3-0ubuntu4                                      amd64        GCC support library (32 bit Version)                                            
ii  libgcc-4.8-dev:amd64                                  4.8.4-2ubuntu1~14.04.3                                amd64        GCC support library (development files)                                         
ii  libgcc1:amd64                                         1:4.9.3-0ubuntu4                                      amd64        GCC support library                                                             
ii  libgcc1:i386                                          1:4.9.3-0ubuntu4                                      i386         GCC support library                               ```

and OS

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.5 LTS"
NAME="Ubuntu"
VERSION="14.04.5 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.5 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

DeamonMV commented 7 years ago

we are building 51D from source - i mean that we build module of 51D fo nginx

DeamonMV commented 7 years ago

We have tied to use 51D_cache 0; And that is resolve problem. I'm not sure that will work when we get more then 2-3K active connection but now its work

Also what do this parameter?

Joseph51D commented 7 years ago

Thanks for the information, I will try to recreate the issue.

51D_Cache sets the size of the target User-Agents cache and the results set cache. This is used in pattern mode to reduce calls to the data set.

Regards, Joe

ben51degrees commented 7 years ago

@stgleb @DeamonMV Thanks for bringing this up, this is a bug which did not get caught in any of our tests as it only happens in a very specific scenario.

The error is occurring when two worker processes match the same set of HTTP headers and try to insert the identical matches into the cache at the same time. Process 1 will lock the cache, insert the match, unlock the cache, then return. Then process 2 will lock the cache, dispose of its match (as it has already been added to the cache by process 1), then return without unlocking the cache. You can see this on line 768 of the module source.

As you can tell, this has not come up for so long as it requires a set of HTTP headers which present in the cache already to arrive at two worker processes in the same <1ms window in order for this to happen.

I have attached a patch which fixes this bug, and it will be included in our next release.

0001-BUG-User-Agent-cache-in-the-shared-memory-zone-lock.patch.zip

Regards, Ben