search

openvstorage / framework

The Framework is a set of components and tools which brings the user an interface (GUI / API) to setup, extend and manage an Open vStorage platform.
Other
27 stars 23 forks source link

increase number of file descriptors for ovs-dtl services #951

Closed dejonghb closed 7 years ago

dejonghb commented 7 years ago

Please add "limit nofile 4096 4096" to the upstart / systemd service scripts for the ovs-dtl (failovercache)

Increases allowed number of open files to 4096 (from 1024)

kinvaris commented 7 years ago

related to: https://github.com/openvstorage/framework/issues/952

kvanhijf commented 7 years ago

openvstorage-2.7.3-rev.4052.81d14d5

@QA: Retrieve PID of DTL service Both on systemd and upstart check /proc//limits and verify the max open files Both hard and soft limit should be on 4096

root@Ubuntu-16-04:~# cat /proc/32281/limits Max open files 4096 4096 files

JeffreyDevloo commented 7 years ago

Steps

Executed:

from ovs.dal.lists.vpoollist import VPoolList
from subprocess import check_output

dtl_paths = []
prefix = '/usr/bin/failovercache --path'
limit = 4096

for v in VPoolList.get_vpools():
    for p in v.storagedrivers[0].partitions:
        if 'DTL' in p.role or 'DTL' in p.sub_role:
            dtl_paths.append(p.path)
            break

for dtl_path in dtl_paths:
    psid = check_output('ps aux | grep "{0} {1}" | grep -v grep'.format(prefix,dtl_path), shell=True).split()[1]

    soft_limit = None
    hard_limit = None
    with open('/proc/{0}/limits'.format(psid), 'r') as f:
        # Softlimit = 2nd column, #Hard limit is 3rd column
        for line in f.readlines():
            if line.startswith('Max open files'):
                soft_limit = line.split()[3]
                hard_limit = line.split()[4]

    if (soft_limit and hard_limit) is not None and (soft_limit and hard_limit) == str(limit):
        # Test passed
        print 'DTL is fine.'
    else:
        raise ValueError('DTL limits are not correct. Found soft limit = {0} and hard limit = {1}'.format(soft_limit, hard_limit))

Output

In [8]: %cpaste
Pasting code; enter '--' alone on the line to stop or use Ctrl-D.
:from ovs.dal.lists.vpoollist import VPoolList
:from subprocess import check_output
:
:dtl_paths = []
:prefix = '/usr/bin/failovercache --path'
:limit = 4096
:
:for v in VPoolList.get_vpools():
:    for p in v.storagedrivers[0].partitions:
:        if 'DTL' in p.role or 'DTL' in p.sub_role:
:            dtl_paths.append(p.path)
:            break
:
:for dtl_path in dtl_paths:
:    psid = check_output('ps aux | grep "{0} {1}" | grep -v grep'.format(prefix,dtl_path), shell=True).split()[1]
:
:    soft_limit = None
:    hard_limit = None
:    with open('/proc/{0}/limits'.format(psid), 'r') as f:
:        # Softlimit = 2nd column, #Hard limit is 3rd column
:        for line in f.readlines():
:            if line.startswith('Max open files'):
:                soft_limit = line.split()[3]
:                hard_limit = line.split()[4]
:
:    if (soft_limit and hard_limit) is not None and (soft_limit and hard_limit) == str(limit):
:        # Test passed
:        print 'DTL is fine.'
:    else:
:        raise ValueError('DTL limits are not correct. Found soft limit = {0} and hard limit = {1}'.format(soft_limit, hard_limit))
:
:--
DTL is fine.
DTL is fine.

Manual:

root@ovs-node1:~# ps -aux | grep dtl | grep -v grep
root     26407  0.0  0.0  56384  6712 ?        Ssl  14:37   0:00 /usr/bin/failovercache --path /mnt/hdd1/myvpool01_write_dtl_1 --address 10.100.199.151 --port 26202 --transport TCP --logrotation --logsink console:
root     28257  0.0  0.0  56384  6484 ?        Ssl  14:40   0:00 /usr/bin/failovercache --path /mnt/hdd1/myvpool02_write_dtl_1 --address 10.100.199.151 --port 26207 --transport TCP --logrotation --logsink console:
root@ovs-node1:~# ps -aux | grep dtl | grep -v grep
root     26407  0.0  0.0  56384  6712 ?        Ssl  14:37   0:00 /usr/bin/failovercache --path /mnt/hdd1/myvpool01_write_dtl_1 --address 10.100.199.151 --port 26202 --transport TCP --logrotation --logsink console:
root     28257  0.0  0.0  56384  6484 ?        Ssl  14:40   0:00 /usr/bin/failovercache --path /mnt/hdd1/myvpool02_write_dtl_1 --address 10.100.199.151 --port 26207 --transport TCP --logrotation --logsink console:
root@ovs-node1:~# cat /proc/26407/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             64100                64100                processes 
Max open files            4096                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       64100                64100                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
root@ovs-node1:~# cat /proc/28257/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             64100                64100                processes 
Max open files            4096                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       64100                64100                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us

Test result

Test passed.

Packages