search

openvstorage / framework

The Framework is a set of components and tools which brings the user an interface (GUI / API) to setup, extend and manage an Open vStorage platform.
Other
27 stars 23 forks source link

vPool configuration changed after adding 4th node #864

Closed kinvaris closed 7 years ago

kinvaris commented 8 years ago

After extending the vpool to the 4th node, suddenly the vPool configuration changed (Dedupe, Cache on Read, write buffer 960, ...). When removing the node, it changed back to normal settings.

Version: unstable

ii  openvstorage                          2.7.2-rev.3884.42ef467-1                                 amd64        openvStorage
ii  openvstorage-backend                  1.7.2-rev.677.e6ec122-1                                  amd64        openvStorage Backend plugin
ii  openvstorage-backend-core             1.7.2-rev.677.e6ec122-1                                  amd64        openvStorage Backend plugin core
ii  openvstorage-backend-webapps          1.7.2-rev.677.e6ec122-1                                  amd64        openvStorage Backend plugin Web Applications
ii  openvstorage-cinder-plugin            1.2.2-rev.38.dcc3b76-1                                   amd64        OpenvStorage Cinder plugin for OpenStack
ii  openvstorage-core                     2.7.2-rev.3884.42ef467-1                                 amd64        openvStorage core
ii  openvstorage-hc                       1.7.2-rev.677.e6ec122-1                                  amd64        openvStorage Backend plugin HyperConverged
ii  openvstorage-sdm                      1.6.2-rev.330.f06c8de-1                                  amd64        Open vStorage Backend ASD Manager
ii  openvstorage-webapps                  2.7.2-rev.3884.42ef467-1                                 amd64        openvStorage Web Applications
jeroenmaelbrancke commented 8 years ago

Got the same issue on another test env. Skype chat between @kvanhijf and myself.

[11:04:27 AM] Jeroen Maelbrancke: hellowkes, vraagje ivm cache strategy op vpool of vdisk
[11:04:34 AM] Jeroen Maelbrancke: als een vpool of vdisk cache strategy op no cache staat. Dit betekent toch dat alles Read/Write onmiddellijk van de backend komen?
[11:04:38 AM] Jeroen Maelbrancke: of ben ik verkeerd?
[11:06:10 AM] Kevin Van Hijfte: we hebben 2 soorten caching nu he
de volumedriver caching, dewelke jij nu bedoelt denk ik
en de alba caching (fragment cache / accelerated alba)
[11:06:45 AM] Kevin Van Hijfte: dus voor hetgeen te bekomen dat jij zegt, moeten ze beiden disabled zijn
[11:08:25 AM] Jeroen Maelbrancke: er is maar 1 backend aangemaakt op die omgeving dus ik verwacht niet dat er een accelerated backend is aangemaakt. Geen Global backend te vinden.
[11:08:50 AM] Jeroen Maelbrancke: Dus de volumedriver caching is die dat ik terug vind op de vpool of vdisk en die staan op no cache
[11:09:10 AM] Jeroen Maelbrancke: dus kan ik er van uitgaan dat deze dus altijd vanuit de backend komt
[11:09:26 AM] Kevin Van Hijfte: nee sry accelerated alba was verkeerd
[11:17:35 AM] Kevin Van Hijfte: '/ovs/vpools/{0}/proxies/{1}/config/{{0}}'.format(vpool.guid, alba_proxy.guid)
[4:56:33 PM] Jeroen Maelbrancke: zeg die error jou iets?
[4:56:33 PM] Jeroen Maelbrancke: Task ovs.storagerouter.add_vpool[1b20110e-1b77-4840-96ce-aa28f92269e9] raised unexpected: RuntimeError('\nMandatory param "tlog_multiplier" with value "20" should be 1 of the following: [1, 1, 16, 2, 8, 4]',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/OpenvStorage/ovs/lib/storagerouter.py", line 312, in add_vpool
    actual_params=vpool.configuration)
  File "/opt/OpenvStorage/ovs/lib/helpers/toolbox.py", line 136, in verify_required_params
    raise RuntimeError('\n' + '\n'.join(error_messages))
RuntimeError: 
Mandatory param "tlog_multiplier" with value "20" should be 1 of the following: [1, 1, 16, 2, 8, 4]
[5:10:38 PM] Jeroen Maelbrancke: 2016-08-24 17:09:02 21300 +0200 - ovs03 - 88927/140329080055616 - celery/celery.pool - 2755 - DEBUG - TaskPool: Apply <function _fast_trace_task at 0x7fa0e372f848> (args:('ovs.storagerouter.add_vpool', '1b107613-8b3a-4a09-a191-ef5e7016f7ed', ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False, u'storage_ip': u'172.19.197.1', u'readcache_size': 0, u'storagerouter_ip': u'10.100.197.1', u'config_params': {u'write_buffer': 128, u'sco_size': 4, u'cache_strategy': u'none', u'dedupe_mode': u'non_dedupe', u'cluster_size': 4, u'dtl_transport': u'tcp', u'dtl_mode': u'a_sync'}, u'writecache_size': 104, u'backend_connection_info': {u'username': u'7510e204-8d37-4913-9c2b-966473f36e10', u'host': u'10.100.197.1', u'password': u'3TnMfa[q6iUx7bjl+]*75V-7:9:O2J5"xpcp|0>HYW:ZSP;h5M1Zaql*zcP-6Feail/u1;PkkYTI4+W[z7G]+Y/t2m3n:pA>9ifKmSwGZ@<Zb?V}fQ6gMP.JQTb#JQ7X', 'local': True, u'port': 443, u'backend': {u'backend': u'a382cbb3-465c-47ac-bbe7-55c799a4ab70', u'metadata': u'default'}}, u'fragment_cache_on_read': True, u'type': u'alba'},), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False,... kwargs:{})
[5:13:03 PM] Jeroen Maelbrancke: 2de node:
        TaskPool: Apply <function _fast_trace_task at 0x7fa0e372f848> (args:('ovs.storagerouter.add_vpool', 'd0277a45-a7bf-464c-882e-d925ea7d6605', ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False, u'storage_ip': u'172.19.197.2', u'readcache_size': 0, u'storagerouter_ip': u'10.100.197.2', u'config_params': {u'write_buffer': 128, u'sco_size': 4, u'cache_strategy': u'none', u'dedupe_mode': u'non_dedupe', u'cluster_size': 4, u'dtl_transport': u'tcp', u'dtl_mode': u'a_sync'}, u'writecache_size': 104, u'backend_connection_info': {u'username': u'7510e204-8d37-4913-9c2b-966473f36e10', u'host': u'10.100.197.1', u'password': u'3TnMfa[q6iUx7bjl+]*75V-7:9:O2J5"xpcp|0>HYW:ZSP;h5M1Zaql*zcP-6Feail/u1;PkkYTI4+W[z7G]+Y/t2m3n:pA>9ifKmSwGZ@<Zb?V}fQ6gMP.JQTb#JQ7X', 'local': True, u'port': 443, u'backend': {u'backend': u'a382cbb3-465c-47ac-bbe7-55c799a4ab70', u'metadata': u'default'}}, u'fragment_cache_on_read': True, u'type': u'alba'},), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False,... kwargs:{})
[5:14:58 PM] Jeroen Maelbrancke: node 3:
TaskPool: Apply <function _fast_trace_task at 0x7f7fd42da848> (args:('ovs.storagerouter.add_vpool', '6c7df291-617b-429a-81b8-b486e592b10f', ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False, u'storage_ip': u'172.19.197.3', u'readcache_size': 0, u'storagerouter_ip': u'10.100.197.3', u'config_params': {u'write_buffer': 128, u'sco_size': 4, u'cache_strategy': u'none', u'dedupe_mode': u'non_dedupe', u'cluster_size': 4, u'dtl_transport': u'tcp', u'dtl_mode': u'a_sync'}, u'writecache_size': 104, u'backend_connection_info': {u'username': u'7510e204-8d37-4913-9c2b-966473f36e10', u'host': u'10.100.197.1', u'password': u'3TnMfa[q6iUx7bjl+]*75V-7:9:O2J5"xpcp|0>HYW:ZSP;h5M1Zaql*zcP-6Feail/u1;PkkYTI4+W[z7G]+Y/t2m3n:pA>9ifKmSwGZ@<Zb?V}fQ6gMP.JQTb#JQ7X', 'local': True, u'port': 443, u'backend': {u'backend': u'a382cbb3-465c-47ac-bbe7-55c799a4ab70', u'metadata': u'default'}}, u'fragment_cache_on_read': True, u'type': u'alba'},), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False,... kwargs:{})
[5:26:39 PM] Jeroen Maelbrancke: en da is de laatste node waar we de error kregen:
TaskPool: Apply <function _fast_trace_task at 0x7f7fd42da848> (args:('ovs.storagerouter.add_vpool', '8ca32636-dc33-41a8-9ca4-8a6a198f6817', ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False, u'storage_ip': u'172.19.197.4', u'readcache_size': 64, u'storagerouter_ip': u'10.100.197.4', u'config_params': {u'write_buffer': 960, u'sco_size': 4, u'cache_strategy': u'on_read', u'dedupe_mode': u'dedupe', u'cluster_size': 4, u'dtl_transport': u'tcp', u'dtl_mode': u'a_sync'}, u'writecache_size': 104, u'backend_connection_info': {u'username': u'7510e204-8d37-4913-9c2b-966473f36e10', u'host': u'10.100.197.1', u'password': u'3TnMfa[q6iUx7bjl+]*75V-7:9:O2J5"xpcp|0>HYW:ZSP;h5M1Zaql*zcP-6Feail/u1;PkkYTI4+W[z7G]+Y/t2m3n:pA>9ifKmSwGZ@<Zb?V}fQ6gMP.JQTb#JQ7X', 'local': True, u'port': 443, u'backend': {u'backend': u'a382cbb3-465c-47ac-bbe7-55c799a4ab70', u'metadata': u'default'}}, u'fragment_cache_on_read': True, u'type': u'alba'},), {}, {'utc': True, u'is_eager': False, 'chord': None, u'group': None, 'args': ({u'vpool_name': u'ops-pool', u'fragment_cache_on_write': False,... kwargs:{})
 Thursday, August 25, 2016
[10:58:34 AM] Kevin Van Hijfte: %cpaste
from ovs.extensions.generic.sshclient import SSHClient
from ovs.extensions.services.service import ServiceManager
clients = []
for ip in ['10.100.193.154', '10.100.193.153', '10.100.193.152', '10.100.193.151']:
    try:
        clients.append(SSHClient(ip, 'root'))
    except:
        print 'GEBROKEN: {0}'.format(ip)
        break
for client in clients:
    ServiceManager.stop_service('watcher-framework', client)
    print 'Stopped watcher on {0}'.format(client.ip)
for client in clients:
    client.run('service memcached restart')
    print 'Restarted memcached on {0}'.format(client.ip)
for client in clients:
    ServiceManager.start_service('watcher-framework', client)
    print 'Started watcher on {0}'.format(client.ip)
--
[11:42:10 AM] Jeroen Maelbrancke: ansible is gedaan
[11:42:15 AM] Jeroen Maelbrancke: deploy maar je stuff
[1:44:52 PM] Kevin Van Hijfte: aangezien ik in vrijdagmodus vertoef, was ik nu vergeten of je het probleem nu al hebt gereproduceerd met de nieuwe code?
[1:44:57 PM] Kevin Van Hijfte: krijg je nu keyerrors?
[1:45:12 PM] Jeroen Maelbrancke: nope alles werkt
[1:45:20 PM] Jeroen Maelbrancke: geen errors te zien
[1:45:49 PM] Kevin Van Hijfte: maar daarvoor kon je da ook hebben toch?  dat alles werkte?    het was niet 100% zeker te reproduceren toch?
[1:46:00 PM] Jeroen Maelbrancke: indd
[1:46:14 PM] Jeroen Maelbrancke: ik heb het 3x geprobeerd en 3x zonder probleem
[1:46:15 PM] Kevin Van Hijfte: mss nog eens proberen?
[1:46:17 PM] Kevin Van Hijfte: ah
[1:46:24 PM] Kevin Van Hijfte: strange
[1:46:31 PM] Jeroen Maelbrancke: indd
[1:47:15 PM] Kevin Van Hijfte: kzou toch 10 keer proberen eigenlijk ;)
[1:49:58 PM] Jeroen Maelbrancke: straks zal ik het nog eens proberen
[2:32:09 PM] Jeroen Maelbrancke: In [7]: from ovs.extensions.generic.system import System
In [19]: machine_details = System.get_my_storagerouter()
In [20]: machine_details.ip
Out[20]: u'10.100.197.3'
khenderick commented 8 years ago

Create a vPool over 4 nodes, and the configuration didn't change after the extend to the last node.

Please re-validate and if still an issue, please give some more info (e.g. all the volumedriver's config files before the 4th node was added, and after).

wimpers commented 7 years ago

Closed by PM.