search

openvstorage / framework

The Framework is a set of components and tools which brings the user an interface (GUI / API) to setup, extend and manage an Open vStorage platform.
Other
27 stars 23 forks source link

Bring volumedriver config sections and fields inline with framework config sections and fields #768

Closed kvanhijf closed 7 years ago

kvanhijf commented 8 years ago

The volumedriver on the Gig environment apparently has a field unknown to the framework being the 'average_entries_per_bin' in the 'content_addressed_cache' This is a known field for the storagedriver, but the framework does not know of this field. Now when mds_checkup for example is executed, the storagedriver configuration file is cleaned, meaning all unknown fields (to the framework) are removed from the config file, thus 'average_entries_per_bin' in this case was removed

@redlicha or @cnanakos should inform the framework when new fields are released

Environment information on which this was witnessed: cpu nodes 10.107.2.11-14 Stor nodes 10.107.2.21-24

Release: Eugene-updates (tweaked)

domsj commented 8 years ago

I think you should take a step back and look at why the framework cleans these unknown keys in the first place. This is due to the fact the volumedriver can't handle old config files. So the framework has to clean up any keys the volumedriver may no longer know. I think it would be better if the volumedriver would not trip over when it receives an old config file. With the current situation you get issues like the one described here. It also forces volumedriver updates (where the config format has changed) to coincide with updates to the framework ... so you can't just throw a newer version onto some node which may need it.

redlicha commented 8 years ago

This key is neither new nor deprecated. There are a number of such private configurables. Why GIG uses that one is beyond my knowledge at this moment, but if their use of it is legitimate then the param should be made public.

cnanakos commented 8 years ago

@redlicha A problem raised on the Gig environment when trying to create new volumes, truncation lasted almost 1min. Gig is using location based caching all the way and on creation the volume gets its own hash table which is preallocated in that case. The default value for the average entries per spine-entry is 2 and with a very large read cache the allocation results on lengthy truncation times. After setting the aforementioned value to 16 or 24 we managed to reduce the allocation time.

wimpers commented 8 years ago

Decision from BAM: @kvanhijf we should make sure that the MDS checkup doesn't update or remove any parameters of the config. Please update the code so this logic is removed. It is up to the voldrv to fail if the config is invalid.

khenderick commented 8 years ago

Some extra information/remarks:

khenderick commented 8 years ago

Based on @wimpers's comments:

khenderick commented 7 years ago

Fixed by #933, packaged in openvstorage-2.7.3-rev.4009.e4c1b59

JeffreyDevloo commented 7 years ago

Steps

Output

config = StorageDriverConfiguration('storagedriver', 'bfcf157c-418a-471a-b4d4-e3494a3e295d', 'myvpool03mdbttNTebUp6umVI')
config.load()
config.configuration['random_key'] = 'random_value'
config.save()
config.load()
config.configuration
{u'backend_connection_manager': {u'alba_connection_host': u'10.100.199.151',
  u'alba_connection_port': 26209,
  u'alba_connection_preset': u'mypreset',
  u'alba_connection_timeout': 15,
  u'alba_connection_transport': u'TCP',
  u'backend_interface_retries_on_error': 5,
  u'backend_interface_retry_backoff_multiplier': 2.0,
  u'backend_interface_retry_interval_secs': 1,
  u'backend_type': u'ALBA'},
 u'content_addressed_cache': {u'read_cache_serialization_path': [],
  u'serialize_read_cache': False},
 u'distributed_lock_store': {u'dls_arakoon_cluster_id': u'voldrv',
  u'dls_arakoon_cluster_nodes': [{u'host': u'10.100.199.152',
    u'node_id': u'CuKz9QCKPEydwgFC',
    u'port': 26420},
   {u'host': u'10.100.199.151',
    u'node_id': u'mdbttNTebUp6umVI',
    u'port': 26420},
   {u'host': u'10.100.199.153',
    u'node_id': u'xDH4v3m6IXuwScu5',
    u'port': 26412}],
  u'dls_type': u'Arakoon'},
 u'distributed_transaction_log': {u'dtl_path': u'/mnt/hdd3/myvpool03_write_dtl_1',
  u'dtl_transport': u'TCP'},
 u'event_publisher': {u'events_amqp_routing_key': u'volumerouter',
  u'events_amqp_uris': [{u'amqp_uri': u'amqp://ovs:0penv5tor4ge@10.100.199.151'},
   {u'amqp_uri': u'amqp://ovs:0penv5tor4ge@10.100.199.153'},
   {u'amqp_uri': u'amqp://ovs:0penv5tor4ge@10.100.199.152'}]},
 u'file_driver': {u'fd_cache_path': u'/mnt/hdd3/myvpool03_write_fd_1',
  u'fd_extent_cache_capacity': u'1024',
  u'fd_namespace': u'fd-myvpool03-bfcf157c-418a-471a-b4d4-e3494a3e295d'},
 u'filesystem': {u'fs_dtl_config_mode': u'Automatic',
  u'fs_dtl_host': u'',
  u'fs_dtl_mode': u'Asynchronous',
  u'fs_enable_network_interface': 1,
  u'fs_enable_shm_interface': 0,
  u'fs_metadata_backend_arakoon_cluster_nodes': [],
  u'fs_metadata_backend_mds_nodes': [{u'host': u'10.100.199.151',
    u'port': 26301}],
  u'fs_metadata_backend_type': u'MDS',
  u'fs_raw_disk_suffix': u'.raw',
  u'fs_virtual_disk_format': u'raw'},
 u'metadata_server': {u'mds_nodes': [{u'db_directory': u'/mnt/hdd1/myvpool03_db_mds_1',
    u'host': u'10.100.199.151',
    u'port': 26301,
    u'scratch_directory': u'/mnt/hdd1/myvpool03_db_mds_1'}]},
 u'random_key': u'random_value',
 u'scocache': {u'backoff_gap': u'2GB',
  u'scocache_mount_points': [{u'path': u'/mnt/hdd3/myvpool03_write_sco_1',
    u'size': u'6455296KiB'}],
  u'trigger_gap': u'1GB'},
 u'threadpool_component': {u'num_threads': 16},
 u'volume_manager': {u'clean_interval': 1,
  u'default_cluster_size': 4096,
  u'dtl_throttle_usecs': 4000,
  u'metadata_path': u'/mnt/hdd1/myvpool03_db_md_1',
  u'non_disposable_scos_factor': 2.0,
  u'number_of_scos_in_tlog': 16,
  u'tlog_path': u'/mnt/hdd1/myvpool03_db_tlog_1'},
 u'volume_registry': {u'vregistry_arakoon_cluster_id': u'voldrv',
  u'vregistry_arakoon_cluster_nodes': [{u'host': u'10.100.199.152',
    u'node_id': u'CuKz9QCKPEydwgFC',
    u'port': 26420},
   {u'host': u'10.100.199.151',
    u'node_id': u'mdbttNTebUp6umVI',
    u'port': 26420},
   {u'host': u'10.100.199.153',
    u'node_id': u'xDH4v3m6IXuwScu5',
    u'port': 26412}]},
 u'volume_router': {u'vrouter_backend_sync_timeout_ms': 5000,
  u'vrouter_file_read_threshold': 1024,
  u'vrouter_file_write_threshold': 1024,
  u'vrouter_id': u'myvpool03mdbttNTebUp6umVI',
  u'vrouter_max_workers': 16,
  u'vrouter_migrate_timeout_ms': 5000,
  u'vrouter_min_workers': 4,
  u'vrouter_redirect_timeout_ms': u'5000',
  u'vrouter_routing_retries': 10,
  u'vrouter_sco_multiplier': 1024,
  u'vrouter_use_fencing': True,
  u'vrouter_volume_read_threshold': 1024,
  u'vrouter_volume_write_threshold': 1024},
 u'volume_router_cluster': {u'vrouter_cluster_id': u'bfcf157c-418a-471a-b4d4-e3494a3e295d'}}

Test result

Test passed. The random key that I added was not thrown out during the save.

Packages