search

openvstorage / framework

The Framework is a set of components and tools which brings the user an interface (GUI / API) to setup, extend and manage an Open vStorage platform.
Other
27 stars 23 forks source link

Framework should cleanup MDS slaves of removed volumes #713

Closed khenderick closed 7 years ago

khenderick commented 8 years ago

Framework should cleanup MDS slaves of removed (non-existing) volumes.

redlicha commented 8 years ago

These can be removed with the MDSClient API:

In [1]: import volumedriver.storagerouter.storagerouterclient as src

In [2]: mds_cfg = src.MDSNodeConfig(host, port)

In [3]: mds_client = src.MDSClient(mds_cfg)

In [4]: mds_client.remove_namespace(nspace)

This should be used for volumes that were removed completely or after removing an MDS instance from a volume's list of MDSes.

redlicha commented 8 years ago

Not an issue after all, the MDS slaves clean up after themselves if the backend namespace is gone:

2016-07-15 01:16:38 830774 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/MetaDataStoreBuilder - 00000000000037c6 -  info - update_metadata_store_: bba94722-355c-4773-882a-
ce713d94b68a: bringing MetaDataStore in sync with backend, requested interval (--, --], check scrub ID: CheckScrubId::F, dry run:DryRun::F, full rebuild: false
2016-07-15 01:16:38 830948 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/AlbaProxyClient - 00000000000037c7 -  info - logMessage: TCPProxy_client(10.42.12.103, 26234)
2016-07-15 01:16:38 831237 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/BackendConnectionInterfaceLogger - 00000000000037c8 -  info - Logger: Entering read bba94722-355c-
4773-882a-ce713d94b68a snapshots.xml
2016-07-15 01:16:38 832569 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/AlbaConnection - 00000000000037c9 -  error - convert_exceptions_: read object: caught Alba proxy e
xception: Proxy_protocol.Protocol.Error.NamespaceDoesNotExist
2016-07-15 01:16:38 832681 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/BackendConnectionInterfaceLogger - 00000000000037ca -  error - ~Logger: Exiting read for bba94722-
355c-4773-882a-ce713d94b68a snapshots.xml with exception
2016-07-15 01:16:38 832720 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/BackendConnectionInterfaceLogger - 00000000000037cb -  error - ~Logger: Exiting read for bba94722-
355c-4773-882a-ce713d94b68a
2016-07-15 01:16:38 832866 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/BackendInterface - 00000000000037cc -  error - fillObject: Problem getting snapshots.xml from bba9
4722-355c-4773-882a-ce713d94b68a: BackendException: namespace does not exist
2016-07-15 01:16:38 833104 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/MDSMetaDataBackend - 00000000000037cd -  info - ~MDSMetaDataBackend: bba94722-355c-4773-882a-ce713
d94b68a: used clusters: 0
2016-07-15 01:16:38 833154 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/MetaDataServerTable - 00000000000037ce -  warning - work_: bba94722-355c-4773-882a-ce713d94b68a: d
oes not exist anymore on the backend, cleaning up
2016-07-15 01:16:38 833183 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/MetaDataServerRocksTable - 00000000000037cf -  info - drop: bba94722-355c-4773-882a-ce713d94b68a: 
dropping the table
2016-07-15 01:16:38 833272 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/RocksLogger - 00000000000037d0 -  info - /mnt/ssd1/vpool7_db_mds_1: Dropped column family with id 
178

2016-07-15 01:16:38 833313 -0700 - sierra03 - 341/0x00007fd92affd700 - volumedriverfs/PeriodicActionPoolTask - 00000000000037d1 -  info - operator(): mds-poll-namespace-bba94722-355c-4
773-882a-ce713d94b68a: bailing out

I obviously misremembered that / got mislead by an env where the namespaces were not removed from the backend - sorry for the noise.

wimpers commented 8 years ago

See last comment of @redlicha , re-open if required.

saelbrec commented 7 years ago

The MDS namespace is only removed when the backend namespace is deleted. But if for some reason the MDS master/slaves get reconfigured this will over time result in MDS namespace possibly existing on all storagedriver instances while only a subset (being mds_safety) requires to have them.

IMO when the MDS configuration of a volume gets reconfigured by the Framework, we should also remove the namespace from the MDS where it's no longer required.

khenderick commented 7 years ago

Fixed by #1404, packaged on dev in openvstorage-2.7.12-dev.1486374024.de8e760

khenderick commented 7 years ago

openvstorage-2.7.12.1