Delete Snapshots: why the oldest becomes bigger and bigger

[yongshengma]

Hello,

I find that some snapshots are very large and can be up to 4TB. After I deleted it, the next snapshot became 4T or even larger. What's going on? Snapshots are eating up space.

Best regards, Yongsheng

[jtorreke]

Hi Yongsheng,

The behavior you see is by design. Whenever you delete a snapshot, that is only a metadata operation. Let's say you have 3 snapshots, each containing 50MB worth of changes. When you delete the middle one, those 50MB of changes are moved to the older snapshot as only the snapshot pointer is removed, but not the data. So the oldest snapshot will be 100MB, the newer stays unchanged and will stay 50MB in size. To clean up the data itself, you'll need scrubbing. Scrubbing is the process which removes data from vdisks which is no longer referenced, by removing snapshots for example. Scrubbing is scheduled to run automatically every 2 hours by default (if I'm not mistaken) - you can also trigger it manually per vdisk from the GUI. When scrubbing finishes, the 100MB snapshot should be something between 50MB and 100MB in size. 50MB when the same LBA's have been re-written between the older and the freshly removed snapshot. 100MB in size when a completely different set of LBA's has been written between those snapshots.

Cheers, Jeroen

[JeffreyDevloo]

Hi Yongshengma

Scrubbing is normally scheduled by the Framework. You'll see task named ovs.generic.execute_scrub pop up every day in your workers.log

Best regards

[yongshengma]

Hi @jtorreke

Thanks!

Do snapshots have concrete data that occupy the space? For example, if the stored data of a vdisk is 50G, then its snapshots also have about 50G totally and these are another 50G space used .

I also see that the last (also the newest) snapshot has the same size of stored data as a vdisk's when I deleted the snapshots one by one from the oldest. Is it coincident or just should be like that?

Best regards, Yongsheng

[yongshengma]

I removed the oldest snapshot but it went to the next oldest which became 5.80T from around 11G. This vdisk's size is only 50G but its snapshot grows to 5.80T .

May 11 15:50:18 N05 celery: 2019-05-11 15:50:18 76100 +0800 - N05 - 2576/140549621679936 - lib/decorators.py - log_message - 140744 - INFO - Ensure single DEDUPED mode - ID 1557561018_v60zL5zuDw - New task ovs.generic.execute_scrub with params {'manual': True, 'vdisk_guids': ['000a43ce-568b-492d-817c-03935d070f49'], 'storagerouter_guid': None} scheduled for execution

[JeffreyDevloo]

Hi Yongshengma

I suspect that you are suffering from a race condition. If a snapshot is deleted while the vdisk is being scrubbed, the scrub results won't be applied and garbage is thrown onto the backend. The reason why I suspect is, is because scrubbing a big snapshot can take sometime and we've seen the issue pop up with bigger volumes in the past.

Could you provide me a snippet of the scrubbing logs to verify my hypothesis? You will find them in your ovs-workers logs under ovs.generic.execute_scrub

Best regards

[yongshengma]

Hi @JeffreyDevloo

I checked it again just now. That 5.8T snapshot is gone indeed.

I grepped on a node:

May 13 10:12:37 N06 celery: 2019-05-13 10:12:37 60000 +0800 - N06 - 35172/140182436366144 - celery/strategy.py - task_message_handler - 151049 - INFO - Received task: ovs.generic.execute_scrub[af8e5b8f-8c9c-4e3f-bacf-35f4a2b58d61]
May 13 10:12:37 N06 celery: 2019-05-13 10:12:37 60400 +0800 - N06 - 42148/140182436366144 - lib/decorators.py - log_message - 151048 - INFO - Ensure single DEDUPED mode - ID 1557713557_KRrgV6v4U7 - New task ovs.generic.execute_scrub with params {'manual': True, 'vdisk_guids': ['000a43ce-568b-492d-817c-03935d070f49'], 'storagerouter_guid': None} scheduled for execution
May 13 10:18:05 N06 celery: 2019-05-13 10:18:05 69400 +0800 - N06 - 42148/140182436366144 - lib/decorators.py - log_message - 151105 - INFO - Ensure single DEDUPED mode - ID 1557713557_KRrgV6v4U7 - Task ovs.generic.execute_scrub finished successfully
May 13 10:18:05 N06 celery: 2019-05-13 10:18:05 73700 +0800 - N06 - 35172/140182436366144 - celery/job.py - on_success - 151073 - INFO - Task ovs.generic.execute_scrub[af8e5b8f-8c9c-4e3f-bacf-35f4a2b58d61] succeeded in 328.136837486s: None
May 13 10:59:56 N06 celery: 2019-05-13 10:59:56 59600 +0800 - N06 - 35172/140182436366144 - celery/strategy.py - task_message_handler - 151238 - INFO - Received task: ovs.generic.execute_scrub[97571185-fabd-4067-9ee6-92984c7996df]
May 13 10:59:56 N06 celery: 2019-05-13 10:59:56 65800 +0800 - N06 - 40059/140182436366144 - lib/decorators.py - log_message - 151240 - INFO - Ensure single DEDUPED mode - ID 1557716396_uVMw6CMkOD - New task ovs.generic.execute_scrub with default params scheduled for execution
May 13 11:17:09 N06 celery: 2019-05-13 11:17:09 83900 +0800 - N06 - 35172/140182436366144 - celery/log.py - log - 151311 - ERROR - Task ovs.generic.execute_scrub[97571185-fabd-4067-9ee6-92984c7996df] raised unexpected: Exception('Errors occurred while scrubbing:\n  - Scrubber - vPool pool - StorageRouter N06 - vDisk chinasciencejournal_mysql - Scrubbing failed\n  - Scrubber - vPool pool - StorageRouter N06 - vDisk KXZK_TestServer02 - Scrubbing failed\n  - Scrubber - vPool pool - StorageRouter N02 - vDisk MDM-qasdata_disk2 - Scrubbing failed\n  - Scrubber - vPool pool - StorageRouter N02 - vDisk uat_IPLoadBalance_disk2 - Scrubbing failed\n  - Scrubber - vPool pool - StorageRouter N01 - vDisk KK-SERVER_02 - Scrubbing failed\n  - Scrubber - vPool pool - StorageRouter N01 - vDisk test_coursegate01-disk2 - Scrubbing failed\n  - Scrubber - vPool pool - StorageRouter N02 - vDisk uat_IPMicronode05 - Scrubbing failed\n  - Scrubber - vPool pool - StorageRouter N02 - vDisk win2008 - Scrubbing failed',)

The error in the last line is caused by those halted vdisks instead of scrub. I have opened another ticket for it. Really appreciate!

[yongshengma]

Hi @JeffreyDevloo

SCRUB role should be assigned only once such as only one node, or on every node?

[jtorreke]

Hi Yongsheng,

You need at least 1 node with a SCRUB role per environment. Per node with a scrub partition, 10 scrubbing threads will be launched. If you assign a scrub role to multiple nodes, you'll be able to scrub more vdisks simultaneously.

[yongshengma]

Thanks!