search

xcp-ng / xcp

Entry point for issues and wiki. Also contains some scripts and sources.
https://xcp-ng.org
1.27k stars 74 forks source link

Coalesce concurrency #341

Open ezaton opened 4 years ago

ezaton commented 4 years ago

This is a hard one. I have a XCP-ng pool with hundreds of VMs. I am using a scripted backup/export tool which orchestrates the process, and it works fine. However - this is a large pool with eight hosts and hundreds of VMs, each have its own unique VDI tree(s). I have hundreds of VDIs on a single SR, and both space and storage-wise - I am working fine. That said - I had to stop the scheduling backups, remove all snapshots and let the system perform its coalesce, because for a 7TB VDIs, on a 20TB SR, I was getting out of space. It has been coalescing for three days now, and it is a slow process, that I know, however - it is serial per-SR. It means that only one coalesce process would run per SR, even for unrelated VDI trees. Also - this process would run only on the pool master, which increases its load. I do not know how, at the moment, but the idea is to lock a tree, and orchestrate a coalesce process on any of the XCP hosts. This is just an idea, but I would love to have it considered.

Thanks

olivierlambert commented 4 years ago

Hi,

This is an idea I had since a while now: https://bugs.xenserver.org/browse/XSO-840

Please note XCP-ng 8.1 will have a better coalesce mechanism with adaptive speed. Maybe it will solve the issue, otherwise, we'll try to lock only the chain.

ezaton commented 4 years ago

I hope so. When 8.1 will be released, I will test it. It is a production pool, so I am not too eager to test beta on it. The whole coalesce thing is a problem - it is serial, it takes about 2-3 minutes for the whole lock-around-the-process to begin (excluding vhd-scan commands, which take their own time). I currently have several hundreds of coalesce-ready VDIs. I do not know if the logs report one-line per chain (then my situation is awful), or one line per VDI in a chain, and the coalescing truncates the whole chain, but either way - I am expecting several more days of coalescing with this task being serial as it is. This is, really, a component which does not scale well.

olivierlambert commented 4 years ago

It doesn't scale indeed, and it was a design choice to be extra careful without to have think deeply. I'm pretty sure it can be improved.

ezaton commented 4 years ago

I hope so. A snip from my SMlog: SR b48c ('zfs-lun0') (1898 VDIs in 170 VHD trees): showing only VHD trees that changed: ....

Also this: grep 19:02:53 /var/log/SMlog | grep coales | wc -l 1553

It is going to take forever. So many VHD trees, and each coalesce is taking its time...

olivierlambert commented 4 years ago

RC1 for XCP-ng 8.1 is really around the corner. You might try it and give feedback here. This will be helpful to see how far we'll continue to dig after that.

ezaton commented 4 years ago

I wonder if this SR change can be back-ported to XCP-ng 8.0. I would refrain from going into RC1 on a production environment. I would love to test this change on a single node, if it is not spread across too many components.

olivierlambert commented 4 years ago

Frankly, I don't see any major risk to go on RC1: the change have been pushed by Citrix in their 7.1 LTS, and CH 8.1 itself is now 2 months old.

Going from RC to final release is really a small gap for us.

stormi commented 3 years ago

For the record, when we'll transition to SMAPIv3, this won't be as necessary, or at least the implementation will be really different.