Zeroed regions in restored images are reported as data (will be backing up full provisioned)

[abbbi]

If a virtual machine has once been fully restored via virtnbdrestore, the next executing backup of the virtual machine is thick provisioned. Looking at an qemu-img mapping of the original and the restored qcow image, it seems the zeroed regions within the data image are reported with "data:true" instead of "data:false", means the current nbd client extention treats them as dirty blocks:

-{ "start": 131072, "length": 917504, "depth": 0, "zero": true, "data": false, "offset": 131072},
+{ "start": 131072, "length": 917504, "depth": 0, "zero": true, "data": true, "offset": 131072},

While the qcow image itself is correctly thin provisioned, this will make blocks appear as "dirty". Not sure how to solve this atm, probably some options for the qemu-nbd process started during restore are necessary.

[abbbi]

Currently solved by just leaving out the write operations for zeroed regions, thus punching "holes" in the image. Not sure if that is the appropriate solution for this, or if the qemu-nbd process that is writing the restored image could handle this via --detect-zeroes option.

Leaving the write operation for zeroed regions, thus punching holes seems OK. The resulting map of a restored virtual machine and the original virtual machines qcow images is the same after restore.

[abbbi]

08:51 <abi> im filling a qcow image with data using a qemu-nbd mapping and a self implemented nbd client. The nbclient is writting at the correct offsets and uses the nbd .zero function for 
            zeroes. The resulting qcow image is thin provisioned, but the zeroed regions are reported with ""zero": true, "data": true", instead of ""zero": true, "data": false" the qemu-nbd 
            process im using to map the image to is started with 
08:51 <abi> --discard=unmap.
08:53 <abi> what would be the correct way here filling the image and keep have the resulting map also report data: false?
08:53 <abi> just skipping the zeroed regions and punching "holes"?
08:53 <abi> or starting the qemu-nbd process with different options (--detect-zeroes?)
11:10 <rwmjones> abi: I think you'll have to ask eblake when he's around
-!- INBOX ::: "Gewinnspiel SMS" <Germaine@korvvarmeri.se>: blad ohne Potenz?
-!- INBOX ::: "Gut Ficken" <Tranise@megagrossmarkt.ch>: abi keine Errektionsprobleme
22:18 <abi> rwmjones: thanks, for now i just removed the .zero calls and poke "holes", then the resulting image shows the same qemu-img map as the original one, i just wonder if thats the 
            right way.
22:53 <rwmjones> abi: maybe asking the obvious but did you try sending nbd_trim requests instead of zero?
22:54 <abi> rwmjones: ive read about the trim function in the manpage yes, but not considered... ill try
22:56 <abi> rwmjones: but what would be the difference to just leaving the holes by only writing data, not zeroes?
22:56 <rwmjones> trim is what punches holes, zero _can_ punch holes unless you set a flag (LIBNBD_CMD_FLAG_NO_HOLE) to force it to prefer zeroing
22:56 <rwmjones> that would work too, assuming the image starts off completely sparse
22:57 <rwmjones> so for example ...
22:57 <rwmjones> $ qemu-img create -f qcow2 empty.qcow2 100M
22:57 <abi> its created by qemu-img with its original size, so i would expect it to be sparse anyways.
22:57 <rwmjones> $ qemu-nbd -t -f qcow2 empty.qcow2 
22:57 <rwmjones> $ nbdinfo --map nbd://localhost 0   104857600    3  hole,zero
22:57 <rwmjones> $ nbdinfo --map nbd://localhost
22:57 <rwmjones>  0   104857600    3  hole,zero
22:58 <rwmjones> $ nbdsh -u nbd://localhost
22:58 <rwmjones> nbd> h.pwrite (b'1' * 512, 0)
22:59 <rwmjones> $ nbdinfo --map nbd://localhost
22:59 <rwmjones>          0       65536    0  data
22:59 <rwmjones>      65536   104792064    3  hole,zero
22:59 <rwmjones> nbd> h.trim(512, 0)
23:00 <rwmjones> now I'm wondering if qemu-nbd supports trim ..?
23:00 <rwmjones> oh I forgot the --discard flag
23:01 <rwmjones> $ qemu-nbd --discard=unmap -t -f qcow2 empty.qcow2 
23:01 <rwmjones> nbd> h.trim(65536,0)
23:01 <rwmjones> $ nbdinfo --map nbd://localhost
23:01 <rwmjones>          0   104857600    3  hole,zero
23:02 <rwmjones> yeah so it does work if you add --discard=map, and if you trim big enough blocks (hint: use h.get_block_size)
23:02 <abi> ok, so thats doing what i would expect. But as im operating on a pristine qcow image, i could leave it aswell.
23:02 <rwmjones> yes you could just leave it if you know it's sparse already