search

coreos / coreos-assembler

Tooling container to assemble CoreOS-like systems
https://coreos.github.io/coreos-assembler/
Apache License 2.0
335 stars 165 forks source link

OSBuild without compression yields GRUB failures #3728

Closed dustymabe closed 5 months ago

dustymabe commented 6 months ago

When we switched OSbuild over to not compressing we started getting failures where images wouldn't boot like:

  Booting `Fedora CoreOS 40.20240207.dev.1 (ostree:0)'

error: ../../grub-core/fs/fshelp.c:257:file
`/boot/ostree/fedora-coreos-f25027bf4dfb2213
7b7c8401ed2396924ba112482afc30d62b7a4ab1310b09db/vmlinuz-6.8.0-0.rc0.20240112gi
t70d201a40823.5.fc40.x86_64' not found.
error: ../../grub-core/loader/i386/pc/linux.c:422:you need to load the kernel
first.

Press any key to continue...
dustymabe commented 6 months ago

some more info.. it appears something with the ext4 filesystem when OSBuild runs the qemu-img convert to get from raw to qcow2 is getting corrupted. I converted the image back into a raw image from the qcow2 and mount up the third partition and when I ls on the ostree/ dir I see:

bash-5.2# ls /mnt/ostree/
[ 7220.329345] EXT4-fs error (device loop0p3): ext4_lookup:1855: inode #65538: comm ls: iget: checksum invalid
ls: cannot access '/mnt/ostree/': Bad message
dustymabe commented 6 months ago

ok further update.. I became increasingly suspicious of the corruption happening earlier in the OSBuild steps themselves rather than when copying the file out of the supermin VM.

I decided to just add a sanity check into the org.osbuild.qemu OSBuild stage that immediately ran a qemu-img compare after it created the qcow2 from the raw image. Here's the patch:

diff --git a/stages/org.osbuild.qemu b/stages/org.osbuild.qemu
index 642b5146..54e707d4 100755
--- a/stages/org.osbuild.qemu
+++ b/stages/org.osbuild.qemu
@@ -219,22 +219,34 @@ def main(inputs, output, options):
     if coroutines:
         print(f"qemu-img coroutines: {coroutines}")
         extra_args += ["-m", coroutines]

     cmd = [
         "qemu-img", "convert",
         "-O", fmt["type"],
         *extra_args,
         source, target
     ]

     subprocess.run(
         cmd, check=True
     )

+    # Sanity check that the image is 100%
+    cmd = [
+        "qemu-img", "compare",
+        "-f", "raw",
+        "-F", fmt["type"],
+        source, target
+    ]
+    subprocess.run(
+        cmd, check=True
+    )
+
+
     return 0

 if __name__ == '__main__':
     args = osbuild.api.arguments()
     r = main(args["inputs"], args["tree"], args["options"])
     sys.exit(r)

and sure enough one of the first runs I went through complained:

⏱  Duration: 0s
Pipeline qemu: dd43158cea505a60ebd511dc09e0fba0247cef4903ebf9c5b2c70e813b54a2cf
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.qemu: dd43158cea505a60ebd511dc09e0fba0247cef4903ebf9c5b2c70e813b54a2cf {
  "filename": "fedora-coreos-39.20240214.dev.0-qemu.x86_64.qcow2.tmp",
  "format": {
    "type": "qcow2",
    "compression": false,
    "compat": "1.1"
  }
}
[   97.999904] audit: type=1400 audit(1707944575.724:72): avc:  denied  { mounton } for  pid=961 comm="mount" path="/srv/cache/osbuild/store/tmp/buildroot-tmp-bzjrq14z/inputs/image/disk.img" dev="vdb1" ino=17809672 scontext=system_u:system_r:ke1
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
Content mismatch at offset 403718656!
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.qemu", line 251, in <module>
    r = main(args["inputs"], args["tree"], args["options"])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/osbuild/bin/org.osbuild.qemu", line 241, in main
    subprocess.run(
  File "/usr/lib64/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['qemu-img', 'compare', '-f', 'raw', '-F', 'qcow2', '/run/osbuild/inputs/image/disk.img', '/run/osbuild/tree/fedora-coreos-39.20240214.dev.0-qemu.x86_64.qcow2.tmp']' returned non-zero exit status 1.

⏱  Duration: 4s

Failed

From the Content mismatch at offset 403718656! line we can see that offset 403718656 is ~ 385 MiB which places it squarely in the /boot/ parition 3.

dustymabe commented 6 months ago

Talked to @jlebon on this. We think (actually more @jlebon, I was just nodding my head to everything he said) that this might be a bug with reflinks on XFS.

The cache qcow2 is an XFS filesystem and when we switch to not using compression for the org.osbuild.qemu stage we think it enables qemu-img convert to use reflinks for some of the data ranges in the image (i.e. to share between the source raw image and newly created qcow2 image).

i.e. this is a theory that could explain why we see issues only after switching off compression. This particular issue reminded @jlebon of https://github.com/coreos/coreos-assembler/pull/935 which we never fully got to the bottom of.

rwmjones commented 6 months ago

So I don't know if it's this, but the classic failure mode when you create a disk image and then boot from it shortly afterwards is:

If this is the scenario, then you need to modify your image creation pipeline so that it does an fsync or similar on the disk image before qemu runs.

Here's how virt-builder does that:

https://github.com/libguestfs/guestfs-tools/blob/acebc8b9962a78d36afe01cae78c11410b483248/builder/builder.ml#L803

dustymabe commented 6 months ago

@rwmjones in the failure I detailed in https://github.com/coreos/coreos-assembler/issues/3728#issuecomment-1944607900 all that is happening is:

and you can see the failure:

Content mismatch at offset 403718656!

Would you anticipate this to be a kernel page cache issue?

rwmjones commented 6 months ago

No that wouldn't the kernel page cache issue.

It's extremely suspicious though. Usually qcow2 and qemu-img are rock solid tools.

dustymabe commented 6 months ago

It's extremely suspicious though. Usually qcow2 and qemu-img are rock solid tools.

Yes :). I'd like to take a moment here and thank you and all the other maintainers of those tools over the years. You've truly built something that is load bearing for half of the internet and the world's economy at this point.

I currently don't think it's an issue in qemu-img, but will let you know if that changes.

I do appreciate you weighing in here, though.

dustymabe commented 6 months ago

OK - I was asked by @sandeen to provide a disk image containing an XFS filesystem with the good and bad generated disk images on it.

I had to make a slight modification to OSBuild to make it save off the bad disk image on failure:

diff --git a/osbuild/pipeline.py b/osbuild/pipeline.py
index af4c3944..93184bf6 100644
--- a/osbuild/pipeline.py
+++ b/osbuild/pipeline.py
@@ -358,6 +358,8 @@ class Pipeline:

             results["stages"].append(r)
             if not r.success:
+                print(f"XXXX failed tree object at {tree.path} committing")
+                object_store.commit(tree, stage.id)
                 cleanup(build_tree, tree)
                 results["success"] = False
                 return results

Then I ran this in a loop:

while cosa buildextend-qemu --force; do 
  echo "COUNT is $count";
  count=$((count+1));
  rm -f cache/cache2.qcow2;
done

It took a couple of iterations, but it did fail with:

⏱  Duration: 0s
Pipeline qemu: e0ab569c34dd263e53d28e56e0b2f9d6f27c4cb7c4fe5d7177f57f8b719fd229
Build
  root: <host>
  runner: org.osbuild.fedora38 (org.osbuild.fedora38)
org.osbuild.qemu: e0ab569c34dd263e53d28e56e0b2f9d6f27c4cb7c4fe5d7177f57f8b719fd229 {
  "filename": "qemu.qcow2",
  "format": {
    "type": "qcow2",
    "compression": false,
    "compat": "1.1"
  }
}
Failed to open file "/sys/fs/selinux/checkreqprot": Read-only file system
Content mismatch at offset 403718656!
Traceback (most recent call last):
  File "/run/osbuild/bin/org.osbuild.qemu", line 251, in <module>
    r = main(args["inputs"], args["tree"], args["options"])
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/run/osbuild/bin/org.osbuild.qemu", line 241, in main
    subprocess.run(
  File "/usr/lib64/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['qemu-img', 'compare', '-f', 'raw', '-F', 'qcow2', '/run/osbuild/inputs/image/disk.img', '/run/osbuild/tree/qemu.qcow2']' returned non-zero exit status 1.

⏱  Duration: 5s
XXXX failed tree object at /srv/cache/osbuild/store/stage/uuid-66d99595d47645869c04b0608ce618f9/data committing

Failed
+ rm -rf /srv/tmp/build.qemu/supermin.out /srv/tmp/build.qemu/supermin.prepare /srv/tmp/build.qemu/supermin.build
+ '[' '!' -f /srv/tmp/build.qemu/rc ']'
++ cat /srv/tmp/build.qemu/rc
+ rc=1
+ '[' -n '' ']'
+ return 1
+ rm -f /srv/builds/39.20240216.dev.0/x86_64/.qemu.building
failed to execute cmd-buildextend-qemu: exit status 1

[coreos-assembler]$
[coreos-assembler]$ rpm -q kernel-core xfsprogs
kernel-core-6.7.3-200.fc39.x86_64
xfsprogs-6.4.0-1.fc39.x86_64

Here's a link to the disk image: cache2.qcow2.zst.

[dustymabe@media cache]$ md5sum cache2.qcow2*
99e49544af7c10ed419afa6040cf3a7e  cache2.qcow2
48718e271056210d82e9ecf04fa7f624  cache2.qcow2.zst

Within the disk XFS filesystem the files for input and output of qemu-img convert are located at:

I was also asked to run xfs_bmap -vvp against the source raw image. Here is the output of that:

``` bash-5.2# xfs_bmap -vvp osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img: EXT: FILE-OFFSET BLOCK-RANGE AG AG-OFFSET TOTAL FLAGS 0: [0..39]: 28028432..28028471 3 (2862800..2862839) 40 100000 1: [40..2047]: hole 2008 2: [2048..2063]: 28023792..28023807 3 (2858160..2858175) 16 100000 3: [2064..2071]: 28023608..28023615 3 (2857976..2857983) 8 100000 4: [2072..2079]: 28023480..28023487 3 (2857848..2857855) 8 100000 5: [2080..2095]: 28020720..28020735 3 (2855088..2855103) 16 100000 6: [2096..2103]: 28020536..28020543 3 (2854904..2854911) 8 100000 7: [2104..2111]: 28020216..28020223 3 (2854584..2854591) 8 100000 8: [2112..2119]: 28020088..28020095 3 (2854456..2854463) 8 100000 9: [2120..2127]: 28019768..28019775 3 (2854136..2854143) 8 100000 10: [2128..2143]: 28019440..28019455 3 (2853808..2853823) 16 100000 11: [2144..2151]: 28019256..28019263 3 (2853624..2853631) 8 100000 12: [2152..2167]: 28019120..28019135 3 (2853488..2853503) 16 100000 13: [2168..2175]: 28018616..28018623 3 (2852984..2852991) 8 100000 14: [2176..2183]: 28018296..28018303 3 (2852664..2852671) 8 100000 15: [2184..2191]: 28017528..28017535 3 (2851896..2851903) 8 100000 16: [2192..2199]: 28017144..28017151 3 (2851512..2851519) 8 100000 17: [2200..4095]: hole 1896 18: [4096..4647]: 28028472..28029023 3 (2862840..2863391) 552 100000 19: [4648..19831]: 31393624..31408807 3 (6227992..6243175) 15184 100000 20: [19832..264191]: hole 244360 21: [264192..264207]: 1747104..1747119 0 (1747104..1747119) 16 000000 22: [264208..264719]: 28045576..28046087 3 (2879944..2880455) 512 100000 23: [264720..264743]: 28025056..28025079 3 (2859424..2859447) 24 100000 24: [264744..264751]: 28046112..28046119 3 (2880480..2880487) 8 100000 25: [264752..264775]: hole 24 26: [264776..264783]: 28029152..28029159 3 (2863520..2863527) 8 100000 27: [264784..264791]: 1747952..1747959 0 (1747952..1747959) 8 000000 28: [264792..265799]: hole 1008 29: [265800..265807]: 28027680..28027687 3 (2862048..2862055) 8 100000 30: [265808..280135]: hole 14328 31: [280136..280159]: 28028192..28028215 3 (2862560..2862583) 24 100000 32: [280160..280167]: 28029160..28029167 3 (2863528..2863535) 8 100000 33: [280168..280239]: 28046120..28046191 3 (2880488..2880559) 72 100000 34: [280240..280247]: 1748304..1748311 0 (1748304..1748311) 8 000000 35: [280248..280359]: 28046200..28046311 3 (2880568..2880679) 112 100000 36: [280360..280367]: 28046440..28046447 3 (2880808..2880815) 8 100000 37: [280368..280575]: hole 208 38: [280576..280591]: 28046312..28046327 3 (2880680..2880695) 16 100000 39: [280592..282623]: hole 2032 40: [282624..282631]: 28017016..28017023 3 (2851384..2851391) 8 100000 41: [282632..284671]: hole 2040 42: [284672..313343]: 28178184..28206855 3 (3012552..3041223) 28672 100000 43: [313344..313359]: 28046328..28046343 3 (2880696..2880711) 16 100000 44: [313360..313863]: hole 504 45: [313864..314119]: 28206856..28207111 3 (3041224..3041479) 256 100000 46: [314120..317439]: hole 3320 47: [317440..346111]: 28210432..28239103 3 (3044800..3073471) 28672 100000 48: [346112..346127]: 28046344..28046359 3 (2880712..2880727) 16 100000 49: [346128..350207]: hole 4080 50: [350208..378879]: 28239104..28267775 3 (3073472..3102143) 28672 100000 51: [378880..378895]: 28046360..28046375 3 (2880728..2880743) 16 100000 52: [378896..382975]: hole 4080 53: [382976..411647]: 28267776..28296447 3 (3102144..3130815) 28672 100000 54: [411648..411663]: 28046376..28046391 3 (2880744..2880759) 16 100000 55: [411664..415743]: hole 4080 56: [415744..428039]: 28296448..28308743 3 (3130816..3143111) 12296 100000 57: [428040..502079]: 28404656..28478695 3 (3239024..3313063) 74040 100000 58: [502080..526335]: hole 24256 59: [526336..526343]: 28046392..28046399 3 (2880760..2880767) 8 100000 60: [526344..526351]: 28024936..28024943 3 (2859304..2859311) 8 100000 61: [526352..526367]: hole 16 62: [526368..526375]: 28024344..28024351 3 (2858712..2858719) 8 100000 63: [526376..526399]: hole 24 64: [526400..526567]: 31387216..31387383 3 (6221584..6221751) 168 100000 65: [526568..526575]: 31387408..31387415 3 (6221776..6221783) 8 100000 66: [526576..542783]: hole 16208 67: [542784..542807]: 31387384..31387407 3 (6221752..6221775) 24 100000 68: [542808..543743]: hole 936 69: [543744..543767]: 28024912..28024935 3 (2859280..2859303) 24 100000 70: [543768..543775]: 28014648..28014655 3 (2849016..2849023) 8 100000 71: [543776..544767]: hole 992 72: [544768..544799]: 28024160..28024191 3 (2858528..2858559) 32 100000 73: [544800..544815]: 28024328..28024343 3 (2858696..2858711) 16 100000 74: [544816..544831]: 28023960..28023975 3 (2858328..2858343) 16 100000 75: [544832..544847]: 28023776..28023791 3 (2858144..2858159) 16 100000 76: [544848..544879]: 28023576..28023607 3 (2857944..2857975) 32 100000 77: [544880..544903]: 28023456..28023479 3 (2857824..2857847) 24 100000 78: [544904..544943]: 28023256..28023295 3 (2857624..2857663) 40 100000 79: [544944..544975]: 28023136..28023167 3 (2857504..2857535) 32 100000 80: [544976..545015]: 28022936..28022975 3 (2857304..2857343) 40 100000 81: [545016..545031]: 28022744..28022759 3 (2857112..2857127) 16 100000 82: [545032..545071]: 28022552..28022591 3 (2856920..2856959) 40 100000 83: [545072..545103]: 28022368..28022399 3 (2856736..2856767) 32 100000 84: [545104..545135]: 28022240..28022271 3 (2856608..2856639) 32 100000 85: [545136..545159]: 28022040..28022063 3 (2856408..2856431) 24 100000 86: [545160..545199]: 28021912..28021951 3 (2856280..2856319) 40 100000 87: [545200..545215]: 28022064..28022079 3 (2856432..2856447) 16 100000 88: [545216..545223]: 28021656..28021663 3 (2856024..2856031) 8 100000 89: [545224..545263]: 28021464..28021503 3 (2855832..2855871) 40 100000 90: [545264..545303]: 28021336..28021375 3 (2855704..2855743) 40 100000 91: [545304..545343]: 28021144..28021183 3 (2855512..2855551) 40 100000 92: [545344..545375]: 28021024..28021055 3 (2855392..2855423) 32 100000 93: [545376..545415]: 28020824..28020863 3 (2855192..2855231) 40 100000 94: [545416..545439]: 28020696..28020719 3 (2855064..2855087) 24 100000 95: [545440..545463]: 28020512..28020535 3 (2854880..2854903) 24 100000 96: [545464..545495]: 28020384..28020415 3 (2854752..2854783) 32 100000 97: [545496..545527]: 28020184..28020215 3 (2854552..2854583) 32 100000 98: [545528..545559]: 28020056..28020087 3 (2854424..2854455) 32 100000 99: [545560..545599]: 28019864..28019903 3 (2854232..2854271) 40 100000 100: [545600..545623]: 28019744..28019767 3 (2854112..2854135) 24 100000 101: [545624..545655]: 28019552..28019583 3 (2853920..2853951) 32 100000 102: [545656..545671]: 28019424..28019439 3 (2853792..2853807) 16 100000 103: [545672..545703]: 28019224..28019255 3 (2853592..2853623) 32 100000 104: [545704..545727]: 28019096..28019119 3 (2853464..2853487) 24 100000 105: [545728..545767]: 28018904..28018943 3 (2853272..2853311) 40 100000 106: [545768..545799]: 28018784..28018815 3 (2853152..2853183) 32 100000 107: [545800..545831]: 28018584..28018615 3 (2852952..2852983) 32 100000 108: [545832..545863]: 28018464..28018495 3 (2852832..2852863) 32 100000 109: [545864..545895]: 28018264..28018295 3 (2852632..2852663) 32 100000 110: [545896..545935]: 28018136..28018175 3 (2852504..2852543) 40 100000 111: [545936..545975]: 28017944..28017983 3 (2852312..2852351) 40 100000 112: [545976..546015]: 28017816..28017855 3 (2852184..2852223) 40 100000 113: [546016..546047]: 28017632..28017663 3 (2852000..2852031) 32 100000 114: [546048..546071]: 28017504..28017527 3 (2851872..2851895) 24 100000 115: [546072..546103]: 28017312..28017343 3 (2851680..2851711) 32 100000 116: [546104..546127]: 28017120..28017143 3 (2851488..2851511) 24 100000 117: [546128..546151]: 28016984..28017007 3 (2851352..2851375) 24 100000 118: [546152..546183]: 28016800..28016831 3 (2851168..2851199) 32 100000 119: [546184..546207]: 28016672..28016695 3 (2851040..2851063) 24 100000 120: [546208..546247]: 28016472..28016511 3 (2850840..2850879) 40 100000 121: [546248..546279]: 28016352..28016383 3 (2850720..2850751) 32 100000 122: [546280..546303]: 28016160..28016183 3 (2850528..2850551) 24 100000 123: [546304..546327]: 28015968..28015991 3 (2850336..2850359) 24 100000 124: [546328..546367]: 28015832..28015871 3 (2850200..2850239) 40 100000 125: [546368..546375]: 28015992..28015999 3 (2850360..2850367) 8 100000 126: [546376..546383]: 28016184..28016191 3 (2850552..2850559) 8 100000 127: [546384..546391]: 28016696..28016703 3 (2851064..2851071) 8 100000 128: [546392..546399]: 28017008..28017015 3 (2851376..2851383) 8 100000 129: [546400..546423]: 28015648..28015671 3 (2850016..2850039) 24 100000 130: [546424..546463]: 28015512..28015551 3 (2849880..2849919) 40 100000 131: [546464..546471]: 28015672..28015679 3 (2850040..2850047) 8 100000 132: [546472..546511]: 28015320..28015359 3 (2849688..2849727) 40 100000 133: [546512..546543]: 28015200..28015231 3 (2849568..2849599) 32 100000 134: [546544..546551]: 28014936..28014943 3 (2849304..2849311) 8 100000 135: [546552..546591]: 28014744..28014783 3 (2849112..2849151) 40 100000 136: [546592..546623]: 28014616..28014647 3 (2848984..2849015) 32 100000 137: [546624..546663]: 28013776..28013815 3 (2848144..2848183) 40 100000 138: [546664..546703]: 28013392..28013431 3 (2847760..2847799) 40 100000 139: [546704..546743]: 28013256..28013295 3 (2847624..2847663) 40 100000 140: [546744..546775]: 28013152..28013183 3 (2847520..2847551) 32 100000 141: [546776..546807]: 28013008..28013039 3 (2847376..2847407) 32 100000 142: [546808..546823]: 28012904..28012919 3 (2847272..2847287) 16 100000 143: [546824..546831]: 28013040..28013047 3 (2847408..2847415) 8 100000 144: [546832..546855]: 28012304..28012327 3 (2846672..2846695) 24 100000 145: [546856..546871]: 28012072..28012087 3 (2846440..2846455) 16 100000 146: [546872..546887]: 28014352..28014367 3 (2848720..2848735) 16 100000 147: [546888..546911]: 27322448..27322471 3 (2156816..2156839) 24 100000 148: [546912..546919]: 27322344..27322351 3 (2156712..2156719) 8 100000 149: [546920..546927]: 27321920..27321927 3 (2156288..2156295) 8 100000 150: [546928..546943]: 27324392..27324407 3 (2158760..2158775) 16 100000 151: [546944..546951]: 27324264..27324271 3 (2158632..2158639) 8 100000 152: [546952..546967]: 27334728..27334743 3 (2169096..2169111) 16 100000 153: [546968..546975]: 27334880..27334887 3 (2169248..2169255) 8 100000 154: [546976..546983]: 27334912..27334919 3 (2169280..2169287) 8 100000 155: [546984..546999]: 27342280..27342295 3 (2176648..2176663) 16 100000 156: [547000..547023]: 26091944..26091967 3 (926312..926335) 24 100000 157: [547024..547047]: 26080488..26080511 3 (914856..914879) 24 100000 158: [547048..547063]: 26072072..26072087 3 (906440..906455) 16 100000 159: [547064..547071]: 26070616..26070623 3 (904984..904991) 8 100000 160: [547072..547079]: 26077768..26077775 3 (912136..912143) 8 100000 161: [547080..547095]: 26038728..26038743 3 (873096..873111) 16 100000 162: [547096..547111]: 26038704..26038719 3 (873072..873087) 16 100000 163: [547112..547119]: 26038672..26038679 3 (873040..873047) 8 100000 164: [547120..547127]: 26038816..26038823 3 (873184..873191) 8 100000 165: [547128..547135]: 26038488..26038495 3 (872856..872863) 8 100000 166: [547136..547159]: 25910344..25910367 3 (744712..744735) 24 100000 167: [547160..547175]: 25935920..25935935 3 (770288..770303) 16 100000 168: [547176..547191]: 25941304..25941319 3 (775672..775687) 16 100000 169: [547192..547207]: 25884128..25884143 3 (718496..718511) 16 100000 170: [547208..547223]: 25998880..25998895 3 (833248..833263) 16 100000 171: [547224..547271]: 25169856..25169903 3 (4224..4271) 48 100000 172: [547272..547311]: 25169368..25169407 3 (3736..3775) 40 100000 173: [547312..547319]: 25169904..25169911 3 (4272..4279) 8 100000 174: [547320..547343]: 28769560..28769583 3 (3603928..3603951) 24 100000 175: [547344..547359]: 28769248..28769263 3 (3603616..3603631) 16 100000 176: [547360..547367]: 28785528..28785535 3 (3619896..3619903) 8 100000 177: [547368..547383]: 28813256..28813271 3 (3647624..3647639) 16 100000 178: [547384..547399]: 28832112..28832127 3 (3666480..3666495) 16 100000 179: [547400..547423]: 28874624..28874647 3 (3708992..3709015) 24 100000 180: [547424..547431]: 28835528..28835535 3 (3669896..3669903) 8 100000 181: [547432..547439]: 28832272..28832279 3 (3666640..3666647) 8 100000 182: [547440..547463]: 28932976..28932999 3 (3767344..3767367) 24 100000 183: [547464..547487]: 29268288..29268311 3 (4102656..4102679) 24 100000 184: [547488..547495]: 29234472..29234479 3 (4068840..4068847) 8 100000 185: [547496..547511]: 29308696..29308711 3 (4143064..4143079) 16 100000 186: [547512..547527]: 29430680..29430695 3 (4265048..4265063) 16 100000 187: [547528..547535]: 29459736..29459743 3 (4294104..4294111) 8 100000 188: [547536..547551]: 29520920..29520935 3 (4355288..4355303) 16 100000 189: [547552..547559]: 29497816..29497823 3 (4332184..4332191) 8 100000 190: [547560..547567]: 29459744..29459751 3 (4294112..4294119) 8 100000 191: [547568..559103]: hole 11536 192: [559104..559111]: 1747360..1747367 0 (1747360..1747367) 8 000000 193: [559112..564615]: 28029176..28034679 3 (2863544..2869047) 5504 100000 194: [564616..564639]: 1747752..1747775 0 (1747752..1747775) 24 000000 195: [564640..575487]: hole 10848 196: [575488..575495]: 28045552..28045559 3 (2879920..2879927) 8 100000 197: [575496..575551]: hole 56 198: [575552..575615]: 25166688..25166751 3 (1056..1119) 64 100000 199: [575616..575719]: 25167424..25167527 3 (1792..1895) 104 100000 200: [575720..575743]: hole 24 201: [575744..575951]: 31374264..31374471 3 (6208632..6208839) 208 100000 202: [575952..575999]: hole 48 203: [576000..576319]: 25166368..25166687 3 (736..1055) 320 100000 204: [576320..576383]: 31376408..31376471 3 (6210776..6210839) 64 100000 205: [576384..576463]: 25166752..25166831 3 (1120..1199) 80 100000 206: [576464..576511]: hole 48 207: [576512..576791]: 25166880..25167159 3 (1248..1527) 280 100000 208: [576792..576831]: hole 40 209: [576832..576879]: 25167528..25167575 3 (1896..1943) 48 100000 210: [576880..576895]: hole 16 211: [576896..576991]: 31374744..31374839 3 (6209112..6209207) 96 100000 212: [576992..577023]: hole 32 213: [577024..577287]: 25167160..25167423 3 (1528..1791) 264 100000 214: [577288..577343]: hole 56 215: [577344..577391]: 25166832..25166879 3 (1200..1247) 48 100000 216: [577392..577407]: hole 16 217: [577408..577511]: 31374840..31374943 3 (6209208..6209311) 104 100000 218: [577512..577535]: hole 24 219: [577536..577807]: 31374472..31374743 3 (6208840..6209111) 272 100000 220: [577808..577855]: hole 48 221: [577856..577903]: 31375960..31376007 3 (6210328..6210375) 48 100000 222: [577904..577919]: hole 16 223: [577920..578007]: 31375152..31375239 3 (6209520..6209607) 88 100000 224: [578008..578047]: hole 40 225: [578048..578215]: 31374984..31375151 3 (6209352..6209519) 168 100000 226: [578216..578239]: hole 24 227: [578240..578287]: 31376472..31376519 3 (6210840..6210887) 48 100000 228: [578288..578303]: hole 16 229: [578304..579023]: 31375240..31375959 3 (6209608..6210327) 720 100000 230: [579024..579071]: hole 48 231: [579072..579471]: 31376008..31376407 3 (6210376..6210775) 400 100000 232: [579472..579519]: hole 48 233: [579520..579567]: 31376784..31376831 3 (6211152..6211199) 48 100000 234: [579568..579583]: hole 16 235: [579584..579847]: 31376520..31376783 3 (6210888..6211151) 264 100000 236: [579848..579903]: hole 56 237: [579904..579959]: 31376840..31376895 3 (6211208..6211263) 56 100000 238: [579960..579967]: hole 8 239: [579968..580431]: 31376904..31377367 3 (6211272..6211735) 464 100000 240: [580432..580479]: hole 48 241: [580480..580567]: 31377416..31377503 3 (6211784..6211871) 88 100000 242: [580568..580607]: hole 40 243: [580608..580775]: 31377544..31377711 3 (6211912..6212079) 168 100000 244: [580776..580799]: hole 24 245: [580800..580847]: 31377736..31377783 3 (6212104..6212151) 48 100000 246: [580848..580863]: hole 16 247: [580864..581111]: 31377800..31378047 3 (6212168..6212415) 248 100000 248: [581112..581119]: hole 8 249: [581120..581391]: 31378056..31378327 3 (6212424..6212695) 272 100000 250: [581392..581439]: hole 48 251: [581440..581487]: 31378376..31378423 3 (6212744..6212791) 48 100000 252: [581488..581503]: hole 16 253: [581504..581583]: 31378440..31378519 3 (6212808..6212887) 80 100000 254: [581584..581631]: hole 48 255: [581632..581967]: 31378568..31378903 3 (6212936..6213271) 336 100000 256: [581968..582015]: hole 48 257: [582016..582103]: 31378952..31379039 3 (6213320..6213407) 88 100000 258: [582104..582143]: hole 40 259: [582144..582607]: 31379080..31379543 3 (6213448..6213911) 464 100000 260: [582608..582655]: hole 48 261: [582656..583015]: 31379592..31379951 3 (6213960..6214319) 360 100000 262: [583016..583039]: hole 24 263: [583040..583111]: 31379976..31380047 3 (6214344..6214415) 72 100000 264: [583112..583167]: hole 56 265: [583168..583431]: 31380104..31380367 3 (6214472..6214735) 264 100000 266: [583432..583487]: hole 56 267: [583488..583543]: 31380424..31380479 3 (6214792..6214847) 56 100000 268: [583544..583551]: hole 8 269: [583552..583631]: 31380488..31380567 3 (6214856..6214935) 80 100000 270: [583632..583679]: hole 48 271: [583680..583919]: 31380616..31380855 3 (6214984..6215223) 240 100000 272: [583920..583935]: hole 16 273: [583936..584183]: 31380872..31381119 3 (6215240..6215487) 248 100000 274: [584184..584191]: hole 8 275: [584192..584535]: 31381128..31381471 3 (6215496..6215839) 344 100000 276: [584536..584575]: hole 40 277: [584576..584679]: 31381512..31381615 3 (6215880..6215983) 104 100000 278: [584680..584703]: hole 24 279: [584704..584935]: 31381640..31381871 3 (6216008..6216239) 232 100000 280: [584936..584959]: hole 24 281: [584960..585135]: 31381896..31382071 3 (6216264..6216439) 176 100000 282: [585136..585151]: hole 16 283: [585152..585199]: 31382088..31382135 3 (6216456..6216503) 48 100000 284: [585200..585215]: hole 16 285: [585216..585575]: 31382152..31382511 3 (6216520..6216879) 360 100000 286: [585576..585599]: hole 24 287: [585600..586223]: 31382536..31383159 3 (6216904..6217527) 624 100000 288: [586224..586239]: hole 16 289: [586240..586519]: 31383176..31383455 3 (6217544..6217823) 280 100000 290: [586520..586559]: hole 40 291: [586560..586607]: 31383496..31383543 3 (6217864..6217911) 48 100000 292: [586608..586623]: hole 16 293: [586624..586735]: 31383560..31383671 3 (6217928..6218039) 112 100000 294: [586736..586751]: hole 16 295: [586752..587015]: 31383688..31383951 3 (6218056..6218319) 264 100000 296: [587016..587071]: hole 56 297: [587072..587119]: 31384008..31384055 3 (6218376..6218423) 48 100000 298: [587120..587135]: hole 16 299: [587136..587247]: 31384072..31384183 3 (6218440..6218551) 112 100000 300: [587248..587263]: hole 16 301: [587264..587695]: 31384200..31384631 3 (6218568..6218999) 432 100000 302: [587696..587711]: hole 16 303: [587712..587767]: 31384648..31384703 3 (6219016..6219071) 56 100000 304: [587768..587775]: hole 8 305: [587776..588039]: 31384712..31384975 3 (6219080..6219343) 264 100000 306: [588040..588095]: hole 56 307: [588096..588135]: 31385032..31385071 3 (6219400..6219439) 40 100000 308: [588136..588159]: hole 24 309: [588160..588231]: 31385096..31385167 3 (6219464..6219535) 72 100000 310: [588232..588287]: hole 56 311: [588288..588559]: 31385224..31385495 3 (6219592..6219863) 272 100000 312: [588560..588607]: hole 48 313: [588608..588663]: 31385544..31385599 3 (6219912..6219967) 56 100000 314: [588664..588671]: hole 8 315: [588672..588759]: 31385608..31385695 3 (6219976..6220063) 88 100000 316: [588760..588799]: hole 40 317: [588800..589111]: 31385736..31386047 3 (6220104..6220415) 312 100000 318: [589112..589183]: hole 72 319: [589184..589279]: 31386120..31386215 3 (6220488..6220583) 96 100000 320: [589280..589311]: hole 32 321: [589312..589487]: 31386248..31386423 3 (6220616..6220791) 176 100000 322: [589488..589567]: hole 80 323: [589568..589639]: 31386504..31386575 3 (6220872..6220943) 72 100000 324: [589640..589695]: hole 56 325: [589696..589775]: 31386632..31386711 3 (6221000..6221079) 80 100000 326: [589776..589823]: hole 48 327: [589824..589935]: 31386760..31386871 3 (6221128..6221239) 112 100000 328: [589936..590079]: hole 144 329: [590080..590279]: 31387016..31387215 3 (6221384..6221583) 200 100000 330: [590280..591871]: hole 1592 331: [591872..596551]: 31388808..31393487 3 (6223176..6227855) 4680 100000 332: [596552..596607]: hole 56 333: [596608..596687]: 31393544..31393623 3 (6227912..6227991) 80 100000 334: [596688..673791]: hole 77104 335: [673792..673807]: 28046088..28046103 3 (2880456..2880471) 16 100000 336: [673808..706559]: hole 32752 337: [706560..706575]: 28046400..28046415 3 (2880768..2880783) 16 100000 338: [706576..788479]: hole 81904 339: [788480..788487]: 1748384..1748391 0 (1748384..1748391) 8 000000 340: [788488..788511]: hole 24 341: [788512..788519]: 28046448..28046455 3 (2880816..2880823) 8 100000 342: [788520..788543]: hole 24 343: [788544..788551]: 28046104..28046111 3 (2880472..2880479) 8 100000 344: [788552..804927]: hole 16376 345: [804928..804943]: 28027432..28027447 3 (2861800..2861815) 16 100000 346: [804944..805887]: hole 944 347: [805888..805903]: 28046424..28046439 3 (2880792..2880807) 16 100000 348: [805904..806911]: hole 1008 349: [806912..806919]: 28004856..28004863 3 (2839224..2839231) 8 000000 350: [806920..821247]: hole 14328 351: [821248..821255]: 28027248..28027255 3 (2861616..2861623) 8 100000 352: [821256..1050623]: hole 229368 353: [1050624..1050879]: 28177528..28177783 3 (3011896..3012151) 256 100000 354: [1050880..1050911]: 28210272..28210303 3 (3044640..3044671) 32 100000 355: [1050912..1050943]: 28207112..28207143 3 (3041480..3041511) 32 100000 356: [1050944..1051071]: 28750936..28751063 3 (3585304..3585431) 128 100000 357: [1051072..1051095]: 28750808..28750831 3 (3585176..3585199) 24 100000 358: [1051096..1051135]: 28207296..28207335 3 (3041664..3041703) 40 100000 359: [1051136..1051199]: 28750664..28750727 3 (3585032..3585095) 64 100000 360: [1051200..1075135]: 28308744..28332679 3 (3143112..3167047) 23936 100000 361: [1075136..1075159]: 28210400..28210423 3 (3044768..3044791) 24 100000 362: [1075160..1075191]: 28207648..28207679 3 (3042016..3042047) 32 100000 363: [1075192..1075199]: 28207408..28207415 3 (3041776..3041783) 8 100000 364: [1075200..1075351]: 28791296..28791447 3 (3625664..3625815) 152 100000 365: [1075352..1075391]: 28207416..28207455 3 (3041784..3041823) 40 100000 366: [1075392..1075487]: 28769304..28769399 3 (3603672..3603767) 96 100000 367: [1075488..1075503]: 28207368..28207383 3 (3041736..3041751) 16 100000 368: [1075504..1075519]: 28207456..28207471 3 (3041824..3041839) 16 100000 369: [1075520..1075679]: 29680632..29680791 3 (4515000..4515159) 160 100000 370: [1075680..1075703]: 28207384..28207407 3 (3041752..3041775) 24 100000 371: [1075704..1075775]: 28785456..28785527 3 (3619824..3619895) 72 100000 372: [1075776..1075943]: 28207480..28207647 3 (3041848..3042015) 168 100000 373: [1075944..1078487]: 28207680..28210223 3 (3042048..3044591) 2544 100000 374: [1078488..1080511]: 28485704..28487727 3 (3320072..3322095) 2024 100000 375: [1080512..1080575]: 28762136..28762199 3 (3596504..3596567) 64 100000 376: [1080576..1083015]: 28487792..28490231 3 (3322160..3324599) 2440 100000 377: [1083016..1083031]: 28484008..28484023 3 (3318376..3318391) 16 100000 378: [1083032..1083063]: 28504544..28504575 3 (3338912..3338943) 32 100000 379: [1083064..1083071]: 28506008..28506015 3 (3340376..3340383) 8 100000 380: [1083072..1083135]: 28769624..28769687 3 (3603992..3604055) 64 100000 381: [1083136..1083719]: 28501600..28502183 3 (3335968..3336551) 584 100000 382: [1083720..1085695]: 28502568..28504543 3 (3336936..3338911) 1976 100000 383: [1085696..1087127]: 28504576..28506007 3 (3338944..3340375) 1432 100000 384: [1087128..1089735]: 28506016..28508623 3 (3340384..3342991) 2608 100000 385: [1089736..1089759]: 28479832..28479855 3 (3314200..3314223) 24 100000 386: [1089760..1089791]: 28613592..28613623 3 (3447960..3447991) 32 100000 387: [1089792..1089855]: 28626088..28626151 3 (3460456..3460519) 64 100000 388: [1089856..1094079]: 28621864..28626087 3 (3456232..3460455) 4224 100000 389: [1094080..1094143]: 28791488..28791551 3 (3625856..3625919) 64 100000 390: [1094144..1096639]: 28626152..28628647 3 (3460520..3463015) 2496 100000 391: [1096640..1096703]: 28806648..28806711 3 (3641016..3641079) 64 100000 392: [1096704..1098295]: 28628712..28630303 3 (3463080..3464671) 1592 100000 393: [1098296..1098415]: 28874344..28874463 3 (3708712..3708831) 120 100000 394: [1098416..1098431]: 28628648..28628663 3 (3463016..3463031) 16 100000 395: [1098432..1098559]: 28874496..28874623 3 (3708864..3708991) 128 100000 396: [1098560..1113407]: 28817200..28832047 3 (3651568..3666415) 14848 100000 397: [1113408..1113495]: 29308760..29308847 3 (4143128..4143215) 88 100000 398: [1113496..1113535]: 28832136..28832175 3 (3666504..3666543) 40 100000 399: [1113536..1113695]: 31249120..31249279 3 (6083488..6083647) 160 100000 400: [1113696..1113727]: 28832176..28832207 3 (3666544..3666575) 32 100000 401: [1113728..1113807]: 29459656..29459735 3 (4294024..4294103) 80 100000 402: [1113808..1113855]: 28832288..28832335 3 (3666656..3666703) 48 100000 403: [1113856..1113919]: 28902496..28902559 3 (3736864..3736927) 64 100000 404: [1113920..1116991]: 28832400..28835471 3 (3666768..3669839) 3072 100000 405: [1116992..1117055]: 28905568..28905631 3 (3739936..3739999) 64 100000 406: [1117056..1123319]: 28835536..28841799 3 (3669904..3676167) 6264 100000 407: [1123320..1123415]: 29478144..29478239 3 (4312512..4312607) 96 100000 408: [1123416..1123455]: 28841896..28841935 3 (3676264..3676303) 40 100000 409: [1123456..1123519]: 28932360..28932423 3 (3766728..3766791) 64 100000 410: [1123520..1124791]: 28842000..28843271 3 (3676368..3677639) 1272 100000 411: [1124792..1124799]: 28835512..28835519 3 (3669880..3669887) 8 100000 412: [1124800..1125679]: 28843280..28844159 3 (3677648..3678527) 880 100000 413: [1125680..1125687]: 28835520..28835527 3 (3669888..3669895) 8 100000 414: [1125688..1125695]: 28843272..28843279 3 (3677640..3677647) 8 100000 415: [1125696..1125759]: 28966864..28966927 3 (3801232..3801295) 64 100000 416: [1125760..1140223]: 28952400..28966863 3 (3786768..3801231) 14464 100000 417: [1140224..1140287]: 28978448..28978511 3 (3812816..3812879) 64 100000 418: [1140288..1144799]: 28966928..28971439 3 (3801296..3805807) 4512 100000 419: [1144800..1144919]: 29497696..29497815 3 (4332064..4332183) 120 100000 420: [1144920..1144959]: 28971560..28971599 3 (3805928..3805967) 40 100000 421: [1144960..1145055]: 29497568..29497663 3 (4331936..4332031) 96 100000 422: [1145056..1145087]: 28971696..28971727 3 (3806064..3806095) 32 100000 423: [1145088..1145215]: 29876856..29876983 3 (4711224..4711351) 128 100000 424: [1145216..1151807]: 28971856..28978447 3 (3806224..3812815) 6592 100000 425: [1151808..1151871]: 29004336..29004399 3 (3838704..3838767) 64 100000 426: [1151872..1154463]: 28978512..28981103 3 (3812880..3815471) 2592 100000 427: [1154464..1156095]: 29096888..29098519 3 (3931256..3932887) 1632 100000 428: [1156096..1156159]: 29071544..29071607 3 (3905912..3905975) 64 100000 429: [1156160..1281695]: 29098584..29224119 3 (3932952..4058487) 125536 100000 430: [1281696..1281719]: 29234416..29234439 3 (4068784..4068807) 24 100000 431: [1281720..1281727]: 29225136..29225143 3 (4059504..4059511) 8 100000 432: [1281728..1281759]: 29234440..29234471 3 (4068808..4068839) 32 100000 433: [1281760..1281887]: 30305336..30305463 3 (5139704..5139831) 128 100000 434: [1281888..1281919]: 29225200..29225231 3 (4059568..4059599) 32 100000 435: [1281920..1282007]: 29520648..29520735 3 (4355016..4355103) 88 100000 436: [1282008..1282031]: 29225232..29225255 3 (4059600..4059623) 24 100000 437: [1282032..1282183]: 31249280..31249431 3 (6083648..6083799) 152 100000 438: [1282184..1311615]: 29390296..29419727 3 (4224664..4254095) 29432 100000 439: [1311616..1311679]: 29437264..29437327 3 (4271632..4271695) 64 100000 440: [1311680..1319487]: 29419792..29427599 3 (4254160..4261967) 7808 100000 441: [1319488..1319551]: 29459792..29459855 3 (4294160..4294223) 64 100000 442: [1319552..1321087]: 29427664..29429199 3 (4262032..4263567) 1536 100000 443: [1321088..1321151]: 29470040..29470103 3 (4304408..4304471) 64 100000 444: [1321152..1322471]: 29429264..29430583 3 (4263632..4264951) 1320 100000 445: [1322472..1322583]: 30592744..30592855 3 (5427112..5427223) 112 100000 446: [1322584..1322623]: 29430696..29430735 3 (4265064..4265103) 40 100000 447: [1322624..1322751]: 30597960..30598087 3 (5432328..5432455) 128 100000 448: [1322752..1329151]: 29430864..29437263 3 (4265232..4271631) 6400 100000 449: [1329152..1329215]: 29520832..29520895 3 (4355200..4355263) 64 100000 450: [1329216..1351543]: 29437328..29459655 3 (4271696..4294023) 22328 100000 451: [1351544..1351639]: 29562360..29562455 3 (4396728..4396823) 96 100000 452: [1351640..1351679]: 29459752..29459791 3 (4294120..4294159) 40 100000 453: [1351680..1351743]: 29544208..29544271 3 (4378576..4378639) 64 100000 454: [1351744..1354975]: 29459856..29463087 3 (4294224..4297455) 3232 100000 455: [1354976..1367855]: 29530504..29543383 3 (4364872..4377751) 12880 100000 456: [1367856..1367943]: 29600904..29600991 3 (4435272..4435359) 88 100000 457: [1367944..1371791]: 29558424..29562271 3 (4392792..4396639) 3848 100000 458: [1371792..1371799]: 29562536..29562543 3 (4396904..4396911) 8 100000 459: [1371800..1371815]: 29562304..29562319 3 (4396672..4396687) 16 100000 460: [1371816..1371991]: 31249432..31249607 3 (6083800..6083975) 176 100000 461: [1371992..1372031]: 29562320..29562359 3 (4396688..4396727) 40 100000 462: [1372032..1372119]: 29626888..29626975 3 (4461256..4461343) 88 100000 463: [1372120..1372159]: 29562496..29562535 3 (4396864..4396903) 40 100000 464: [1372160..1372223]: 29577808..29577871 3 (4412176..4412239) 64 100000 465: [1372224..1465447]: 29683952..29777175 3 (4518320..4611543) 93224 100000 466: [1465448..1465599]: 31249608..31249759 3 (6083976..6084127) 152 100000 467: [1465600..1488519]: 29777328..29800247 3 (4611696..4634615) 22920 100000 468: [1488520..1488543]: 29843584..29843607 3 (4677952..4677975) 24 100000 469: [1488544..1488575]: 29808120..29808151 3 (4642488..4642519) 32 100000 470: [1488576..1488735]: 31249760..31249919 3 (6084128..6084287) 160 100000 471: [1488736..1488759]: 29808152..29808175 3 (4642520..4642543) 24 100000 472: [1488760..1488831]: 29843632..29843703 3 (4678000..4678071) 72 100000 473: [1488832..1546751]: 29999984..30057903 3 (4834352..4892271) 57920 100000 474: [1546752..1546815]: 29896544..29896607 3 (4730912..4730975) 64 100000 475: [1546816..1550783]: 30057968..30061935 3 (4892336..4896303) 3968 100000 476: [1550784..1550847]: 29894176..29894239 3 (4728544..4728607) 64 100000 477: [1550848..1554687]: 30062000..30065839 3 (4896368..4900207) 3840 100000 478: [1554688..1554711]: 30057944..30057967 3 (4892312..4892335) 24 100000 479: [1554712..1554751]: 30061936..30061975 3 (4896304..4896343) 40 100000 480: [1554752..1554839]: 29881688..29881775 3 (4716056..4716143) 88 100000 481: [1554840..1554879]: 30057904..30057943 3 (4892272..4892311) 40 100000 482: [1554880..1555007]: 30636536..30636663 3 (5470904..5471031) 128 100000 483: [1555008..1555031]: 30636496..30636519 3 (5470864..5470887) 24 100000 484: [1555032..1555071]: 29949536..29949575 3 (4783904..4783943) 40 100000 485: [1555072..1555175]: 31258192..31258295 3 (6092560..6092663) 104 100000 486: [1555176..1555199]: 29949576..29949599 3 (4783944..4783967) 24 100000 487: [1555200..1555351]: 31258296..31258447 3 (6092664..6092815) 152 100000 488: [1555352..1555391]: 29937232..29937271 3 (4771600..4771639) 40 100000 489: [1555392..1555479]: 31258448..31258535 3 (6092816..6092903) 88 100000 490: [1555480..1555519]: 29937272..29937311 3 (4771640..4771679) 40 100000 491: [1555520..1555679]: 31258536..31258695 3 (6092904..6093063) 160 100000 492: [1555680..1555711]: 29937056..29937087 3 (4771424..4771455) 32 100000 493: [1555712..1555775]: 31258696..31258759 3 (6093064..6093127) 64 100000 494: [1555776..1563975]: 30230928..30239127 3 (5065296..5073495) 8200 100000 495: [1563976..1563999]: 30305040..30305063 3 (5139408..5139431) 24 100000 496: [1564000..1564159]: 31258760..31258919 3 (6093128..6093287) 160 100000 497: [1564160..1569655]: 30327664..30333159 3 (5162032..5167527) 5496 100000 498: [1569656..1574823]: 30382208..30387375 3 (5216576..5221743) 5168 100000 499: [1574824..1574911]: 31258920..31259007 3 (6093288..6093375) 88 100000 500: [1574912..1576895]: 30387464..30389447 3 (5221832..5223815) 1984 100000 501: [1576896..1576959]: 31259008..31259071 3 (6093376..6093439) 64 100000 502: [1576960..1602815]: 30389512..30415367 3 (5223880..5249735) 25856 100000 503: [1602816..1602879]: 31259072..31259135 3 (6093440..6093503) 64 100000 504: [1602880..1609519]: 30415432..30422071 3 (5249800..5256439) 6640 100000 505: [1609520..1609607]: 31259136..31259223 3 (6093504..6093591) 88 100000 506: [1609608..1614783]: 30644032..30649207 3 (5478400..5483575) 5176 100000 507: [1614784..1614943]: 31259224..31259383 3 (6093592..6093751) 160 100000 508: [1614944..1614975]: 30649208..30649239 3 (5483576..5483607) 32 100000 509: [1614976..1615127]: 31259384..31259535 3 (6093752..6093903) 152 100000 510: [1615128..1615135]: 30649240..30649247 3 (5483608..5483615) 8 100000 511: [1615136..1615359]: 31259536..31259759 3 (6093904..6094127) 224 100000 512: [1615360..1617543]: 30649472..30651655 3 (5483840..5486023) 2184 100000 513: [1617544..1618647]: 30680296..30681399 3 (5514664..5515767) 1104 100000 514: [1618648..1628127]: 30681408..30690887 3 (5515776..5525255) 9480 100000 515: [1628128..1628183]: 30699688..30699743 3 (5534056..5534111) 56 100000 516: [1628184..1628207]: 30710824..30710847 3 (5545192..5545215) 24 100000 517: [1628208..1629063]: 30712840..30713695 3 (5547208..5548063) 856 100000 518: [1629064..1629071]: 30710864..30710871 3 (5545232..5545239) 8 100000 519: [1629072..1633703]: 30713704..30718335 3 (5548072..5552703) 4632 100000 520: [1633704..1633719]: 30710872..30710887 3 (5545240..5545255) 16 100000 521: [1633720..1633727]: 30713696..30713703 3 (5548064..5548071) 8 100000 522: [1633728..1633791]: 31259760..31259823 3 (6094128..6094191) 64 100000 523: [1633792..1636375]: 30763832..30766415 3 (5598200..5600783) 2584 100000 524: [1636376..1636407]: 30649424..30649455 3 (5483792..5483823) 32 100000 525: [1636408..1636415]: 30798144..30798151 3 (5632512..5632519) 8 100000 526: [1636416..1636543]: 31259824..31259951 3 (6094192..6094319) 128 100000 527: [1636544..1637719]: 30804752..30805927 3 (5639120..5640295) 1176 100000 528: [1637720..1640663]: 30806808..30809751 3 (5641176..5644119) 2944 100000 529: [1640664..1644943]: 30813400..30817679 3 (5647768..5652047) 4280 100000 530: [1644944..1644967]: 30657304..30657327 3 (5491672..5491695) 24 100000 531: [1644968..1644991]: 30818496..30818519 3 (5652864..5652887) 24 100000 532: [1644992..1645055]: 31259952..31260015 3 (6094320..6094383) 64 100000 533: [1645056..1663615]: 30999832..31018391 3 (5834200..5852759) 18560 100000 534: [1663616..1698367]: 31062368..31097119 3 (5896736..5931487) 34752 100000 535: [1698368..1698431]: 31315560..31315623 3 (6149928..6149991) 64 100000 536: [1698432..1698615]: 31097184..31097367 3 (5931552..5931735) 184 100000 537: [1698616..1698663]: 31315664..31315711 3 (6150032..6150079) 48 100000 538: [1698664..1698687]: 31097416..31097439 3 (5931784..5931807) 24 100000 539: [1698688..1698815]: 31359408..31359535 3 (6193776..6193903) 128 100000 540: [1698816..1702847]: 31097568..31101599 3 (5931936..5935967) 4032 100000 541: [1702848..1702911]: 31315800..31315863 3 (6150168..6150231) 64 100000 542: [1702912..1706311]: 31101664..31105063 3 (5936032..5939431) 3400 100000 543: [1706312..1706335]: 31315712..31315735 3 (6150080..6150103) 24 100000 544: [1706336..1706367]: 31105088..31105119 3 (5939456..5939487) 32 100000 545: [1706368..1706823]: 31359536..31359991 3 (6193904..6194359) 456 100000 546: [1706824..1707959]: 31105576..31106711 3 (5939944..5941079) 1136 100000 547: [1707960..1708039]: 31316064..31316143 3 (6150432..6150511) 80 100000 548: [1708040..1708095]: 31106792..31106847 3 (5941160..5941215) 56 100000 549: [1708096..1708159]: 31315944..31316007 3 (6150312..6150375) 64 100000 550: [1708160..1710423]: 31106912..31109175 3 (5941280..5943543) 2264 100000 551: [1710424..1710431]: 31111200..31111207 3 (5945568..5945575) 8 100000 552: [1710432..1710439]: 31106776..31106783 3 (5941144..5941151) 8 100000 553: [1710440..1712255]: 31173152..31174967 3 (6007520..6009335) 1816 100000 554: [1712256..1712383]: 31359992..31360119 3 (6194360..6194487) 128 100000 555: [1712384..1715527]: 31175096..31178239 3 (6009464..6012607) 3144 100000 556: [1715528..1715535]: 31178560..31178567 3 (6012928..6012935) 8 100000 557: [1715536..1715583]: 31265584..31265631 3 (6099952..6099999) 48 100000 558: [1715584..1716647]: 31360120..31361183 3 (6194488..6195551) 1064 100000 559: [1716648..1716655]: 31265632..31265639 3 (6100000..6100007) 8 100000 560: [1716656..1717519]: 31361184..31362047 3 (6195552..6196415) 864 100000 561: [1717520..1717567]: 31265640..31265687 3 (6100008..6100055) 48 100000 562: [1717568..1717703]: 31362048..31362183 3 (6196416..6196551) 136 100000 563: [1717704..1717751]: 31265824..31265871 3 (6100192..6100239) 48 100000 564: [1717752..1717839]: 31316472..31316559 3 (6150840..6150927) 88 100000 565: [1717840..1717871]: 31265960..31265991 3 (6100328..6100359) 32 100000 566: [1717872..1717967]: 31362184..31362279 3 (6196552..6196647) 96 100000 567: [1717968..1717999]: 31266088..31266119 3 (6100456..6100487) 32 100000 568: [1718000..1718103]: 31362280..31362383 3 (6196648..6196751) 104 100000 569: [1718104..1718135]: 31266224..31266255 3 (6100592..6100623) 32 100000 570: [1718136..1718215]: 31316192..31316271 3 (6150560..6150639) 80 100000 571: [1718216..1718255]: 31266336..31266375 3 (6100704..6100743) 40 100000 572: [1718256..1718503]: 31362384..31362631 3 (6196752..6196999) 248 100000 573: [1718504..1718527]: 31266376..31266399 3 (6100744..6100767) 24 100000 574: [1718528..1718695]: 31362632..31362799 3 (6197000..6197167) 168 100000 575: [1718696..1752799]: 31266568..31300671 3 (6100936..6135039) 34104 100000 576: [1752800..2073087]: hole 320288 577: [2073088..2073135]: 28177784..28177831 3 (3012152..3012199) 48 100000 578: [2073136..2073167]: hole 32 579: [2073168..2073191]: 28207144..28207167 3 (3041512..3041535) 24 100000 580: [2073192..2073375]: 30641224..30641407 3 (5475592..5475775) 184 100000 581: [2073376..2073407]: 28207168..28207199 3 (3041536..3041567) 32 100000 582: [2073408..2073543]: 28769112..28769247 3 (3603480..3603615) 136 100000 583: [2073544..2081879]: 28332680..28341015 3 (3167048..3175383) 8336 100000 584: [2081880..2082063]: 31105120..31105303 3 (5939488..5939671) 184 100000 585: [2082064..2082111]: 28210224..28210271 3 (3044592..3044639) 48 100000 586: [2082112..2082207]: 28841800..28841895 3 (3676168..3676263) 96 100000 587: [2082208..2082239]: 28210368..28210399 3 (3044736..3044767) 32 100000 588: [2082240..2082391]: 29039328..29039479 3 (3873696..3873847) 152 100000 589: [2082392..2082431]: 28027912..28027951 3 (2862280..2862319) 40 100000 590: [2082432..2082527]: 28883968..28884063 3 (3718336..3718431) 96 100000 591: [2082528..2082559]: 28027800..28027831 3 (2862168..2862199) 32 100000 592: [2082560..2082687]: 28971728..28971855 3 (3806096..3806223) 128 100000 593: [2082688..2085103]: 28490232..28492647 3 (3324600..3327015) 2416 100000 594: [2085104..2085127]: 28613640..28613663 3 (3448008..3448031) 24 100000 595: [2085128..2085183]: 28508624..28508679 3 (3342992..3343047) 56 100000 596: [2085184..2085247]: 28808192..28808255 3 (3642560..3642623) 64 100000 597: [2085248..2160167]: 28508744..28583663 3 (3343112..3418031) 74920 100000 598: [2160168..2160175]: 28613664..28613671 3 (3448032..3448039) 8 100000 599: [2160176..2160191]: 28613624..28613639 3 (3447992..3448007) 16 100000 600: [2160192..2160255]: 28832048..28832111 3 (3666416..3666479) 64 100000 601: [2160256..2170559]: 28630304..28640607 3 (3464672..3474975) 10304 100000 602: [2170560..2170623]: 28832208..28832271 3 (3666576..3666639) 64 100000 603: [2170624..2173015]: 28640672..28643063 3 (3475040..3477431) 2392 100000 604: [2173016..2173207]: 31105304..31105495 3 (5939672..5939863) 192 100000 605: [2173208..2173247]: 28640608..28640647 3 (3474976..3475015) 40 100000 606: [2173248..2173375]: 29308568..29308695 3 (4142936..4143063) 128 100000 607: [2173376..2203559]: 28844160..28874343 3 (3678528..3708711) 30184 100000 608: [2203560..2203679]: 31097440..31097559 3 (5931808..5931927) 120 100000 609: [2203680..2203711]: 28874464..28874495 3 (3708832..3708863) 32 100000 610: [2203712..2203863]: 31249920..31250071 3 (6084288..6084439) 152 100000 611: [2203864..2203903]: 28874648..28874687 3 (3709016..3709055) 40 100000 612: [2203904..2203967]: 29021392..29021455 3 (3855760..3855823) 64 100000 613: [2203968..2206847]: 28874752..28877631 3 (3709120..3711999) 2880 100000 614: [2206848..2206911]: 29023504..29023567 3 (3857872..3857935) 64 100000 615: [2206912..2209639]: 28877696..28880423 3 (3712064..3714791) 2728 100000 616: [2209640..2209727]: 29680520..29680607 3 (4514888..4514975) 88 100000 617: [2209728..2213183]: 28880512..28883967 3 (3714880..3718335) 3456 100000 618: [2213184..2213311]: 31174968..31175095 3 (6009336..6009463) 128 100000 619: [2213312..2216871]: 28884096..28887655 3 (3718464..3722023) 3560 100000 620: [2216872..2216959]: 29876728..29876815 3 (4711096..4711183) 88 100000 621: [2216960..2240047]: 28981104..29004191 3 (3815472..3838559) 23088 100000 622: [2240048..2240135]: 30305208..30305295 3 (5139576..5139663) 88 100000 623: [2240136..2240159]: 29004280..29004303 3 (3838648..3838671) 24 100000 624: [2240160..2240415]: 31250072..31250327 3 (6084440..6084695) 256 100000 625: [2240416..2240447]: 29004304..29004335 3 (3838672..3838703) 32 100000 626: [2240448..2240511]: 29039520..29039583 3 (3873888..3873951) 64 100000 627: [2240512..2250055]: 29004400..29013943 3 (3838768..3848311) 9544 100000 628: [2250056..2251071]: 29224120..29225135 3 (4058488..4059503) 1016 100000 629: [2251072..2251079]: 29225256..29225263 3 (4059624..4059631) 8 100000 630: [2251080..2251135]: 29225144..29225199 3 (4059512..4059567) 56 100000 631: [2251136..2251199]: 29258352..29258415 3 (4092720..4092783) 64 100000 632: [2251200..2260351]: 29225264..29234415 3 (4059632..4068783) 9152 100000 633: [2260352..2260415]: 29314904..29314967 3 (4149272..4149335) 64 100000 634: [2260416..2284287]: 29234480..29258351 3 (4068848..4092719) 23872 100000 635: [2284288..2284351]: 29053368..29053431 3 (3887736..3887799) 64 100000 636: [2284352..2290111]: 29258416..29264175 3 (4092784..4098543) 5760 100000 637: [2290112..2290143]: 29520736..29520767 3 (4355104..4355135) 32 100000 638: [2290144..2290175]: 29268104..29268135 3 (4102472..4102503) 32 100000 639: [2290176..2290327]: 31250328..31250479 3 (6084696..6084847) 152 100000 640: [2290328..2290367]: 29268248..29268287 3 (4102616..4102655) 40 100000 641: [2290368..2290463]: 30641080..30641175 3 (5475448..5475543) 96 100000 642: [2290464..2290495]: 29268136..29268167 3 (4102504..4102535) 32 100000 643: [2290496..2290655]: 31250480..31250639 3 (6084848..6085007) 160 100000 644: [2290656..2290687]: 29268168..29268199 3 (4102536..4102567) 32 100000 645: [2290688..2290839]: 31250640..31250791 3 (6085008..6085159) 152 100000 646: [2290840..2297791]: 29463088..29470039 3 (4297456..4304407) 6952 100000 647: [2297792..2297855]: 29596184..29596247 3 (4430552..4430615) 64 100000 648: [2297856..2305895]: 29470104..29478143 3 (4304472..4312511) 8040 100000 649: [2305896..2305991]: 31250792..31250887 3 (6085160..6085255) 96 100000 650: [2305992..2306023]: 29478240..29478271 3 (4312608..4312639) 32 100000 651: [2306024..2306207]: 31250888..31251071 3 (6085256..6085439) 184 100000 652: [2306208..2306239]: 29478272..29478303 3 (4312640..4312671) 32 100000 653: [2306240..2306367]: 31251072..31251199 3 (6085440..6085567) 128 100000 654: [2306368..2325503]: 29478432..29497567 3 (4312800..4331935) 19136 100000 655: [2325504..2325599]: 31251200..31251295 3 (6085568..6085663) 96 100000 656: [2325600..2325631]: 29497664..29497695 3 (4332032..4332063) 32 100000 657: [2325632..2325759]: 31251296..31251423 3 (6085664..6085791) 128 100000 658: [2325760..2348583]: 29497824..29520647 3 (4332192..4355015) 22824 100000 659: [2348584..2348703]: 31251424..31251543 3 (6085792..6085911) 120 100000 660: [2348704..2348735]: 29520768..29520799 3 (4355136..4355167) 32 100000 661: [2348736..2348871]: 31251544..31251679 3 (6085912..6086047) 136 100000 662: [2348872..2349663]: 29520936..29521727 3 (4355304..4356095) 792 100000 663: [2349664..2349687]: 29520896..29520919 3 (4355264..4355287) 24 100000 664: [2349688..2349695]: 29520824..29520831 3 (4355192..4355199) 8 100000 665: [2349696..2349823]: 31251680..31251807 3 (6086048..6086175) 128 100000 666: [2349824..2357695]: 29800248..29808119 3 (4634616..4642487) 7872 100000 667: [2357696..2357759]: 29846904..29846967 3 (4681272..4681335) 64 100000 668: [2357760..2393159]: 29808184..29843583 3 (4642552..4677951) 35400 100000 669: [2393160..2393183]: 29846664..29846687 3 (4681032..4681055) 24 100000 670: [2393184..2393207]: 29843608..29843631 3 (4677976..4677999) 24 100000 671: [2393208..2393279]: 30597672..30597743 3 (5432040..5432111) 72 100000 672: [2393280..2396239]: 29843704..29846663 3 (4678072..4681031) 2960 100000 673: [2396240..2396271]: 29850296..29850327 3 (4684664..4684695) 32 100000 674: [2396272..2396287]: 29846696..29846711 3 (4681064..4681079) 16 100000 675: [2396288..2396431]: 31251808..31251951 3 (6086176..6086319) 144 100000 676: [2396432..2396479]: 29846856..29846903 3 (4681224..4681271) 48 100000 677: [2396480..2396543]: 29881824..29881887 3 (4716192..4716255) 64 100000 678: [2396544..2399871]: 29846968..29850295 3 (4681336..4684663) 3328 100000 679: [2399872..2399935]: 29644952..29645015 3 (4479320..4479383) 64 100000 680: [2399936..2426303]: 29850360..29876727 3 (4684728..4711095) 26368 100000 681: [2426304..2426391]: 30597832..30597919 3 (5432200..5432287) 88 100000 682: [2426392..2426431]: 29876816..29876855 3 (4711184..4711223) 40 100000 683: [2426432..2426559]: 31251952..31252079 3 (6086320..6086447) 128 100000 684: [2426560..2430975]: 29876984..29881399 3 (4711352..4715767) 4416 100000 685: [2430976..2437983]: 30065840..30072847 3 (4900208..4907215) 7008 100000 686: [2437984..2437999]: 29949600..29949615 3 (4783968..4783983) 16 100000 687: [2438000..2438015]: 30061976..30061991 3 (4896344..4896359) 16 100000 688: [2438016..2438111]: 31252080..31252175 3 (6086448..6086543) 96 100000 689: [2438112..2438143]: 29937088..29937119 3 (4771456..4771487) 32 100000 690: [2438144..2438295]: 31252176..31252327 3 (6086544..6086695) 152 100000 691: [2438296..2438335]: 29919328..29919367 3 (4753696..4753735) 40 100000 692: [2438336..2438399]: 29627032..29627095 3 (4461400..4461463) 64 100000 693: [2438400..2438431]: 30641040..30641071 3 (5475408..5475439) 32 100000 694: [2438432..2438463]: 29919368..29919399 3 (4753736..4753767) 32 100000 695: [2438464..2438615]: 31260016..31260167 3 (6094384..6094535) 152 100000 696: [2438616..2438655]: 29919400..29919439 3 (4753768..4753807) 40 100000 697: [2438656..2438751]: 31260168..31260263 3 (6094536..6094631) 96 100000 698: [2438752..2438783]: 29919200..29919231 3 (4753568..4753599) 32 100000 699: [2438784..2438935]: 31260264..31260415 3 (6094632..6094783) 152 100000 700: [2438936..2438975]: 29919232..29919271 3 (4753600..4753639) 40 100000 701: [2438976..2439071]: 31260416..31260511 3 (6094784..6094879) 96 100000 702: [2439072..2439103]: 29937120..29937151 3 (4771488..4771519) 32 100000 703: [2439104..2439231]: 31260512..31260639 3 (6094880..6095007) 128 100000 704: [2439232..2482527]: 30239128..30282423 3 (5073496..5116791) 43296 100000 705: [2482528..2482543]: 30317784..30317799 3 (5152152..5152167) 16 100000 706: [2482544..2482559]: 30305064..30305079 3 (5139432..5139447) 16 100000 707: [2482560..2482687]: 31260640..31260767 3 (6095008..6095135) 128 100000 708: [2482688..2521871]: 30333160..30372343 3 (5167528..5206711) 39184 100000 709: [2521872..2548351]: 30422072..30448551 3 (5256440..5282919) 26480 100000 710: [2548352..2548415]: 31260768..31260831 3 (6095136..6095199) 64 100000 711: [2548416..2577087]: 30448616..30477287 3 (5282984..5311655) 28672 100000 712: [2577088..2577151]: 31260832..31260895 3 (6095200..6095263) 64 100000 713: [2577152..2579527]: 30477352..30479727 3 (5311720..5314095) 2376 100000 714: [2579528..2579567]: 31097368..31097407 3 (5931736..5931775) 40 100000 715: [2579568..2579583]: 30479768..30479783 3 (5314136..5314151) 16 100000 716: [2579584..2579647]: 31260896..31260959 3 (6095264..6095327) 64 100000 717: [2579648..2580479]: 30479848..30480679 3 (5314216..5315047) 832 100000 718: [2580480..2584383]: 30651656..30655559 3 (5486024..5489927) 3904 100000 719: [2584384..2584447]: 31260960..31261023 3 (6095328..6095391) 64 100000 720: [2584448..2586127]: 30655624..30657303 3 (5489992..5491671) 1680 100000 721: [2586128..2586151]: 30641408..30641431 3 (5475776..5475799) 24 100000 722: [2586152..2586175]: 30657328..30657351 3 (5491696..5491719) 24 100000 723: [2586176..2586263]: 31261024..31261111 3 (6095392..6095479) 88 100000 724: [2586264..2586303]: 30657440..30657479 3 (5491808..5491847) 40 100000 725: [2586304..2586463]: 31261112..31261271 3 (6095480..6095639) 160 100000 726: [2586464..2586495]: 30657480..30657511 3 (5491848..5491879) 32 100000 727: [2586496..2586583]: 31261272..31261359 3 (6095640..6095727) 88 100000 728: [2586584..2586623]: 30657600..30657639 3 (5491968..5492007) 40 100000 729: [2586624..2586751]: 31261360..31261487 3 (6095728..6095855) 128 100000 730: [2586752..2588359]: 30657928..30659535 3 (5492296..5493903) 1608 100000 731: [2588360..2589567]: 30665792..30666999 3 (5500160..5501367) 1208 100000 732: [2589568..2595063]: 30690888..30696383 3 (5525256..5530751) 5496 100000 733: [2595064..2600079]: 30718336..30723351 3 (5552704..5557719) 5016 100000 734: [2600080..2600111]: 31056816..31056847 3 (5891184..5891215) 32 100000 735: [2600112..2600127]: 30710848..30710863 3 (5545216..5545231) 16 100000 736: [2600128..2600191]: 31261488..31261551 3 (6095856..6095919) 64 100000 737: [2600192..2611183]: 30766416..30777407 3 (5600784..5611775) 10992 100000 738: [2611184..2611191]: 30809808..30809815 3 (5644176..5644183) 8 100000 739: [2611192..2611199]: 30809752..30809759 3 (5644120..5644127) 8 100000 740: [2611200..2611263]: 31261552..31261615 3 (6095920..6095983) 64 100000 741: [2611264..2614839]: 30809824..30813399 3 (5644192..5647767) 3576 100000 742: [2614840..2614847]: 30809760..30809767 3 (5644128..5644135) 8 100000 743: [2614848..2614863]: 30833816..30833831 3 (5668184..5668199) 16 100000 744: [2614864..2615679]: 30817680..30818495 3 (5652048..5652863) 816 100000 745: [2615680..2615743]: 31261616..31261679 3 (6095984..6096047) 64 100000 746: [2615744..2626783]: 30818560..30829599 3 (5652928..5663967) 11040 100000 747: [2626784..2626807]: 31041760..31041783 3 (5876128..5876151) 24 100000 748: [2626808..2626815]: 30833808..30833815 3 (5668176..5668183) 8 100000 749: [2626816..2626879]: 31261680..31261743 3 (6096048..6096111) 64 100000 750: [2626880..2626903]: 31315736..31315759 3 (6150104..6150127) 24 100000 751: [2626904..2626943]: 30818520..30818559 3 (5652888..5652927) 40 100000 752: [2626944..2627095]: 31362800..31362951 3 (6197168..6197319) 152 100000 753: [2627096..2627135]: 30809768..30809807 3 (5644136..5644175) 40 100000 754: [2627136..2627199]: 31316320..31316383 3 (6150688..6150751) 64 100000 755: [2627200..2629223]: 31109176..31111199 3 (5943544..5945567) 2024 100000 756: [2629224..2629231]: 31106784..31106791 3 (5941152..5941159) 8 100000 757: [2629232..2632831]: 31111208..31114807 3 (5945576..5949175) 3600 100000 758: [2632832..2632895]: 31362952..31363015 3 (6197320..6197383) 64 100000 759: [2632896..2637823]: 31114872..31119799 3 (5949240..5954167) 4928 100000 760: [2637824..2638287]: 31363016..31363479 3 (6197384..6197847) 464 100000 761: [2638288..2648575]: 31120264..31130551 3 (5954632..5964919) 10288 100000 762: [2648576..2648639]: 31363480..31363543 3 (6197848..6197911) 64 100000 763: [2648640..2655167]: 31130616..31137143 3 (5964984..5971511) 6528 100000 764: [2655168..2655207]: 31363544..31363583 3 (6197912..6197951) 40 100000 765: [2655208..2655231]: 31265688..31265711 3 (6100056..6100079) 24 100000 766: [2655232..2656399]: 31363584..31364751 3 (6197952..6199119) 1168 100000 767: [2656400..2656415]: 31265712..31265727 3 (6100080..6100095) 16 100000 768: [2656416..2656583]: 31364752..31364919 3 (6199120..6199287) 168 100000 769: [2656584..2656615]: 31265728..31265759 3 (6100096..6100127) 32 100000 770: [2656616..2656719]: 31364920..31365023 3 (6199288..6199391) 104 100000 771: [2656720..2656735]: 31265760..31265775 3 (6100128..6100143) 16 100000 772: [2656736..2657423]: 31365024..31365711 3 (6199392..6200079) 688 100000 773: [2657424..2657455]: 31265776..31265807 3 (6100144..6100175) 32 100000 774: [2657456..2657831]: 31365712..31366087 3 (6200080..6200455) 376 100000 775: [2657832..2657855]: 31265872..31265895 3 (6100240..6100263) 24 100000 776: [2657856..2658055]: 31366088..31366287 3 (6200456..6200655) 200 100000 777: [2658056..2658103]: 31265896..31265943 3 (6100264..6100311) 48 100000 778: [2658104..2658183]: 31366288..31366367 3 (6200656..6200735) 80 100000 779: [2658184..2658231]: 31266024..31266071 3 (6100392..6100439) 48 100000 780: [2658232..2658319]: 31366368..31366455 3 (6200736..6200823) 88 100000 781: [2658320..2658351]: 31266160..31266191 3 (6100528..6100559) 32 100000 782: [2658352..2658447]: 31366456..31366551 3 (6200824..6200919) 96 100000 783: [2658448..2658487]: 31266288..31266327 3 (6100656..6100695) 40 100000 784: [2658488..2658567]: 31366552..31366631 3 (6200920..6200999) 80 100000 785: [2658568..2658607]: 31266408..31266447 3 (6100776..6100815) 40 100000 786: [2658608..2658735]: 31366632..31366759 3 (6201000..6201127) 128 100000 787: [2658736..2658751]: 31266448..31266463 3 (6100816..6100831) 16 100000 788: [2658752..2658903]: 31366760..31366911 3 (6201128..6201279) 152 100000 789: [2658904..2658935]: 31266464..31266495 3 (6100832..6100863) 32 100000 790: [2658936..2659007]: 31366912..31366983 3 (6201280..6201351) 72 100000 791: [2659008..2673863]: 31300672..31315527 3 (6135040..6149895) 14856 100000 792: [2673864..3095551]: hole 421688 793: [3095552..3095599]: 28177832..28177879 3 (3012200..3012247) 48 100000 794: [3095600..3146439]: 28046456..28097295 3 (2880824..2931663) 50840 100000 795: [3146440..3226703]: hole 80264 796: [3226704..3226727]: 28207200..28207223 3 (3041568..3041591) 24 100000 797: [3226728..3226903]: 30649248..30649423 3 (5483616..5483791) 176 100000 798: [3226904..3226943]: 28207224..28207263 3 (3041592..3041631) 40 100000 799: [3226944..3227071]: 28769432..28769559 3 (3603800..3603927) 128 100000 800: [3227072..3281327]: 28341016..28395271 3 (3175384..3229639) 54256 100000 801: [3281328..3281351]: 28628664..28628687 3 (3463032..3463055) 24 100000 802: [3281352..3281383]: 28478696..28478727 3 (3313064..3313095) 32 100000 803: [3281384..3281407]: 28478864..28478887 3 (3313232..3313255) 24 100000 804: [3281408..3281495]: 28880424..28880511 3 (3714792..3714879) 88 100000 805: [3281496..3281519]: 28478976..28478999 3 (3313344..3313367) 24 100000 806: [3281520..3281535]: 28478888..28478903 3 (3313256..3313271) 16 100000 807: [3281536..3281687]: 29621592..29621743 3 (4455960..4456111) 152 100000 808: [3281688..3281719]: 28479000..28479031 3 (3313368..3313399) 32 100000 809: [3281720..3281727]: 28478904..28478911 3 (3313272..3313279) 8 100000 810: [3281728..3281983]: 31119800..31120055 3 (5954168..5954423) 256 100000 811: [3281984..3282119]: 28478728..28478863 3 (3313096..3313231) 136 100000 812: [3282120..3282879]: 28479032..28479791 3 (3313400..3314159) 760 100000 813: [3282880..3282943]: 28832336..28832399 3 (3666704..3666767) 64 100000 814: [3282944..3285215]: 28479856..28482127 3 (3314224..3316495) 2272 100000 815: [3285216..3285255]: 28484040..28484079 3 (3318408..3318447) 40 100000 816: [3285256..3285311]: 28487728..28487783 3 (3322096..3322151) 56 100000 817: [3285312..3285327]: 28484080..28484095 3 (3318448..3318463) 16 100000 818: [3285328..3291143]: 28492648..28498463 3 (3327016..3332831) 5816 100000 819: [3291144..3291183]: 28835472..28835511 3 (3669840..3669879) 40 100000 820: [3291184..3291199]: 28508680..28508695 3 (3343048..3343063) 16 100000 821: [3291200..3291327]: 29308888..29309015 3 (4143256..4143383) 128 100000 822: [3291328..3321255]: 28583664..28613591 3 (3418032..3447959) 29928 100000 823: [3321256..3321343]: 28971440..28971527 3 (3805808..3805895) 88 100000 824: [3321344..3326935]: 28613680..28619271 3 (3448048..3453639) 5592 100000 825: [3326936..3326943]: 28613672..28613679 3 (3448040..3448047) 8 100000 826: [3326944..3434543]: 28643064..28750663 3 (3477432..3585031) 107600 100000 827: [3434544..3434639]: 28971600..28971695 3 (3805968..3806063) 96 100000 828: [3434640..3434687]: 28750760..28750807 3 (3585128..3585175) 48 100000 829: [3434688..3434775]: 29004192..29004279 3 (3838560..3838647) 88 100000 830: [3434776..3434815]: 28750896..28750935 3 (3585264..3585303) 40 100000 831: [3434816..3434943]: 29430736..29430863 3 (4265104..4265231) 128 100000 832: [3434944..3446015]: 28751064..28762135 3 (3585432..3596503) 11072 100000 833: [3446016..3446079]: 28841936..28841999 3 (3676304..3676367) 64 100000 834: [3446080..3452991]: 28762200..28769111 3 (3596568..3603479) 6912 100000 835: [3452992..3453143]: 31252328..31252479 3 (6086696..6086847) 152 100000 836: [3453144..3453183]: 28769264..28769303 3 (3603632..3603671) 40 100000 837: [3453184..3453279]: 31252480..31252575 3 (6086848..6086943) 96 100000 838: [3453280..3453311]: 28769400..28769431 3 (3603768..3603799) 32 100000 839: [3453312..3453463]: 31252576..31252727 3 (6086944..6087095) 152 100000 840: [3453464..3453503]: 28769584..28769623 3 (3603952..3603991) 40 100000 841: [3453504..3453567]: 29621784..29621847 3 (4456152..4456215) 64 100000 842: [3453568..3465863]: 28769688..28781983 3 (3604056..3616351) 12296 100000 843: [3465864..3471295]: 28887656..28893087 3 (3722024..3727455) 5432 100000 844: [3471296..3471359]: 30309432..30309495 3 (5143800..5143863) 64 100000 845: [3471360..3480703]: 28893152..28902495 3 (3727520..3736863) 9344 100000 846: [3480704..3480767]: 30317824..30317887 3 (5152192..5152255) 64 100000 847: [3480768..3483775]: 28902560..28905567 3 (3736928..3739935) 3008 100000 848: [3483776..3483839]: 30389448..30389511 3 (5223816..5223879) 64 100000 849: [3483840..3486695]: 28905632..28908487 3 (3740000..3742855) 2856 100000 850: [3486696..3494143]: 29013944..29021391 3 (3848312..3855759) 7448 100000 851: [3494144..3494207]: 30415368..30415431 3 (5249736..5249799) 64 100000 852: [3494208..3496255]: 29021456..29023503 3 (3855824..3857871) 2048 100000 853: [3496256..3496319]: 30448552..30448615 3 (5282920..5282983) 64 100000 854: [3496320..3512055]: 29023568..29039303 3 (3857936..3873671) 15736 100000 855: [3512056..3512167]: 31252728..31252839 3 (6087096..6087207) 112 100000 856: [3512168..3512191]: 29039304..29039327 3 (3873672..3873695) 24 100000 857: [3512192..3512343]: 31252840..31252991 3 (6087208..6087359) 152 100000 858: [3512344..3512383]: 29039480..29039519 3 (3873848..3873887) 40 100000 859: [3512384..3512447]: 30477288..30477351 3 (5311656..5311719) 64 100000 860: [3512448..3520727]: 29039584..29047863 3 (3873952..3882231) 8280 100000 861: [3520728..3524655]: 29264176..29268103 3 (4098544..4102471) 3928 100000 862: [3524656..3524751]: 31252992..31253087 3 (6087360..6087455) 96 100000 863: [3524752..3524799]: 29268200..29268247 3 (4102568..4102615) 48 100000 864: [3524800..3524863]: 30479784..30479847 3 (5314152..5314215) 64 100000 865: [3524864..3552607]: 29268312..29296055 3 (4102680..4130423) 27744 100000 866: [3552608..3552767]: 31253088..31253247 3 (6087456..6087615) 160 100000 867: [3552768..3564919]: 29296216..29308367 3 (4130584..4142735) 12152 100000 868: [3564920..3564951]: 28971528..28971559 3 (3805896..3805927) 32 100000 869: [3564952..3564991]: 29308400..29308439 3 (4142768..4142807) 40 100000 870: [3564992..3565079]: 31041592..31041679 3 (5875960..5876047) 88 100000 871: [3565080..3565119]: 29308528..29308567 3 (4142896..4142935) 40 100000 872: [3565120..3565271]: 31253248..31253399 3 (6087616..6087767) 152 100000 873: [3565272..3565311]: 29308720..29308759 3 (4143088..4143127) 40 100000 874: [3565312..3565399]: 31253400..31253487 3 (6087768..6087855) 88 100000 875: [3565400..3565439]: 29308848..29308887 3 (4143216..4143255) 40 100000 876: [3565440..3565567]: 31253488..31253615 3 (6087856..6087983) 128 100000 877: [3565568..3571455]: 29309016..29314903 3 (4143384..4149271) 5888 100000 878: [3571456..3571519]: 30498920..30498983 3 (5333288..5333351) 64 100000 879: [3571520..3629695]: 29314968..29373143 3 (4149336..4207511) 58176 100000 880: [3629696..3629727]: 28947432..28947463 3 (3781800..3781831) 32 100000 881: [3629728..3629759]: 29380744..29380775 3 (4215112..4215143) 32 100000 882: [3629760..3629919]: 31253616..31253775 3 (6087984..6088143) 160 100000 883: [3629920..3629951]: 29380776..29380807 3 (4215144..4215175) 32 100000 884: [3629952..3630039]: 31253776..31253863 3 (6088144..6088231) 88 100000 885: [3630040..3630079]: 29419728..29419767 3 (4254096..4254135) 40 100000 886: [3630080..3630143]: 30600888..30600951 3 (5435256..5435319) 64 100000 887: [3630144..3638919]: 29521728..29530503 3 (4356096..4364871) 8776 100000 888: [3638920..3638943]: 29596112..29596135 3 (4430480..4430503) 24 100000 889: [3638944..3638967]: 29520800..29520823 3 (4355168..4355191) 24 100000 890: [3638968..3639111]: 31253864..31254007 3 (6088232..6088375) 144 100000 891: [3639112..3639935]: 29543384..29544207 3 (4377752..4378575) 824 100000 892: [3639936..3639999]: 30631928..30631991 3 (5466296..5466359) 64 100000 893: [3640000..3654151]: 29544272..29558423 3 (4378640..4392791) 14152 100000 894: [3654152..3654175]: 29621536..29621559 3 (4455904..4455927) 24 100000 895: [3654176..3654207]: 29562272..29562303 3 (4396640..4396671) 32 100000 896: [3654208..3654359]: 31254008..31254159 3 (6088376..6088527) 152 100000 897: [3654360..3654399]: 29562456..29562495 3 (4396824..4396863) 40 100000 898: [3654400..3654463]: 30655560..30655623 3 (5489928..5489991) 64 100000 899: [3654464..3661527]: 29562560..29569623 3 (4396928..4403991) 7064 100000 900: [3661528..3661815]: 29881400..29881687 3 (4715768..4716055) 288 100000 901: [3661816..3661911]: 31254160..31254255 3 (6088528..6088623) 96 100000 902: [3661912..3661951]: 29881784..29881823 3 (4716152..4716191) 40 100000 903: [3661952..3662015]: 31046584..31046647 3 (5880952..5881015) 64 100000 904: [3662016..3674303]: 29881888..29894175 3 (4716256..4728543) 12288 100000 905: [3674304..3674367]: 31097120..31097183 3 (5931488..5931551) 64 100000 906: [3674368..3676671]: 29894240..29896543 3 (4728608..4730911) 2304 100000 907: [3676672..3676735]: 31101600..31101663 3 (5935968..5936031) 64 100000 908: [3676736..3699327]: 29896608..29919199 3 (4730976..4753567) 22592 100000 909: [3699328..3699415]: 31254256..31254343 3 (6088624..6088711) 88 100000 910: [3699416..3699455]: 29919288..29919327 3 (4753656..4753695) 40 100000 911: [3699456..3699583]: 31254344..31254471 3 (6088712..6088839) 128 100000 912: [3699584..3717183]: 29919456..29937055 3 (4753824..4771423) 17600 100000 913: [3717184..3717279]: 31254472..31254567 3 (6088840..6088935) 96 100000 914: [3717280..3717311]: 29937152..29937183 3 (4771520..4771551) 32 100000 915: [3717312..3717439]: 31254568..31254695 3 (6088936..6089063) 128 100000 916: [3717440..3721079]: 29937312..29940951 3 (4771680..4775319) 3640 100000 917: [3721080..3721103]: 29850328..29850351 3 (4684696..4684719) 24 100000 918: [3721104..3721151]: 29937184..29937231 3 (4771552..4771599) 48 100000 919: [3721152..3721215]: 31105496..31105559 3 (5939864..5939927) 64 100000 920: [3721216..3840839]: 30072848..30192471 3 (4907216..5026839) 119624 100000 921: [3840840..3861319]: 30282424..30302903 3 (5116792..5137271) 20480 100000 922: [3861320..3861863]: 30197032..30197575 3 (5031400..5031943) 544 100000 923: [3861864..3863879]: 30302904..30304919 3 (5137272..5139287) 2016 100000 924: [3863880..3897231]: 30197576..30230927 3 (5031944..5065295) 33352 100000 925: [3897232..3897263]: 30479728..30479759 3 (5314096..5314127) 32 100000 926: [3897264..3897279]: 30304920..30304935 3 (5139288..5139303) 16 100000 927: [3897280..3897431]: 31254696..31254847 3 (6089064..6089215) 152 100000 928: [3897432..3897471]: 30305088..30305127 3 (5139456..5139495) 40 100000 929: [3897472..3897535]: 31106712..31106775 3 (5941080..5941143) 64 100000 930: [3897536..3897559]: 31105064..31105087 3 (5939432..5939455) 24 100000 931: [3897560..3897567]: 30305128..30305135 3 (5139496..5139503) 8 100000 932: [3897568..3897815]: 31261744..31261991 3 (6096112..6096359) 248 100000 933: [3897816..3897855]: 30305136..30305175 3 (5139504..5139543) 40 100000 934: [3897856..3898015]: 31261992..31262151 3 (6096360..6096519) 160 100000 935: [3898016..3898047]: 30305176..30305207 3 (5139544..5139575) 32 100000 936: [3898048..3898143]: 31262152..31262247 3 (6096520..6096615) 96 100000 937: [3898144..3898175]: 30305304..30305335 3 (5139672..5139703) 32 100000 938: [3898176..3898303]: 31262248..31262375 3 (6096616..6096743) 128 100000 939: [3898304..3902271]: 30305464..30309431 3 (5139832..5143799) 3968 100000 940: [3902272..3902335]: 31262376..31262439 3 (6096744..6096807) 64 100000 941: [3902336..3907079]: 30309496..30314239 3 (5143864..5148607) 4744 100000 942: [3907080..3907087]: 30316064..30316071 3 (5150432..5150439) 8 100000 943: [3907088..3910215]: 30372344..30375471 3 (5206712..5209839) 3128 100000 944: [3910216..3910231]: 30498880..30498895 3 (5333248..5333263) 16 100000 945: [3910232..3910271]: 30387376..30387415 3 (5221744..5221783) 40 100000 946: [3910272..3910399]: 31262440..31262567 3 (6096808..6096935) 128 100000 947: [3910400..3928599]: 30480680..30498879 3 (5315048..5333247) 18200 100000 948: [3928600..3928623]: 30592680..30592703 3 (5427048..5427071) 24 100000 949: [3928624..3928639]: 30498904..30498919 3 (5333272..5333287) 16 100000 950: [3928640..3928703]: 31262568..31262631 3 (6096936..6096999) 64 100000 951: [3928704..4022399]: 30498984..30592679 3 (5333352..5427047) 93696 100000 952: [4022400..4022423]: 31120240..31120263 3 (5954608..5954631) 24 100000 953: [4022424..4022463]: 30592704..30592743 3 (5427072..5427111) 40 100000 954: [4022464..4022591]: 31262632..31262759 3 (6097000..6097127) 128 100000 955: [4022592..4027391]: 30592872..30597671 3 (5427240..5432039) 4800 100000 956: [4027392..4027479]: 31262760..31262847 3 (6097128..6097215) 88 100000 957: [4027480..4027519]: 30597760..30597799 3 (5432128..5432167) 40 100000 958: [4027520..4027679]: 31262848..31263007 3 (6097216..6097375) 160 100000 959: [4027680..4027711]: 30597800..30597831 3 (5432168..5432199) 32 100000 960: [4027712..4027799]: 31263008..31263095 3 (6097376..6097463) 88 100000 961: [4027800..4027839]: 30597920..30597959 3 (5432288..5432327) 40 100000 962: [4027840..4027975]: 31263096..31263231 3 (6097464..6097599) 136 100000 963: [4027976..4028463]: 30598096..30598583 3 (5432464..5432951) 488 100000 964: [4028464..4031863]: 30659536..30662935 3 (5493904..5497303) 3400 100000 965: [4031864..4036655]: 30667000..30671791 3 (5501368..5506159) 4792 100000 966: [4036656..4036735]: 31263232..31263311 3 (6097600..6097679) 80 100000 967: [4036736..4037015]: 30657640..30657919 3 (5492008..5492287) 280 100000 968: [4037016..4037103]: 30657512..30657599 3 (5491880..5491967) 88 100000 969: [4037104..4037183]: 30657352..30657431 3 (5491720..5491799) 80 100000 970: [4037184..4040439]: 30696384..30699639 3 (5530752..5534007) 3256 100000 971: [4040440..4076903]: 30723352..30759815 3 (5557720..5594183) 36464 100000 972: [4076904..4077055]: 31263312..31263463 3 (6097680..6097831) 152 100000 973: [4077056..4091359]: 30777408..30791711 3 (5611776..5626079) 14304 100000 974: [4091360..4095567]: 30829600..30833807 3 (5663968..5668175) 4208 100000 975: [4095568..4095591]: 31263464..31263487 3 (6097832..6097855) 24 100000 976: [4095592..4095615]: 30833832..30833855 3 (5668200..5668223) 24 100000 977: [4095616..4095679]: 31263488..31263551 3 (6097856..6097919) 64 100000 978: [4095680..4099575]: 30833920..30837815 3 (5668288..5672183) 3896 100000 979: [4099576..4099591]: 30833872..30833887 3 (5668240..5668255) 16 100000 980: [4099592..4099607]: 30833856..30833871 3 (5668224..5668239) 16 100000 981: [4099608..4099623]: 30699664..30699679 3 (5534032..5534047) 16 100000 982: [4099624..4099647]: 30833888..30833911 3 (5668256..5668279) 24 100000 983: [4099648..4099711]: 31263552..31263615 3 (6097920..6097983) 64 100000 984: [4099712..4108743]: 31018392..31027423 3 (5852760..5861791) 9032 100000 985: [4108744..4108775]: 31366984..31367015 3 (6201352..6201383) 32 100000 986: [4108776..4108799]: 31041536..31041559 3 (5875904..5875927) 24 100000 987: [4108800..4108951]: 31367016..31367167 3 (6201384..6201535) 152 100000 988: [4108952..4108991]: 31041720..31041759 3 (5876088..5876127) 40 100000 989: [4108992..4109055]: 31367168..31367231 3 (6201536..6201599) 64 100000 990: [4109056..4111783]: 31137144..31139871 3 (5971512..5974239) 2728 100000 991: [4111784..4118015]: hole 6232 992: [4118016..4118063]: 28177880..28177927 3 (3012248..3012295) 48 100000 993: [4118064..4118095]: hole 32 994: [4118096..4118111]: 28207264..28207279 3 (3041632..3041647) 16 100000 995: [4118112..4118271]: 29296056..29296215 3 (4130424..4130583) 160 100000 996: [4118272..4118303]: 29308368..29308399 3 (4142736..4142767) 32 100000 997: [4118304..4118335]: 28207336..28207367 3 (3041704..3041735) 32 100000 998: [4118336..4118399]: 28210304..28210367 3 (3044672..3044735) 64 100000 999: [4118400..4118463]: 28750832..28750895 3 (3585200..3585263) 64 100000 1000: [4118464..4127847]: 28395272..28404655 3 (3229640..3239023) 9384 100000 1001: [4127848..4128031]: 31120056..31120239 3 (5954424..5954607) 184 100000 1002: [4128032..4128039]: 31105560..31105567 3 (5939928..5939935) 8 100000 1003: [4128040..4128063]: 28478912..28478935 3 (3313280..3313303) 24 100000 1004: [4128064..4128159]: 29430584..29430679 3 (4264952..4265047) 96 100000 1005: [4128160..4128191]: 28478936..28478967 3 (3313304..3313335) 32 100000 1006: [4128192..4128343]: 29777176..29777327 3 (4611544..4611695) 152 100000 1007: [4128344..4128383]: 28479792..28479831 3 (3314160..3314199) 40 100000 1008: [4128384..4128519]: 29846712..29846847 3 (4681080..4681215) 136 100000 1009: [4128520..4130399]: 28482128..28484007 3 (3316496..3318375) 1880 100000 1010: [4130400..4130415]: 28628688..28628703 3 (3463056..3463071) 16 100000 1011: [4130416..4130431]: 28484024..28484039 3 (3318392..3318407) 16 100000 1012: [4130432..4130495]: 28874688..28874751 3 (3709056..3709119) 64 100000 1013: [4130496..4132095]: 28484104..28485703 3 (3318472..3320071) 1600 100000 1014: [4132096..4135231]: 28498464..28501599 3 (3332832..3335967) 3136 100000 1015: [4135232..4135615]: 28502184..28502567 3 (3336552..3336935) 384 100000 1016: [4135616..4135639]: 28640648..28640671 3 (3475016..3475039) 24 100000 1017: [4135640..4135679]: 28508696..28508735 3 (3343064..3343103) 40 100000 1018: [4135680..4135807]: 29478304..29478431 3 (4312672..4312799) 128 100000 1019: [4135808..4138399]: 28619272..28621863 3 (3453640..3456231) 2592 100000 1020: [4138400..4141871]: 28781984..28785455 3 (3616352..3619823) 3472 100000 1021: [4141872..4141951]: 29308440..29308519 3 (4142808..4142887) 80 100000 1022: [4141952..4147711]: 28785536..28791295 3 (3619904..3625663) 5760 100000 1023: [4147712..4147863]: 31051512..31051663 3 (5885880..5886031) 152 100000 1024: [4147864..4147903]: 28791448..28791487 3 (3625816..3625855) 40 100000 1025: [4147904..4147967]: 28877632..28877695 3 (3712000..3712063) 64 100000 1026: [4147968..4163063]: 28791552..28806647 3 (3625920..3641015) 15096 100000 1027: [4163064..4163135]: 28915776..28915847 3 (3750144..3750215) 72 100000 1028: [4163136..4164607]: 28806720..28808191 3 (3641088..3642559) 1472 100000 1029: [4164608..4164671]: 28893088..28893151 3 (3727456..3727519) 64 100000 1030: [4164672..4169655]: 28808256..28813239 3 (3642624..3647607) 4984 100000 1031: [4169656..4169687]: 28750728..28750759 3 (3585096..3585127) 32 100000 1032: [4169688..4169711]: 28813272..28813295 3 (3647640..3647663) 24 100000 1033: [4169712..4169727]: 28813240..28813255 3 (3647608..3647623) 16 100000 1034: [4169728..4169823]: 31254848..31254943 3 (6089216..6089311) 96 100000 1035: [4169824..4169855]: 28027536..28027567 3 (2861904..2861935) 32 100000 1036: [4169856..4170015]: 25169472..25169631 3 (3840..3999) 160 100000 1037: [4170016..4170047]: 28027400..28027431 3 (2861768..2861799) 32 100000 1038: [4170048..4170207]: 25169632..25169791 3 (4000..4159) 160 100000 1039: [4170208..4170239]: 28027216..28027247 3 (2861584..2861615) 32 100000 1040: [4170240..4170303]: 25169792..25169855 3 (4160..4223) 64 100000 1041: [4170304..4174207]: 28813296..28817199 3 (3647664..3651567) 3904 100000 1042: [4174208..4181495]: 28908488..28915775 3 (3742856..3750143) 7288 100000 1043: [4181496..4181567]: 31254944..31255015 3 (6089312..6089383) 72 100000 1044: [4181568..4198079]: 28915848..28932359 3 (3750216..3766727) 16512 100000 1045: [4198080..4198143]: 31106848..31106911 3 (5941216..5941279) 64 100000 1046: [4198144..4198591]: 28932424..28932871 3 (3766792..3767239) 448 100000 1047: [4198592..4198719]: 31255016..31255143 3 (6089384..6089511) 128 100000 1048: [4198720..4213119]: 28933000..28947399 3 (3767368..3781767) 14400 100000 1049: [4213120..4213183]: 31114808..31114871 3 (5949176..5949239) 64 100000 1050: [4213184..4218119]: 28947464..28952399 3 (3781832..3786767) 4936 100000 1051: [4218120..4218143]: 28932952..28932975 3 (3767320..3767343) 24 100000 1052: [4218144..4218175]: 28947400..28947431 3 (3781768..3781799) 32 100000 1053: [4218176..4218327]: 31255144..31255295 3 (6089512..6089663) 152 100000 1054: [4218328..4218367]: 28932872..28932911 3 (3767240..3767279) 40 100000 1055: [4218368..4218455]: 31255296..31255383 3 (6089664..6089751) 88 100000 1056: [4218456..4218495]: 28932912..28932951 3 (3767280..3767319) 40 100000 1057: [4218496..4218559]: 31130552..31130615 3 (5964920..5964983) 64 100000 1058: [4218560..4224063]: 29047864..29053367 3 (3882232..3887735) 5504 100000 1059: [4224064..4224127]: 31178616..31178679 3 (6012984..6013047) 64 100000 1060: [4224128..4242239]: 29053432..29071543 3 (3887800..3905911) 18112 100000 1061: [4242240..4242303]: 31255384..31255447 3 (6089752..6089815) 64 100000 1062: [4242304..4243263]: 29071608..29072567 3 (3905976..3906935) 960 100000 1063: [4243264..4243327]: 31255448..31255511 3 (6089816..6089879) 64 100000 1064: [4243328..4267583]: 29072632..29096887 3 (3907000..3931255) 24256 100000 1065: [4267584..4267607]: 29098560..29098583 3 (3932928..3932951) 24 100000 1066: [4267608..4267647]: 29098520..29098559 3 (3932888..3932927) 40 100000 1067: [4267648..4267743]: 31255512..31255607 3 (6089880..6089975) 96 100000 1068: [4267744..4267775]: 29072568..29072599 3 (3906936..3906967) 32 100000 1069: [4267776..4267935]: 31255608..31255767 3 (6089976..6090135) 160 100000 1070: [4267936..4267967]: 29072600..29072631 3 (3906968..3906999) 32 100000 1071: [4267968..4268351]: 31255768..31256151 3 (6090136..6090519) 384 100000 1072: [4268352..4275951]: 29373144..29380743 3 (4207512..4215111) 7600 100000 1073: [4275952..4276039]: 31256152..31256239 3 (6090520..6090607) 88 100000 1074: [4276040..4285503]: 29380832..29390295 3 (4215200..4224663) 9464 100000 1075: [4285504..4285535]: 28884064..28884095 3 (3718432..3718463) 32 100000 1076: [4285536..4285567]: 29427600..29427631 3 (4261968..4261999) 32 100000 1077: [4285568..4285727]: 31256240..31256399 3 (6090608..6090767) 160 100000 1078: [4285728..4285759]: 29427632..29427663 3 (4262000..4262031) 32 100000 1079: [4285760..4285911]: 31256400..31256551 3 (6090768..6090919) 152 100000 1080: [4285912..4285951]: 29429200..29429239 3 (4263568..4263607) 40 100000 1081: [4285952..4286015]: 31256552..31256615 3 (6090920..6090983) 64 100000 1082: [4286016..4294199]: 29569624..29577807 3 (4403992..4412175) 8184 100000 1083: [4294200..4294271]: 31256616..31256687 3 (6090984..6091055) 72 100000 1084: [4294272..4312503]: 29577880..29596111 3 (4412248..4430479) 18232 100000 1085: [4312504..4312527]: 29429240..29429263 3 (4263608..4263631) 24 100000 1086: [4312528..4312575]: 29596136..29596183 3 (4430504..4430551) 48 100000 1087: [4312576..4312639]: 31256688..31256751 3 (6091056..6091119) 64 100000 1088: [4312640..4317295]: 29596248..29600903 3 (4430616..4435271) 4656 100000 1089: [4317296..4317383]: 31256752..31256839 3 (6091120..6091207) 88 100000 1090: [4317384..4337927]: 29600992..29621535 3 (4435360..4455903) 20544 100000 1091: [4337928..4337951]: 29419768..29419791 3 (4254136..4254159) 24 100000 1092: [4337952..4337983]: 29621560..29621591 3 (4455928..4455959) 32 100000 1093: [4337984..4338135]: 31256840..31256991 3 (6091208..6091359) 152 100000 1094: [4338136..4338175]: 29621744..29621783 3 (4456112..4456151) 40 100000 1095: [4338176..4338239]: 31256992..31257055 3 (6091360..6091423) 64 100000 1096: [4338240..4343279]: 29621848..29626887 3 (4456216..4461255) 5040 100000 1097: [4343280..4343375]: 31257056..31257151 3 (6091424..6091519) 96 100000 1098: [4343376..4343423]: 29626984..29627031 3 (4461352..4461399) 48 100000 1099: [4343424..4343487]: 31257152..31257215 3 (6091520..6091583) 64 100000 1100: [4343488..4361343]: 29627096..29644951 3 (4461464..4479319) 17856 100000 1101: [4361344..4361407]: 31257216..31257279 3 (6091584..6091647) 64 100000 1102: [4361408..4396911]: 29645016..29680519 3 (4479384..4514887) 35504 100000 1103: [4396912..4396999]: 31257280..31257367 3 (6091648..6091735) 88 100000 1104: [4397000..4397023]: 29680608..29680631 3 (4514976..4514999) 24 100000 1105: [4397024..4397183]: 31257368..31257527 3 (6091736..6091895) 160 100000 1106: [4397184..4400343]: 29680792..29683951 3 (4515160..4518319) 3160 100000 1107: [4400344..4408871]: 29940952..29949479 3 (4775320..4783847) 8528 100000 1108: [4408872..4409071]: 31257528..31257727 3 (6091896..6092095) 200 100000 1109: [4409072..4409087]: 29949480..29949495 3 (4783848..4783863) 16 100000 1110: [4409088..4409215]: 31257728..31257855 3 (6092096..6092223) 128 100000 1111: [4409216..4459575]: 29949624..29999983 3 (4783992..4834351) 50360 100000 1112: [4459576..4459671]: 31257856..31257951 3 (6092224..6092319) 96 100000 1113: [4459672..4459711]: 29949496..29949535 3 (4783864..4783903) 40 100000 1114: [4459712..4459775]: 31257952..31258015 3 (6092320..6092383) 64 100000 1115: [4459776..4464335]: 30192472..30197031 3 (5026840..5031399) 4560 100000 1116: [4464336..4464511]: 31258016..31258191 3 (6092384..6092559) 176 100000 1117: [4464512..4464543]: 31263616..31263647 3 (6097984..6098015) 32 100000 1118: [4464544..4464575]: 30304936..30304967 3 (5139304..5139335) 32 100000 1119: [4464576..4464735]: 31263648..31263807 3 (6098016..6098175) 160 100000 1120: [4464736..4464767]: 30304968..30304999 3 (5139336..5139367) 32 100000 1121: [4464768..4465175]: 31263808..31264215 3 (6098176..6098583) 408 100000 1122: [4465176..4465215]: 30305000..30305039 3 (5139368..5139407) 40 100000 1123: [4465216..4465279]: 31264216..31264279 3 (6098584..6098647) 64 100000 1124: [4465280..4467103]: 30314240..30316063 3 (5148608..5150431) 1824 100000 1125: [4467104..4467111]: 30317800..30317807 3 (5152168..5152175) 8 100000 1126: [4467112..4468823]: 30316072..30317783 3 (5150440..5152151) 1712 100000 1127: [4468824..4468847]: 29380808..29380831 3 (4215176..4215199) 24 100000 1128: [4468848..4468863]: 30317808..30317823 3 (5152176..5152191) 16 100000 1129: [4468864..4468927]: 31264280..31264343 3 (6098648..6098711) 64 100000 1130: [4468928..4478703]: 30317888..30327663 3 (5152256..5162031) 9776 100000 1131: [4478704..4485439]: 30375472..30382207 3 (5209840..5216575) 6736 100000 1132: [4485440..4485463]: 31264344..31264367 3 (6098712..6098735) 24 100000 1133: [4485464..4485503]: 30387416..30387455 3 (5221784..5221823) 40 100000 1134: [4485504..4485567]: 31264368..31264431 3 (6098736..6098799) 64 100000 1135: [4485568..4487871]: 30598584..30600887 3 (5432952..5435255) 2304 100000 1136: [4487872..4487935]: 31264432..31264495 3 (6098800..6098863) 64 100000 1137: [4487936..4518911]: 30600952..30631927 3 (5435320..5466295) 30976 100000 1138: [4518912..4518975]: 31264496..31264559 3 (6098864..6098927) 64 100000 1139: [4518976..4523479]: 30631992..30636495 3 (5466360..5470863) 4504 100000 1140: [4523480..4523511]: 31264560..31264591 3 (6098928..6098959) 32 100000 1141: [4523512..4523519]: 30636528..30636535 3 (5470896..5470903) 8 100000 1142: [4523520..4523647]: 31264592..31264719 3 (6098960..6099087) 128 100000 1143: [4523648..4528023]: 30636664..30641039 3 (5471032..5475407) 4376 100000 1144: [4528024..4528055]: 31264720..31264751 3 (6099088..6099119) 32 100000 1145: [4528056..4528063]: 30641072..30641079 3 (5475440..5475447) 8 100000 1146: [4528064..4528159]: 31264752..31264847 3 (6099120..6099215) 96 100000 1147: [4528160..4528191]: 30641176..30641207 3 (5475544..5475575) 32 100000 1148: [4528192..4528343]: 31264848..31264999 3 (6099216..6099367) 152 100000 1149: [4528344..4528359]: 30641208..30641223 3 (5475576..5475591) 16 100000 1150: [4528360..4528575]: 31265000..31265215 3 (6099368..6099583) 216 100000 1151: [4528576..4531167]: 30641440..30644031 3 (5475808..5478399) 2592 100000 1152: [4531168..4534023]: 30662936..30665791 3 (5497304..5500159) 2856 100000 1153: [4534024..4542527]: 30671792..30680295 3 (5506160..5514663) 8504 100000 1154: [4542528..4542535]: 30681400..30681407 3 (5515768..5515775) 8 100000 1155: [4542536..4542543]: 30699744..30699751 3 (5534112..5534119) 8 100000 1156: [4542544..4542567]: 30699640..30699663 3 (5534008..5534031) 24 100000 1157: [4542568..4542583]: 30649456..30649471 3 (5483824..5483839) 16 100000 1158: [4542584..4542591]: 30699680..30699687 3 (5534048..5534055) 8 100000 1159: [4542592..4542655]: 31265216..31265279 3 (6099584..6099647) 64 100000 1160: [4542656..4553727]: 30699752..30710823 3 (5534120..5545191) 11072 100000 1161: [4553728..4553791]: 31265280..31265343 3 (6099648..6099711) 64 100000 1162: [4553792..4555743]: 30710888..30712839 3 (5545256..5547207) 1952 100000 1163: [4555744..4559759]: 30759816..30763831 3 (5594184..5598199) 4016 100000 1164: [4559760..4559783]: 31265344..31265367 3 (6099712..6099735) 24 100000 1165: [4559784..4559791]: 30797312..30797319 3 (5631680..5631687) 8 100000 1166: [4559792..4559807]: 30797608..30797623 3 (5631976..5631991) 16 100000 1167: [4559808..4559871]: 31265368..31265431 3 (6099736..6099799) 64 100000 1168: [4559872..4565471]: 30791712..30797311 3 (5626080..5631679) 5600 100000 1169: [4565472..4565759]: 30797320..30797607 3 (5631688..5631975) 288 100000 1170: [4565760..4566279]: 30797624..30798143 3 (5631992..5632511) 520 100000 1171: [4566280..4566287]: 30809816..30809823 3 (5644184..5644191) 8 100000 1172: [4566288..4572887]: 30798152..30804751 3 (5632520..5639119) 6600 100000 1173: [4572888..4573767]: 30805928..30806807 3 (5640296..5641175) 880 100000 1174: [4573768..4735783]: 30837816..30999831 3 (5672184..5834199) 162016 100000 1175: [4735784..4749895]: 31027424..31041535 3 (5861792..5875903) 14112 100000 1176: [4749896..4749919]: 31265432..31265455 3 (6099800..6099823) 24 100000 1177: [4749920..4749951]: 31041560..31041591 3 (5875928..5875959) 32 100000 1178: [4749952..4750015]: 31265456..31265519 3 (6099824..6099887) 64 100000 1179: [4750016..4750039]: 31315864..31315887 3 (6150232..6150255) 24 100000 1180: [4750040..4750079]: 31041680..31041719 3 (5876048..5876087) 40 100000 1181: [4750080..4750143]: 31265520..31265583 3 (6099888..6099951) 64 100000 1182: [4750144..4754943]: 31041784..31046583 3 (5876152..5880951) 4800 100000 1183: [4754944..4755007]: 31367232..31367295 3 (6201600..6201663) 64 100000 1184: [4755008..4759863]: 31046648..31051503 3 (5881016..5885871) 4856 100000 1185: [4759864..4759903]: 31367296..31367335 3 (6201664..6201703) 40 100000 1186: [4759904..4759911]: 31051504..31051511 3 (5885872..5885879) 8 100000 1187: [4759912..4760063]: 31367336..31367487 3 (6201704..6201855) 152 100000 1188: [4760064..4761791]: 31051664..31053391 3 (5886032..5887759) 1728 100000 1189: [4761792..4761855]: 31367488..31367551 3 (6201856..6201919) 64 100000 1190: [4761856..4765175]: 31053456..31056775 3 (5887824..5891143) 3320 100000 1191: [4765176..4765247]: 31367552..31367623 3 (6201920..6201991) 72 100000 1192: [4765248..4770767]: 31056848..31062367 3 (5891216..5896735) 5520 100000 1193: [4770768..4770775]: 31053440..31053447 3 (5887808..5887815) 8 100000 1194: [4770776..4770815]: 31056776..31056815 3 (5891144..5891183) 40 100000 1195: [4770816..4771727]: 31367624..31368535 3 (6201992..6202903) 912 100000 1196: [4771728..4771775]: 31053392..31053439 3 (5887760..5887807) 48 100000 1197: [4771776..4771903]: 31368536..31368663 3 (6202904..6203031) 128 100000 1198: [4771904..4805183]: 31139872..31173151 3 (5974240..6007519) 33280 100000 1199: [4805184..4805191]: 31105568..31105575 3 (5939936..5939943) 8 100000 1200: [4805192..4805511]: 31178240..31178559 3 (6012608..6012927) 320 100000 1201: [4805512..4805519]: 31097560..31097567 3 (5931928..5931935) 8 100000 1202: [4805520..4805567]: 31178568..31178615 3 (6012936..6012983) 48 100000 1203: [4805568..4805631]: 31368664..31368727 3 (6203032..6203095) 64 100000 1204: [4805632..4876071]: 31178680..31249119 3 (6013048..6083487) 70440 100000 1205: [4876072..4877583]: 31368728..31370239 3 (6203096..6204607) 1512 100000 1206: [4877584..4877623]: 31266120..31266159 3 (6100488..6100527) 40 100000 1207: [4877624..4877903]: 31370240..31370519 3 (6204608..6204887) 280 100000 1208: [4877904..4877911]: 31266192..31266199 3 (6100560..6100567) 8 100000 1209: [4877912..4878407]: 31370520..31371015 3 (6204888..6205383) 496 100000 1210: [4878408..4878423]: 31266200..31266215 3 (6100568..6100583) 16 100000 1211: [4878424..4878567]: 31371016..31371159 3 (6205384..6205527) 144 100000 1212: [4878568..4878583]: 31266256..31266271 3 (6100624..6100639) 16 100000 1213: [4878584..4879983]: 31371160..31372559 3 (6205528..6206927) 1400 100000 1214: [4879984..4879999]: 31266272..31266287 3 (6100640..6100655) 16 100000 1215: [4880000..4880071]: 31372560..31372631 3 (6206928..6206999) 72 100000 1216: [4880072..4880127]: 31266496..31266551 3 (6100864..6100919) 56 100000 1217: [4880128..4880271]: 31372632..31372775 3 (6207000..6207143) 144 100000 1218: [4880272..4880303]: 31265992..31266023 3 (6100360..6100391) 32 100000 1219: [4880304..4880415]: 31372776..31372887 3 (6207144..6207255) 112 100000 1220: [4880416..4880431]: 31266072..31266087 3 (6100440..6100455) 16 100000 1221: [4880432..4880607]: 31372888..31373063 3 (6207256..6207431) 176 100000 1222: [4880608..4880639]: 31315528..31315559 3 (6149896..6149927) 32 100000 1223: [4880640..4880719]: 31373064..31373143 3 (6207432..6207511) 80 100000 1224: [4880720..4880743]: 31315640..31315663 3 (6150008..6150031) 24 100000 1225: [4880744..4880839]: 31373144..31373239 3 (6207512..6207607) 96 100000 1226: [4880840..4880879]: 31315760..31315799 3 (6150128..6150167) 40 100000 1227: [4880880..4880967]: 31373240..31373327 3 (6207608..6207695) 88 100000 1228: [4880968..4881023]: 31315888..31315943 3 (6150256..6150311) 56 100000 1229: [4881024..4881095]: 31373328..31373399 3 (6207696..6207767) 72 100000 1230: [4881096..4881143]: 31316016..31316063 3 (6150384..6150431) 48 100000 1231: [4881144..4881223]: 31373400..31373479 3 (6207768..6207847) 80 100000 1232: [4881224..4881271]: 31316144..31316191 3 (6150512..6150559) 48 100000 1233: [4881272..4881351]: 31373480..31373559 3 (6207848..6207927) 80 100000 1234: [4881352..4881399]: 31316272..31316319 3 (6150640..6150687) 48 100000 1235: [4881400..4881479]: 31373560..31373639 3 (6207928..6208007) 80 100000 1236: [4881480..4881511]: 31316400..31316431 3 (6150768..6150799) 32 100000 1237: [4881512..4881631]: 31373640..31373759 3 (6208008..6208127) 120 100000 1238: [4881632..4881639]: 31316432..31316439 3 (6150800..6150807) 8 100000 1239: [4881640..4881751]: 31373760..31373871 3 (6208128..6208239) 112 100000 1240: [4881752..4881775]: 31316440..31316463 3 (6150808..6150831) 24 100000 1241: [4881776..4882071]: 31373872..31374167 3 (6208240..6208535) 296 100000 1242: [4882072..4882079]: 31316464..31316471 3 (6150832..6150839) 8 100000 1243: [4882080..4882175]: 31374168..31374263 3 (6208536..6208631) 96 100000 1244: [4882176..4925015]: 31316568..31359407 3 (6150936..6193775) 42840 100000 1245: [4925016..5140223]: hole 215208 1246: [5140224..5140479]: 28177928..28178183 3 (3012296..3012551) 256 100000 1247: [5140480..5142487]: hole 2008 1248: [5142488..5142527]: 28028392..28028431 3 (2862760..2862799) 40 100000 1249: [5142528..20971519]: hole 15828992 FLAG Values: 0100000 Shared extent 0010000 Unwritten preallocated extent 0001000 Doesn't begin on stripe unit 0000100 Doesn't end on stripe unit 0000010 Doesn't begin on stripe width 0000001 Doesn't end on stripe width ```
dustymabe commented 6 months ago

As a lazy way to test this in other environments I opened a test PR against OSBuild that would then run in their CI

Sure enough CI failed on two environments:

sandeen commented 6 months ago

Thanks. So the disk.img does indeed have many reflinked extents, for starters (extents with flags 100000) When I compare the original disk image to the converted-back-to-raw qcow2 image, the qcow2 image seems to have many ranges of zeros rather than correct data:

# qemu-img convert osbuild/store/objects/e0ab569c34dd263e53d28e56e0b2f9d6f27c4cb7c4fe5d7177f57f8b719fd229/data/tree/qemu.qcow2 ../disk.img
# cmp -l ./osbuild/store/objects/0a4be93ab2bb5c17851a57574f3af0cffe135c58ccfc840e4f0ea37dda36f77c/data/tree/disk.img ../disk.img | awk '{print $3}' | sort | uniq 
0

Not sure what to make of this yet... my first thought is that it's possibly related to preallocated/unwritten extents in the original disk.img but that's just a guess.

XanClic commented 6 months ago

Hi,

Can you upload the pre-convert raw image somewhere, or is there an easy way for me to get it from some pipeline (like https://github.com/osbuild/osbuild/pull/1594)? (I’ve tried downloading the cache2.qcow2.zst that you linked, but the download is very slow for me (~30 kB/s), so would take more than a day to download.)

dustymabe commented 6 months ago

Talked with @XanClic and she was able to get the file downloaded

dustymabe commented 6 months ago

Some more context here about what OSBuild is doing that might be unique here.

OSbuild builds things in stages. We have stages that do one or two things, but the entire output from that stage gets saved and copied into the next stages for the next work item.

A simplified version of what is going on here is:

So what's happening here is there is a disk.img from the raw-image stage that gets copied into the raw-qemu-image stage and then modified slightly which then gets qemu-img converted in the qemu stage.

So there is an original disk.img from the raw-image pipeline that propably shares reflinks with the disk.img from the raw-qemu-image pipeline that then shares reflinks with the qemu.qcow2 from the qemu pipeline.

This series of steps may be what is contributing to the problem here.

Also note this isn't 100% reproducible. Sometimes I can reproduce it every time and other times only 10% or so.

dustymabe commented 6 months ago

More data:

I patched OSBuild to not cp using --refink=auto when it copies the first disk.img to the second disk.img in the workflow described above:

diff --git a/stages/org.osbuild.copy b/stages/org.osbuild.copy
index 5bc9067f..a7570ee7 100755
--- a/stages/org.osbuild.copy
+++ b/stages/org.osbuild.copy
@@ -157,7 +157,7 @@ def main(args, options):

         print(f"copying '{src}' -> '{dst}'")

-        cmd = ["cp", "-a", "--reflink=auto"]
+        cmd = ["cp", "-a", "--reflink=never"]
         if remove_destination:
             cmd.append("--remove-destination")
         subprocess.run(cmd + [src, dst], check=True)

I ran the loop 30 times and didn't see any failures.

XanClic commented 6 months ago

So far, I haven’t found any reproducer on my system (and I’d like to have some simple local reproducer :)).

What I’ve tried that didn’t work (50 iterations for most items, 1000 for the last):

However, I haven’t scrubbed the image between runs (i.e. basically rm cache2.qcow2 && zstd -d cache2.qcow2.zst), that’s one thing I still ought to try (but will of course take much longer).

XanClic commented 6 months ago

However, I haven’t scrubbed the image between runs (i.e. basically rm cache2.qcow2 && zstd -d cache2.qcow2.zst), that’s one thing I still ought to try (but will of course take much longer).

I’ve done that now for the VM case (doesn’t take that long with hot-plugging and -unplugging), and still couldn’t reproduce it (50 runs).

dustymabe commented 6 months ago

OK I might have a more minimal reproducer that doesn't include OSBuild.

I took a look at exactly what the OSbuild stages were doing when copying things around and wrote a script to try to simulate it more closely. The summary is that we are now:

Using the filesystem I uploaded before, mount it and cd into the root of that filesystem and then run this script:

[builder@2f7c40c67cf8 srv]$ cat repro.sh 
#/usr/bin/bash
set -eux -o pipefail

[ ! -e /dev/loop-control ] && mknod /dev/loop-control c 10 237

mkdir -p mnt

count=1
while true; do
    echo "COUNT is $count"; count=$((count+1))
    cp --remove-destination --reflink=always \
        osbuild/store/objects/06ead42642e12a8f1ab109bf29935661ed4060a0bc59accbae7549a2f0cd900f/data/tree/disk.img ./
    truncate --size=$((10 * 1024 * 1024 * 1024)) disk.img
    loopdev=$(losetup -f --show --partscan disk.img)
    sleep 1
    mount ${loopdev}p3 ./mnt/
    sed -i -E 's/^options (.*)$/options \1 ignition.platform.id=qemu console=ttyS0 console=ttyS0,115200n8 ignition.platform.id=qemu/' \
         ./mnt/loader/entries/*.conf
    umount ./mnt
    qemu-img convert -O qcow2 -o compat=1.1 disk.img qemu.qcow2
    qemu-img compare -f raw -F qcow2 disk.img qemu.qcow2
done
bash-5.2# bash /srv/repro.sh 
+ '[' '!' -e /dev/loop-control ']'
+ mkdir -p mnt
+ count=1
+ true
+ echo 'COUNT is 1'
COUNT is 1
+ count=2
+ cp --remove-destination --reflink=always osbuild/store/objects/06ead42642e12a8f1ab109bf29935661ed4060a0bc59accbae7549a2f0cd900f/data/tree/disk.img ./
+ truncate --size=10737418240 disk.img
++ losetup -f --show --partscan disk.img
[  234.459387] loop0: detected capacity change from 0 to 20971520
[  234.461171] GPT:Primary header thinks Alt. header is not at the end of the disk.
[  234.462675] GPT:5142527 != 20971519
[  234.463235] GPT:Alternate GPT header not at the end of the disk.
[  234.463979] GPT:5142527 != 20971519
[  234.464393] GPT: Use GNU Parted to correct GPT errors.
[  234.465016]  loop0: p1 p2 p3 p4
+ loopdev=/dev/loop0
+ sleep 1
+ mount /dev/loop0p3 ./mnt/
[  235.488936] EXT4-fs (loop0p3): mounted filesystem 96d15588-3596-4b3c-adca-a2ff7279ea63 r/w with ordered data mode. Quota mode: none.
+ sed -i -E 's/^options (.*)$/options \1 ignition.platform.id=qemu console=ttyS0 console=ttyS0,115200n8 ignition.platform.id=qemu/' ./mnt/loader/entries/ostree-1-fedora-coreos.conf
+ umount ./mnt
[  235.515810] EXT4-fs (loop0p3): unmounting filesystem 96d15588-3596-4b3c-adca-a2ff7279ea63.
+ qemu-img convert -O qcow2 -o compat=1.1 disk.img qemu.qcow2
+ qemu-img compare -f raw -F qcow2 disk.img qemu.qcow2
Content mismatch at offset 403718656!
sandeen commented 6 months ago

Nice job on the reproducer! If I omit the step where it modifies the copied disk.img via sed, it passes. If I md5sum the copied disk.img before mounting it, it passes.

Not sure what to make of this yet.

XanClic commented 6 months ago

Thanks, that works indeed!

With debugging information put into qemu-img, I can see that it believes the offset is zero, whereas it is not zero when you actually inspect it. qemu-img gets this information via SEEK_HOLE/SEEK_DATA.

There seems to be some inconsistency in this hole information. Fully reading disk.img seems to update this information, so putting a cat disk.img > /dev/null before the qemu-img convert has the test pass.

Consequently, neither qcow2 nor qemu-img nor what target filesystem you use seem to be of importance, but beware that if you use qemu-img compare, it will again check hole information, so the following passes for me:

qemu-img convert -f raw -O raw disk.img /tmp/qemu.img
qemu-img compare -f raw -F raw disk.img /tmp/qemu.img

(Side note, I actually have no idea why it does report a mismatch when using qcow2. The hole information in disk.img should still be wrong, making qemu-img compare skip the same areas as convert did. Perhaps the qcow2 clustering changes something about the compare granularity.)

So this says both images are identical, but if you use diff -q instead of qemu-img compare (or compare a hexdump), you will indeed see that they differ, because diff actually reads the whole image. Same if you put a cat disk.img > /dev/null in between qemu-img convert and qemu-img compare (because this seems to update the hole information).

So replacing qemu-img convert+qemu-img compare by the following reproduces the problem completely without qemu or qcow2:

cp --sparse=always disk.img /tmp/qemu.img
diff -q disk.img /tmp/qemu.img

(Again, note that using qemu-img compare instead of diff would report for both images to be identical because it checks hole information and skips everything that’s reportedly zero. But if you run it after something has read the whole image (e.g. after diff, which does that), it reports a difference.)

dustymabe commented 6 months ago

A lot to unpack there. Thanks @XanClic! I'm not sure quite what to make of it all, but there is one thing you mention in there that seems interesting:

Consequently, neither qcow2 nor qemu-img nor what target filesystem you use seem to be of importance

I'd argue that the target filesystem is of importance here. We switched from xfs to ext4 to get unblocked on this and that is working fine in our pipeline. I've also been running builds on Fedora Cloud VMs (BTRFS, which also supports reflinks) and haven't seen any issues there.

I'm not saying it is 100% filesystem related, but there is at least some data to point towards it being related.

XanClic commented 6 months ago

Oh, absolutely, the (source) filesystem does matter. I just meant the target filesystem of the copy/convert operation (for qemu.qcow2), which I changed to tmpfs, and still saw the problem.

I do believe that the (seemingly) incorrect hole information for disk.img is XFS-related.

sandeen commented 6 months ago

Patch from dchinner:

https://lore.kernel.org/linux-xfs/20240220224928.3356-1-david@fromorbit.com/T/#u

Still trying to craft a simple reproducer though.

dustymabe commented 5 months ago

Landed upstream in https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=4b2f459d86252619448455013f581836c8b1b7da

Hasn't been backported to any stable branches yet.

dustymabe commented 5 months ago

This was backported to Fedora 6.7 series in https://gitlab.com/cki-project/kernel-ark/-/commit/c0412c5250f7bdb2ea61b61f27eada0d8a135692 and should land in F39 with kernel-6.7.11-200 (https://bodhi.fedoraproject.org/updates/FEDORA-2024-2fcce4ffb7)

dustymabe commented 5 months ago

With the new kernel landed and our cache qcow switched back to XFS we can now close this.