Structured 3D dataset - corruptions in zip files

[ok1zjf]

Many thanks for making this dataset available! I have downloaded the Structured3D_panorama_00.zip ... Structured3D_panorama_17.zip from the Azure cloud and found that all except 14,15,16,17 have some files corrupted inside the zip archives. I have downloaded the archives on multiple university and my private internet connections and used different applications to unzip the files (unzip, 7z in Ubuntu 22.04 and with Windows 10 archive manager). In all cases, the same errors were encountered. For example, unzipping the Structured3D_panorama_13.zip results in the following errors:

unzip -q Structured3D_panorama_13.zip -d /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/ error: invalid compressed data to inflate /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02601/2D_rendering/843/panorama/simple/normal.png /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_coldlight.png bad CRC 406b2c0b (should be 53007390) file #1444: bad zipfile offset (local header sig): 536390646 /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02609/2D_rendering/1235/panorama/simple/rgb_warmlight.png bad CRC fbc88119 (should be 35d6a845) file #1486: bad zipfile offset (local header sig): 553273189 file #1487: bad zipfile offset (local header sig): 553279282 file #1488: bad zipfile offset (local header sig): 553279355 file #1489: bad zipfile offset (local header sig): 553279437 file #1490: bad zipfile offset (local header sig): 553279568 file #1491: bad zipfile offset (local header sig): 553279656 file #1492: bad zipfile offset (local header sig): 553519416 /media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_coldlight.png bad CRC c4090c4d (should be 02b78ee6) file #1579: bad zipfile offset (local header sig): 586400993

and with 7z

7z x -o/media/jiri/Pluto/datasets/Structured3D/Structured3D_panorama/13/ Structured3D_panorama_13.zip

7-Zip [64] 16.02 : Copyright (c) 1999-2016 Igor Pavlov : 2016-05-21 p7zip Version 16.02 (locale=en_GB.UTF-8,Utf16=on,HugeFiles=on,64 bits,8 CPUs Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz (306F2),ASM,AES-NI)

Scanning the drive for archives: 1 file, 12381936537 bytes (12 GiB)

Extracting archive: Structured3D_panorama_13.zip

--
Path = Structured3D_panorama_13.zip Type = zip Physical Size = 12381936537 64-bit = +

ERROR: Data Error : Structured3D/scene_02601/2D_rendering/843/panorama/simple/normal.png ERROR: CRC Failed : Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_coldlight.png ERROR: Headers Error : Structured3D/scene_02608/2D_rendering/345/panorama/full/rgb_rawlight.png ERROR: CRC Failed : Structured3D/scene_02609/2D_rendering/1235/panorama/simple/rgb_warmlight.png ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1235/panorama/simple/semantic.png ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419
ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/camera_xyz.txt ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty/albedo.png ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/1419/panorama/empty/depth.png ERROR: CRC Failed : Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_coldlight.png ERROR: Headers Error : Structured3D/scene_02609/2D_rendering/4681/panorama/empty/rgb_rawlight.png

Sub items Errors: 13 Archives with Errors: 1 Sub items Errors: 13

Could you please verify that these files are fine in the Azure cloud? If yes, then perhaps the files are corrupted only on the Azure mirror in the UK, which would be a very serious fault in Azure data management.

Kind Regards Jiri

[bertjiazheng]

I had a quick check and found the files on Azure Cloud had already been corrupted. But it seems that only a few images have been affected. For now, you may ignore the corrupted data.

[ok1zjf]

Thank you for confirming the issue. For what it is worth, here is a list of the corrupted scenes in the Structured3D_panorama. In each scene, one or multiple files are corrupted.

00 scene_00010 scene_00013 scene_00014

01 scene_00206 scene_00211 scene_00212 scene_00213

02 scene_00403 scene_00408 scene_00409 scene_00410 scene_00411 scene_00412

03 scene_00609 scene_00612 scene_00613

04 scene_00806 scene_00809 scene_00810

05 scene_01009 scene_01010

06 scene_01209 scene_01210

07 scene_01410 scene_01411 scene_01412

08 scene_01605 scene_01607 scene_01608

09 scene_01813 scene_01814 scene_01815

11 scene_02211 scene_02212 scene_02213

12 scene_02409 scene_02411

13 scene_02601 scene_02608 scene_02609

[fengenxiang1123]

Yes, I also encountered this problem.