Issue with the Structure3D preprocessing.

[madaan-nikhil]

Thanks for making your repo public. We are trying to use the repository for processing Structure3D dataset to get the labelled dataset and 3d point clouds. However, the current preprocessing script throws an exception with the latest version of the dataset. Can you please look into it?

zipfile.BadZipFile: Bad magic number for file header

[yuxiaoguo]

No problem. I will look into it. However, as we mentioned, we don't have the latest version of Structure3D. Thus, we need to download it first. Additionally, you may post the whole failure log here in case we can't reproduce the failure.

[yuxiaoguo]

Does the failure happen in some or all zip files? If just a few zip files are cropped, please give me the name of files. I can test or figure out workaround on these files. And I notice that the latest download README.md says that: "Structured3D_perspective_full_09.zip (corrupted: refer to https://github.com/bertjiazheng/Structured3D/issues/30)" If there any connections with your issue?

[madaan-nikhil]

Thanks for your response. Here is the complete log.

Traceback (most recent call last):
  File "/home/nikhilmadaan/anaconda3/envs/uni3drc/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/nikhilmadaan/anaconda3/envs/uni3drc/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/nfs/turbo/coe-chaijy/datasets/structured3d/Uni3DScenes/datasets/structured3d/__init__.py", line 338, in _mp_format_dataset
    = self._view2points(zip_reader, room_path)
  File "/nfs/turbo/coe-chaijy/datasets/structured3d/Uni3DScenes/datasets/structured3d/__init__.py", line 240, in _view2points
    cam_paras, attr_images = __class__.read_camera_and_image(\
  File "/nfs/turbo/coe-chaijy/datasets/structured3d/Uni3DScenes/datasets/structured3d/__init__.py", line 111, in read_camera_and_image
    color_image = cv2.imdecode(np.frombuffer(io.BytesIO(zip_reader.read(\
  File "/nfs/turbo/coe-chaijy/datasets/structured3d/Uni3DScenes/graphics_utils/stream_io.py", line 299, in read
    return self._zips_meta[z_idx].read(file_name)
  File "/home/nikhilmadaan/anaconda3/envs/uni3drc/lib/python3.8/zipfile.py", line 1475, in read
    with self.open(name, "r", pwd) as fp:
  File "/home/nikhilmadaan/anaconda3/envs/uni3drc/lib/python3.8/zipfile.py", line 1535, in open
    raise BadZipFile("Bad magic number for file header")
zipfile.BadZipFile: Bad magic number for file header

It happens for all the zip files. Please let me know if you want me to share something else to replicate the issue.

[madaan-nikhil]

It doesnt seem to be related to the corrupted file, since we get this error for other files too.

[yuxiaoguo]

No problem. I will try it. If I can't reproduce the error, I may give an alternative option that make you unzip the files by yourself, and the code process the data with unzipped files rather than raw zip files.

[yuxiaoguo]

Could you give more details about your OS, python and zip file package information? And the downloaded Structured3D files could be decompressed with unzip or other similar commands? I have tried but won't get the error. However, My OS is Win11. As I mentioned, an alternative option to read the unzipped files is a good solution. But I can't guarantee when I can finish it, maybe this weekend or end of July, depended on the workloads in my hand. Thus, if you ensure that the failure is from the python zip file package rather than the file itself (by unzip the file with other tools), I will promote the priority, for other people may have the same issue, due to difference in OS or python environment.

[madaan-nikhil]

Thanks for looking into it. I tried modifying the loader as you suggested and it worked. Thanks.

[yuxiaoguo]

Great to hear that. It will be very appreciated if you consider merging your modification into the main master via a pull request.

[Lucyliuwen]

Hello, thank you for your public great work! I have met the same problem, and I tried to unzip the .zip following the above option but I encountered another problem:

unzip -d dataset/ Structured3D_perspective_full_00.zip Archive: Structured3D_perspective_full_00.zip warning [Structured3D_perspective_full_00.zip]: 8326737875 extra bytes at beginning or within zipfile (attempting to process anyway) error [Structured3D_perspective_full_00.zip]: start of central directory not found; zipfile corrupt. (please check that you have transferred or created the zipfile in the appropriate BINARY mode and that you have compiled UnZip properly)

which seems to indicate that this is not a zip file format, so I can not unzip it. So how can i solve this problem? could you share your solution for this, Thank you !

Thanks for looking into it. I tried modifying the loader as you suggested and it worked. Thanks.

[yuxiaoguo]

In order to load an unzipped folder rather than zipped ones, you need to change the code snippets in Structured3DDataGen. The class uses an abstract zip-related class to regard a folder multiple zip files as a general folder. You need to reimplement the functions in GroupZipIO to read the file list, images, binaries directly from a folder. If you can wait, I will release an official modification in the weekend.