struct error: unpack requires a buffer of 4 bytes

[bycloudai]

Hi, thank you so much for sharing this amazing work to everyone! I am currently testing if MultiNeRF is possible to run on Windows machine, and right now with this command:

python -m train --gin_configs=configs\360.gin --gin_bindings="Config.data_dir = '%DATA_DIR%'" --gin_bindings="Config.checkpoint_dir = '%DATA_DIR%\checkpoints'" --logtostderr

I would run into this error:

I0822 23:45:01.236814 21060 xla_bridge.py:160] Remote TPU is not linked into jax; skipping remote TPU.
I0822 23:45:01.236814 21060 xla_bridge.py:333] Unable to initialize backend 'tpu_driver': Could not initialize backend 'tpu_driver'
I0822 23:45:01.369816 21060 xla_bridge.py:333] Unable to initialize backend 'rocm': NOT_FOUND: Could not find registered platform with name: "rocm". Available platform names are: CUDA Interpreter Host
I0822 23:45:01.374814 21060 xla_bridge.py:333] Unable to initialize backend 'tpu': module 'jaxlib.xla_extension' has no attribute 'get_tpu_client'
C:\Users\user\anaconda3\envs\multinerf\lib\site-packages\jax\_src\lib\xla_bridge.py:506: UserWarning: jax.host_id has been renamed to jax.process_index. This alias will eventually be removed; please update your code.
  warnings.warn(
Traceback (most recent call last):
  File "C:\Users\user\anaconda3\envs\multinerf\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Users\user\anaconda3\envs\multinerf\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "F:\MultiNeRF\multinerf\train.py", line 288, in <module>
    app.run(main)
  File "C:\Users\user\anaconda3\envs\multinerf\lib\site-packages\absl\app.py", line 308, in run
    _run_main(main, args)
  File "C:\Users\user\anaconda3\envs\multinerf\lib\site-packages\absl\app.py", line 254, in _run_main
    sys.exit(main(argv))
  File "F:\MultiNeRF\multinerf\train.py", line 55, in main
    dataset = datasets.load_dataset('train', config.data_dir, config)
  File "F:\MultiNeRF\multinerf\internal\datasets.py", line 52, in load_dataset
    return dataset_dict[config.dataset_loader](split, train_dir, config)
  File "F:\MultiNeRF\multinerf\internal\datasets.py", line 295, in __init__
    self._load_renderings(config)
  File "F:\MultiNeRF\multinerf\internal\datasets.py", line 584, in _load_renderings
    pose_data = NeRFSceneManager(colmap_dir).process()
  File "F:\MultiNeRF\multinerf\internal\datasets.py", line 77, in process
    self.load_cameras()
  File "F:\MultiNeRF\multinerf\internal/pycolmap/pycolmap\scene_manager.py", line 90, in load_cameras
    self._load_cameras_bin(input_file)
  File "F:\MultiNeRF\multinerf\internal/pycolmap/pycolmap\scene_manager.py", line 102, in _load_cameras_bin
    num_cameras = struct.unpack('L', f.read(8))[0]
struct.error: unpack requires a buffer of 4 bytes

Please let me know if you know any solutions for this error! Any help would be appreciated.

Currently my Jax is 0.13.4 and the Jaxlib is 0.13.4 too Running on CUDA11.1 with python 3.9

[jonbarron]

I have no idea (I don't have a windows computer to compare with) but my instinct is to try adding 'rb' to the line where you open the problematic file, if that isn't already specified.

[bycloudai]

@jonbarron unfortunately it is already satisfied...

[jonbarron]

Oh too bad. Sounds like it might be related to a difference between Windows and Linux filesystems? I don't really know.

[bycloudai]

Ya it seems to work just by adjusting the f.read(8) to f.read(4), I should've figured that out myself hahaha That worked as a solution but was caught with another one

  File "F:\MultiNeRF\multinerf\internal\datasets.py", line 584, in _load_renderings
    pose_data = NeRFSceneManager(colmap_dir).process()
  File "F:\MultiNeRF\multinerf\internal\datasets.py", line 78, in process
    self.load_images()
  File "F:\MultiNeRF\multinerf\internal/pycolmap/pycolmap\scene_manager.py", line 131, in load_images
    self._load_images_bin(input_file)
  File "F:\MultiNeRF\multinerf\internal/pycolmap/pycolmap\scene_manager.py", line 162, in _load_images_bin
    points_array.fromfile(f, 3 * num_points2D)
OverflowError: Python int too large to convert to C ssize_t

any help would be appreciated!

[Bra1nsen]

updates?

[briantcantwell]

I've also hit this issue. Any update?

[softyoda]

I have the same issue

  File "C:\Users\intel\Documents\05_3DGS\gsplat\examples\datasets\colmap.py", line 50, in __init__
    manager.load_cameras()
  File "C:\Users\intel\AppData\Roaming\Python\Python311\site-packages\pycolmap\scene_manager.py", line 90, in load_cameras
    self._load_cameras_bin(input_file)
  File "C:\Users\intel\AppData\Roaming\Python\Python311\site-packages\pycolmap\scene_manager.py", line 102, in _load_cameras_bin
    num_cameras = struct.unpack('L', f.read(8))[0]
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack requires a buffer of 4 bytes

and when I try to set the f.read(8) to f.read(4) i have a new error

  File "C:\Users\intel\Documents\05_3DGS\gsplat\examples\datasets\colmap.py", line 50, in __init__
    manager.load_cameras()
  File "C:\Users\intel\AppData\Roaming\Python\Python311\site-packages\pycolmap\scene_manager.py", line 90, in load_cameras
    self._load_cameras_bin(input_file)
  File "C:\Users\intel\AppData\Roaming\Python\Python311\site-packages\pycolmap\scene_manager.py", line 105, in _load_cameras_bin
    camera_id, camera_type, w, h = struct.unpack('IiLL', f.read(24))
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
struct.error: unpack requires a buffer of 16 bytes

[chgara]

datasets

Hi I have the same problem, have you been able to solve it?

[whm5815664]

datasets

Hi I have the same problem, have you been able to solve it?

I didn't encounter this issue when deploying the project on Linux, but I did encounter it on Windows. Try modifying it to ### struct.unpack('IiLL', f.read(16))