search

lpiccinelli-eth / UniDepth

Universal Monocular Metric Depth Estimation
Other
473 stars 39 forks source link

Encountered `RuntimeError: expected scalar type Double but found Float` using manual intrinsics input #9

Closed AAArthurLiu closed 3 months ago

AAArthurLiu commented 3 months ago

I am running demo.py. The original script runs fine. Substituting the intrinsics from loading from .npy file to an manual input results in runtime error. Could not figure out why. Any help is appreciated!

Specifically, the intrinsics_torch = torch.from_numpy(np.load("assets/demo/intrinsics.npy")) is replaced by

    intrinsics_torch = torch.from_numpy(
        np.array(
            [[860.729675, 0.0, 645.56073], [0.0, 859.77063, 353.41275], [0.0, 0.0, 1.0]]
        )
    )

Logs and Errors

Torch version: 2.2.0+cu118
Downloading: "https://github.com/lpiccinelli-eth/unidepth/zipball/main" to /home/xxx/.cache/torch/hub/main.zip
Instantiate: dinov2_vitl14
UniDepthV1_ViTL14 is loaded with:
        missing keys: ['pixel_encoder.register_tokens']
        additional keys: []
Traceback (most recent call last):
  File "/home/xxx/Documents/UniDepth/./scripts/demo.py", line 50, in <module>
    demo(model)
  File "/home/xxx/Documents/UniDepth/./scripts/demo.py", line 18, in demo
    predictions = model.infer(rgb_torch, intrinsics_torch)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.cache/torch/hub/lpiccinelli-eth_unidepth_main/unidepth/models/unidepthv1/unidepthv1.py", line 228, in infer
    pred_intrinsics, predictions, _ = self.pixel_decoder(inputs, {})
                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.cache/torch/hub/lpiccinelli-eth_unidepth_main/unidepth/models/unidepthv1/decoder.py", line 526, in forward
    out8, out4, out2, depth_features = self.depth_layer(
                                       ^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.cache/torch/hub/lpiccinelli-eth_unidepth_main/unidepth/models/unidepthv1/decoder.py", line 227, in forward
    rays_embedding_16 = self.project_rays16(rsh_cart_8(rays_embedding_16))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/.cache/torch/hub/lpiccinelli-eth_unidepth_main/unidepth/layers/mlp.py", line 29, in forward
    x = self.norm(x)
        ^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/modules/normalization.py", line 201, in forward
    return F.layer_norm(
           ^^^^^^^^^^^^^
  File "/home/xxx/miniconda3/envs/unidepth/lib/python3.11/site-packages/torch/nn/functional.py", line 2546, in layer_norm
    return torch.layer_norm(input, normalized_shape, weight, bias, eps, torch.backends.cudnn.enabled)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: expected scalar type Double but found Float

Environment

Used a conda environment following steps in README

AAArthurLiu commented 3 months ago

Figured it out. Contrary to the error message, the intrinsics should be type float32 instead of float64. Simply add np.array(..)..astype(np.float32) fixed it.