search

TRI-ML / packnet-sfm

TRI-ML Monocular Depth Estimation Repository
https://tri-ml.github.io/packnet-sfm/
MIT License
1.23k stars 242 forks source link

How to get proper depth from regular images #156

Closed mslavescu closed 3 years ago

mslavescu commented 3 years ago

Hi,

I setup the docker same as here: https://github.com/TRI-ML/packnet-sfm/issues/154#issue-928707614

I wownloaded all these 4 checkpoints in /data/checkpoints

And I tested this image with PackNetSAN01_HR_sup_K.ckpt and ResNet18_MR_selfsup_D.ckpt image

And I got these results:

When I ran the same image with MiDaS v3 I get this result, which looks much better than those above: image

How can I get similar depth or better than MiDaS v3 using packnet-sfm on non KITTI images?

It works fine on KITTI tiny images as I shown here: https://github.com/TRI-ML/packnet-sfm/issues/154#issuecomment-867254872

If I run with PackNetSAN01_HR_sup_K.ckpt I get this error:

root@ms1-PCx0Dx:/workspace/packnet-sfm# python3 scripts/infer.py --checkpoint /data/checkpoints/PackNetSAN01_HR_sup_K.ckpt --input in/test-midas3/ --output out/test-midas3/
image_shape ()
### Preparing Model
Model: SemiSupModel
DepthNet: PackNetSlimEnc01
Traceback (most recent call last):
  File "scripts/infer.py", line 162, in 
    main(args)
  File "scripts/infer.py", line 129, in main
    model_wrapper = ModelWrapper(config, load_datasets=False)
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 57, in __init__
    self.prepare_model(resume)
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 72, in prepare_model
    self.model = setup_model(self.config.model, self.config.prepared)
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 463, in setup_model
    model.add_depth_net(setup_depth_net(config.depth_net, prepared))
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 403, in setup_depth_net
    args={**config, **kwargs},
  File "/workspace/packnet-sfm/packnet_sfm/utils/load.py", line 110, in load_class_args_create
    class_type = load_class(filename, paths, concat)
  File "/workspace/packnet-sfm/packnet_sfm/utils/load.py", line 105, in load_class
    raise ValueError('Unknown class {}'.format(filename))
ValueError: Unknown class PackNetSlimEnc01

Same issue with PackNetSAN01_HR_sup_D.ckpt:

root@ms1-PCx0Dx:/workspace/packnet-sfm# python3 scripts/infer.py --checkpoint /data/checkpoints/PackNetSAN01_HR_sup_D.ckpt --input in/test-midas3/ --output out/test-midas3/
image_shape (384, 640)
### Preparing Model
Model: SemiSupModel
DepthNet: PackNetSlimEnc01
Traceback (most recent call last):
  File "scripts/infer.py", line 162, in 
    main(args)
  File "scripts/infer.py", line 129, in main
    model_wrapper = ModelWrapper(config, load_datasets=False)
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 57, in __init__
    self.prepare_model(resume)
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 72, in prepare_model
    self.model = setup_model(self.config.model, self.config.prepared)
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 463, in setup_model
    model.add_depth_net(setup_depth_net(config.depth_net, prepared))
  File "/workspace/packnet-sfm/packnet_sfm/models/model_wrapper.py", line 403, in setup_depth_net
    args={**config, **kwargs},
  File "/workspace/packnet-sfm/packnet_sfm/utils/load.py", line 110, in load_class_args_create
    class_type = load_class(filename, paths, concat)
  File "/workspace/packnet-sfm/packnet_sfm/utils/load.py", line 105, in load_class
    raise ValueError('Unknown class {}'.format(filename))
ValueError: Unknown class PackNetSlimEnc01
maliksyria commented 3 years ago

same error here

VitorGuizilini-TRI commented 3 years ago

Our models were trained using driving datasets, so it's expected that they do not transfer well to these new domains. You would have to train new models on more diverse datasets (like MiDaS did), to achieve this level of accuracy.

VitorGuizilini-TRI commented 3 years ago

About the error, seems like the pre-trained models use a different name for the encoder, and you try explicitly setting the network in your config file during inference? The training scripts provide the correct configuration, you can use those for inference as well.