search

isl-org / DPT

Dense Prediction Transformers
MIT License
1.96k stars 254 forks source link

Deflicker #45

Closed makseq closed 2 years ago

makseq commented 3 years ago

Thank you for such great software! Do you have ideas on how to reduce the flickering effect between frames from the same video?

AlexeyAB commented 3 years ago

  1. Try to use model with absolute depth so it will not recalculate dynamic range for each frame: python run_monodepth.py --model_type dpt_hybrid_nyu --absolute_depth

  2. Additionally, you can average the output between 3 frames

  3. Finally the model should be re-trained on Video with spatio-temporal patches instead of spatial patches or/and it should use RNN/GRU/LSTM-layers

You can use at least the 1st or 1+2 approaches.

makseq commented 3 years ago
  1. Yes, I've done it already. It's still flicking..
  2. Aren't you afraid that the result of step (2) will be blurry?
AlexeyAB commented 3 years ago

Yes, the 2nd will result in blur on fast motion. Therefore, it is best used for video - the 3rd: spatio-temporal patches, RNN / GRU / LSTM, ... or combine Mono-depth with SLAM

makseq commented 3 years ago

Do you know about approaches like this: https://github.com/princeton-vl/DeepV2D/ ? Does it produce more stable results?

AlexeyAB commented 3 years ago

It is better to compare them for your specific task, because they are suitable for different cases:

makseq commented 3 years ago

@AlexeyAB Thank you very much for your responses!

jiaqixuac commented 2 years ago

Hi @makseq , sorry to bother you. Did you find the solution for this? I want to infer depth for some random videos (wide data distribution, seems opt for DPT), and I also get flicker issues.

makseq commented 2 years ago

@jiaqixuac no :( looks like a difficult task..