search

LiheYoung / Depth-Anything

[CVPR 2024] Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. Foundation Model for Monocular Depth Estimation
https://depth-anything.github.io
Apache License 2.0
6.4k stars 492 forks source link

Not detecting foreground elements #201

Open eyeEmotion opened 3 weeks ago

eyeEmotion commented 3 weeks ago

I've been testing the Depth Anything v2 (Large) model a bit and compared it to Depth Anything v1. Alhtough it is an obvious improvement over v1, I came across a piece that ommits a lot of parts in a scene. I think it has to do because of the mirror next to it. It probably flattens things in the mirror, but also flattens the objects next to it.

I'll try to show it here:

Original image:

daysOfThunder_original_04

Depth Anything V1:

daysOfThunder_dptAnyV1_04

Depth Anything V2 (large):

daysOfThunder_dptAnyV2_04

As you can see, the flowers in front of the mirror are picked up and rendered in version 1, but not in version 2. But in version 1, it does create depth for the reflection in the mirror, which also isn't right, in comparison to version 2.

Cheers.

LiheYoung commented 3 weeks ago

Hi @eyeEmotion, thank you for providing this valuable failure case. We also think it is affected by the mirror. In the future we will dig into it and strengthen our models.

eyeEmotion commented 2 weeks ago

Ok, thanks.

In this example, I think it's a combination of the mirror next to it, as it being dark elements, where it is struggling. I've come across another example, where there was no mirror, but the front objects were 'in the dark' and it just ignored it. The mirror/glass/window thing is a difficult problem to tackle. I see every AI struggling with it. As with a mirror reflection, everything needs to be flat, But with a window, you still need depth. This is were Depth Anyhting v2 also struggles with. I'll post an example of that aswell later.

Special effects also seem to still be a struggle, and in some cases seems to even take a step back in v2 compared to v1. I'll post that example aswell later on.