2emoore4
commented
3 years ago As a DepthAI user, I want to emphasize the importance of having clean/accurate/precise depth maps - it's clear that deep learning is the key to achieving this.
It's definitely possible to clean up depth maps with more traditional filtering, with something like the Bilateral Solver: https://drive.google.com/file/d/1zFzCaFwkGK1EGmJ_KEqb-ZsRJhfUKN2S/view
However there has been much more work recently to apply deep learning to 3d image generation, and more work is coming all the time.
Stereo Magnification introduced Multi Plane Images, and used differentiable rendering to learn to generate them from stereo images: https://people.eecs.berkeley.edu/~tinghuiz/projects/mpi/
Many have extended on this idea, but much of the latest work uses dozens of input images, instead of just two:
DeepView: https://augmentedperception.github.io/deepview/ Immersive Light Field Video w/ Layered Meshes: https://augmentedperception.github.io/deepviewvideo/ Neural Radiance Fields: https://www.matthewtancik.com/nerf
(Not all of these output MPIs, but all are fairly similar)
There's also plenty of recent work around monocular depth estimation, like MiDaS from Intel: https://github.com/intel-isl/MiDaS
Some take existing 3d photos, and try to inpaint disocclusions, so that inaccuracies are less noticeable: https://shihmengli.github.io/3D-Photo-Inpainting/
Luxonis-Brandon
saching13
tersekmatija
dhruvmsheth
nickjrz
PINTO0309
ibaiGorordo
ghost
gurbain
ecmnet
edgarriba
garybradski
ZlodeiBaal
Why not give it a try if you are interested?
Start with the
why:The
whyof this effort (and initial research) is that any many applications depth cameras (and even sometimes LIDAR) are not sufficient to successfully detect objects in varied conditions. Specifically, for Luxonis’ potential customers, this is directly limiting their business success:Autonomous wheelchairs. The functionality above it would be HUGE for this application as existing solutions are struggling with the output of D435 depth. It gets tricked too easily and misses objects even w/ aggressive host-side filtering and other detection techniques.
Autonomous lawn mowing. This use-case is also struggling with object detection using D435. The system can't identify soccer-ball sized things reliably even with significant host-side post-processing and then need to be able to identify down to baseball sized things.
Volumetric estimation of low-visual-interest objects. Disparity depth struggles significantly with objects (particularly large objects) of low visual interest as it lacks features to match. Neural networks can leverage latent information from training that overcomes this limitation - allowing volumetric estimation where traditional algorithmic-based disparity-depth solutions cannot adequately perform.
The original idea of DepthAI is to not solve this sort of problem, but it is well suited to solving it.
Background:
As of now, the core use of DepthAI is to run 2D Object Detectors (e.g. MobileNetSSDv2) and fuse them with stereo depth to be able to get real-time 3D position of objects that the neural network identifies. See here for it finding my son's XYZ position for example. This solution is not applicable to the above two customers because the type of object must be known to the neural network. Their needs are to avoid any object, not just known ones, and specifically objects which are hard to pick up, which are lost/missed by traditional stereo depth vision.
New Modality of Use
So one idea we had recently was to leverage the neural compute engines (and SHAVES) of the Myriad X to make better depth - so that such difficult objects which traditional stereo depth misses - could be detected with the depth that’s improved by the neural network.
Implementing this capability, the capability to run neural inference to produce the depth map directly, or to improve the results of the disparity-produced depth map, is hugely enabling for the use-cases mentioned above, and likely many others.
Move to the
how:The majority of the work of how to make this happen will be in researching what research has been done, and what techniques are sufficiently light-weight to be run on DepthAI directly. Below is some initial research to that end:
Google Mannequin Challenge:
Blog Explaining it: https://ai.googleblog.com/2019/05/moving-camera-moving-people-deep.html Dataset: https://google.github.io/mannequinchallenge/www/index.html Github: https://github.com/google/mannequinchallenge Notice in a lot of caes this is actually quite good looking depth just from a single camera. Imagine how amazing it could look with 2 or 3 cameras.
Could produce just insanely good depth maps.
KITTI DataSet:
http://www.cvlibs.net/datasets/kitti/eval_scene_flow.php?benchmark=stereo
So check this out. A whole bunch of ground truth data, with calibration pictures, etc. So this could be used to train a neural network for sure on this sort of processing.
And then there's a leaderboard downbelow of those who have.
PapersWithCode:
PapersWithCode is generally awesome. They have a slack even.
https://paperswithcode.com/task/stereo-depth-estimation
Others and Random Notes:
So have a dig through there. This one from there seems pretty neat: https://github.com/CVLAB-Unibo/Real-time-self-adaptive-deep-stereo
These guys seem like they're getting decent results too: https://arxiv.org/pdf/1803.09719v3.pdf
So on a lot of these it's a matter of figuring out which ones are light enough weight and so on to see about porting.
Notice this one uses KITTI dataset as well: https://www.cs.toronto.edu/~urtasun/publications/luo_etal_cvpr16.pdf
SparseNN depth completion https://www.youtube.com/watch?v=rN6D3QmMNuU&feature=youtu.be
ROXANNE Consistent video depth estimation https://roxanneluo.github.io/Consistent-Video-Depth-Estimation/