search

microsoft / Azure-Kinect-Sensor-SDK

A cross platform (Linux and Windows) user mode SDK to read data from your Azure Kinect device.
https://Azure.com/Kinect
MIT License
1.5k stars 620 forks source link

Hi quality animation 3d scan to 3d mesh conversion #675

Closed Ed-3d closed 5 years ago

Ed-3d commented 5 years ago

Hi. I am architect and designer and i am interesting in work with kinect azure.

I want to scan animation sample - running man in rain cloak.

I see some samples on youtube : https://www.youtube.com/watch?v=pgRTDUvxWoA (old version kinect) https://www.youtube.com/watch?v=1bodwMQ2zv8 https://www.youtube.com/watch?v=Y45RJYq_zuk

But this samples are not clear and do not display maximum quality.

I have a few question:

1) If i use multiple kinect azure (in manual it says 9 kinect can be synchronized) does it increase quality of final point cloud? 2) If i use multiple kinect azure can i record animation with minimum 24 frames per second without big artifacts? 3) If i use multiple kinect azure can i record animation with minimum 10 by 10 cm dynamic visible physical surface deformations(folds of clothes) without big artifacts? 4) Where can i see how "result point cloud" (or 3d mesh reconstruction) by 6 or 9 sync kinects look? (resolution and practical quality)

rabbitdaxi commented 5 years ago

@Ed-3d , thank you for the questions. It is definitely a cool project that you are trying to work on, some feedback for your questions, hope they help

  1. Using multiple kinect azure will provide more views and depth data points of your target scene/object. As you can see that from single view depth camera, either because the invalidated depth pixels (e.g. due to distance/object reflectivity ...) or the fov itself, people may want to use multiple kinect to cover all views and produce a fused point cloud which can help reconstruct a more complete object/sence
  2. The frame rate depends on the compute resource as well as the USB bandwidth. With the minimum system requirement, you can get to our guaranteed 30fps. People choose to connect 1 kinect to 1 pc with multiple pc, or more than 1 kinect to 1 pc. Therefore, you need to design your setup based on your compute resources
  3. The resolution of the depth image is quite high, therefore, from the point cloud perspective, you can use Azure Kinect Viewer 3d view to experiment whether you can get the details by looking at the point cloud, then your meshing algorithm as well as the surface reflectivity/material will also impact the final surface deformation details.
  4. Currently, we do not have a published sample for multiple kinect fused point cloud (it is quite user scenario specific, different people want to do different setup, and further the calibration method/fusion algorithm should be consider the specific situation). The links you find out that sync 2 or 3 are the best examples.

Hope these information helps. Let us know if you have more questions.

Ed-3d commented 5 years ago

@rabbitdaxi thanx for answer, yes i have some extra question.

the fact is that I came up with a conceptually new method for creating and displaying real-time graphics combining both classical techniques and an innovative approach and thats why i really need your little help.

1 - one object (running man on treadmill - i will do tests) and we have 6-9 sync kinects, please send (show me) any example of result meshing based on Azure(maybe someone walk or sit or waving hand), its very important part and every kinect azure is the same...maybe some average quality. Its no examples of kinect Azure result 3d mesh at all - only noisy point clouds. I think you have someone.

2 - Everybody use kinects in the same way - one central object and all kinects "watch" on it (offcourse some diffrent angles and sides) and kinects vision cover all entire central object. If we have 3 kinects we get 3 point cloud of object that combine in 1 result point cloud. Thats what everybody do....Can we use kinect Azure in another way. Imagine running man and 9 kinects. Every single kinect directed closer to a running man to improve quality of scan and we split our target by 3 zones 1-head and sholders 2-chest 3-legs, 3 x 3 patches = 3 point clouds of one main object....how we shall stich seams of clouds = how we combine 3 point clouds in 1 final point cloud seamlessly?

rabbitdaxi commented 5 years ago

@Ed-3d comments to your 2 questions:

  1. The sensor SDK scope is to provide the raw sensor output such as depth image, point clouds real-time. Meshing is definitely one thing quite useful and also there are lots of common approaches that people do or we internally use. Such as MarchingCubes, Poisson... It could be a great sample that you or many other developer can contribute to the Azure Kinect Samples repo. You also can open feature request on feedback hub, the team will prioritize based on resources.
  2. Your idea of amplify each of human sub-zones and combine them together is definitely a good idea. Although the technique behind your idea and many other people that leveraging multiple Azure Kinect synchronized are quite similar. Essentially, you need to calibrate extrinsics between all Kinects, then you can define your world coordinate and transform all the point cloud to the same coordinate from all Kinects. Then depends on user scenario, people may do various post-processing to get to the results they want, such as outlier removal, meshing, smoothing, temporal consistency and so on.

In general, your feedback are great. Thank you for providing feedback.