Machine Learning / Deep Learning

[mmatiaschek]

The Learning/Prediction Pipeline

Objective:

3 million children are dying of malnutrition every year. We need a game-changer to identify malnutrition of children, to replace the manual measures of weight and height, which are costly, slow and often inaccurate. Our mobile app scans collects 3D point clouds and video data from children to extract anthropometric measurements like

• height,
• head-circumference,
• middle-upper-arm-circumference,
• body volume / body composition / weight.

Also, for rapid assessment especially in offline regions, and for cheap mass-market smartphones, classification of severely or moderate acute malnourished, normal or overweight children from video would be valuable.

Special attention is given to the fact that uncooperative children and imperfect lighting and internet connectivity need to be addressed for a useful general approach.

Our goal is to do online learning, so we can gradually improve the quality of our measurements.

Available data

Our current Dataset consist of

• gold standard anthropometric measurements of height, weight, MUAC and head-circumference from 12 healthy children
• more than 700 3d scans of those children (~240 complete front/360°turn/back scans)
• more than 30.000 point clouds (min/avg/max per scan 1/30/918)
• each point cloud has a timestamp and up to 38.000 points
• each point has XYZ float values in meters

which amounts to 10 GB of zip compressed point cloud data

Throughout the next 6 month we will collect data of

• ~10.000 children of which around 40% will be MAM and 2-4% are SAM
  • https://actionagainsthunger.ca/what-is-acute-malnutrition/types-of-acute-malnutrition/
• with gold standard anthroprometric measurements as above mentioned
• with a refined scanning process that will provide between 6 to 30 point clouds per scan
• a confidence value with every XYZ point (XYZC)
• RGB or YUV video of the children

more than 250 GB of data

Concepts to explore

Absolute millimeters from Point Clouds

• [ ] Data preparation
  • noise reduction #46
  • point cloud rating and selection
  • Area learning, person extraction
  • device pose elimination
• [ ] Feature extraction
  • point cloud segmentation
  • body part detection
  • pose estimation
  • skeleton generation
• [ ] result generation
  • [ ] intermediate results
  • UX #34
  • combining 2D/3D features
  • spatio-temporal graph made up of XYZC points
  • [ ] final results
  • Height/Length
  • Weight
  • Head Circumference
  • Middle Upper Arm Circumference

SAM/MAM/Normal/Overweight Classification

• [ ] image classification via AutoML
• [ ] Geometric Morphometrics for Anthropology

[mmatiaschek]

Interesting work on point clouds

• https://github.com/charlesq34/pointnet2
• https://github.com/griegler/octnet
• https://github.com/lijiannuist/Vote3Deep_lidar
• http://shihaowu.net/
• Frustum PointNets for 3D Object Detection from RGB-D Data
• 3D Point Cloud Classification and Segmentation using 3D Modified Fisher Vector Representation for Convolutional Neural Networks

List of Links: https://github.com/bluebox42/3D-deep-learning

[AI-Guru]

After a couple of weeks working in the project I would love to share some ideas.

Multi-View CNN - Something to definitely have a look at

See for example: http://vis-www.cs.umass.edu/mvcnn/

Or in a speaking picture:

What does it do? It takes into account an ordered sequence of pictures. It is the 360deg view we have been talking about a couple of times.

How to turn this into a process? What are the alternatives for capturing images?

1. Let the kid rotate a couple of degrees, then take a photo. Repeat a couple of times until you reach full circle. Could be very bad when dealing with uncooperative kids.
2. Let the measurement-pro take rotate. Let the kid stand still and move around it. The app could automatically take pictures using the gyroscope. This would be similar to panorama-photos-stitching on smartphones, but inverted. I would at least do an experiment if you could automatically take pictures based on rotation.
3. Create an array of n cameras in a circle and take photos with that. This is definitely something that would work well. But it is also against the app idea.

As a start we could comb through the already existing data and see if a fitting dataset could be derived from that.

I believe this can be generalized to use also pointclouds instead of pictures.

One option for showing the point: Artificial data.

You can get some inspiration from the following, excellent Apple-article: https://machinelearning.apple.com/2017/07/07/GAN.html

Bottom line: Usually you need way too many images to solve problems that are similar to ours. A solution would be to use rendered data. Apple found out that this works. And they had to add a refining step that makes the artificial data more realistic.

I did some quick research and experiments. There is already some research done that is in our domain: http://grail.cs.washington.edu/projects/digital-human/pub/allen03space-submit.pdf Looks like parametrized models. You can change body metrics (e.g. weight, height) and body features (e.g. skin tone) of 3D models.

All you need to to is to create a pipeline that automatically generates images/pointclouds from some parametrized models. I tried Blender for a couple of minutes. It can be called from the command-line, this making it perfect for automatic rendering of 3D data. Note that I did not do an extensive research. I just used Blender as an example. Here are some simple images rendered from a simple model:

Definitely have a look at Apple's ARKit2

Look first:

See this picture animated in this great article: https://medium.com/@mohams3ios01/an-introduction-to-arkit-2-world-mapping-5b38827f8ec0

ARKit2 is currently beta. It will be available in autumn. Amongst other things it makes the World-Map available to developers. This is a data-structure that maps reality into a point-cloud of some sorts based on data from the motion sensors and 2d-camera data.

Questions:

1. What is the resolution of the World-Map? Is it high enough to facilitate Machine Learning?
2. Can the World-Map be used for the rotating data-set generation (see above)?
3. Is a similar functionality available on Android?
4. How will this technology will improve over the next couple of years? Might it fit our needs in the near future?