Add 3D visualization components

[ewpatton]

Describe the desired feature

App Inventor does not really provide any capabilities for manipulating/rendering 3D models.

Give an example of how this feature would be used

With new extensions like Facemesh, it would be useful to provide tools for manipulating 3D models based on inputs provided by other components.

Why doesn't the current App Inventor system address this use case?

While it may be possible to simulate 3D rendering using a component like the Canvas, execution will be much slower in the blocks vs taking advantage of the GPU rendering pipeline.

Why is this feature beneficial to App Inventor's educational mission?

This feature will enhance our AI offerings and also opens the avenue to make more games, which is often a way to introduce students to computational thinking concepts.

[preetvadaliya]

@ewpatton I want to work on this issue.

[halatmit]

This is a new capability for App Inventor. Whoever works on this should create not only the components, but also a few apps that demonstrate how kids and teachers might use it. Also, look at other methods for doing this before creating something ad hoc. For example, what if anything does Google MLKit do in Facemesh (which has versions both for Android and iOS)?

[preetvadaliya]

@halatmit sir,

I discussed with @ewpatton when I was looking for an Internship with MIT App Inventor, at that time he suggested me to read some thesis and also suggest me to implement it using OpenGL.

then I started exploring OpenGL and I found that it will not only help to draw and render 3D components also in the future we can implement Augmented Reality using AR Core.

[ewpatton]

@preetvadaliya You should probably start with a design document for what components you would add, what blocks they have, and how they work together.

[halatmit]

@preetvadaliya AR Core sounds like a good thing to build on, since it supports both Android and iOS. I agree with Evan that this should start with a design document. Also, start with just a piece of AR Core and use it to implement an app that might be made by a kid. That's better than starting by working to implement the entire platform.

PS: One drawback of AR Core is that it apparently can't run on older phones. But we can deal with that issue. when we get there.

Also, as Evan recommended, read Nichole Clarke's thesis Accepted thesis

[preetvadaliya]

@halatmit I started writing disign document for this, I already read the Nichole Clarke's thesis Evan sir suggested me this too.

This one thing can open lots of door for development in ML and AI because it is compatible with Tensorflow Lite, AI Core, ML kit.

[VSofficial]

I think using SceneKit componenets will do the work