Component for retrieving objects from a 3D model

[feiss]

A very convenient scenario for handling 3d scenes could be having all the objects in the scene saved in just one 3d file (obj, json, gltf,...) with the objects inside the scene properly named. So there would be just one file to load, just one <a-asset-item>, simplifying also all file handling and workflow.

For this to work, we would need a way of assigning the objects from the scene to individual entities. Something like this could be nice:

<a-assets>
  <a-asset-item id="room" src="room.gltf"></a-asset-item>
</a-assets>

<a-entity model-object="from:#room; obj:chair"></a-entity>
<a-entity model-object="from:#room; obj:table"></a-entity>

Is there anything like this available already?

[donmccurdy]

+1, there was a similar question on SO: http://stackoverflow.com/questions/42074856/aframe-clone-object3d-within-entity/42076326?noredirect=1#comment71360013_42076326

Probably an external/registry component for starters?

[dmarcos]

Should be a property of the obj-model, gltf-model... or agnostic to those?

[donmccurdy]

Agnostic I think, not aware of any format-specific needs.

[feiss]

Yeah, once the model is loaded and in a THREEJS object, we can extract any object in an agnostic fashion.

@donmccurdy yeah, something like that component. Do you think it is such an edge case that should not be in core?

[donmccurdy]

I'm happy either way. Maybe I'm not familiar enough with modeling workflows to know if bundling assets into a single file is common/useful.

[fernandojsg]

We should differentiate the uses where you want to clone the object or just use a reference to it

[dmarcos]

Could this be a good feature for 0.8.0?. Once a model is loaded there's a lot manual work to do to manipulate the model (change textures, access sub models, animate...) and A-Frame is not making it easier to deal with them. There's an opportunity for us to provide a convenient API to manipulate models after they are loaded (obj, collada or gltf). @feiss idea was that any component can define an onModelLoaded method that gets called whenever a model loads on the entity. a model object is passed as an argument that contains an convenient API for manipulation either declarative like the one originally proposed (https://github.com/aframevr/aframe/issues/2408#issue-208426083) or imperative like:

• model[name] to access sub objects.
• model.setLambertMaterial(selectorObj, lambertProperties)

[donmccurdy]

Once a model is loaded there's a lot manual work to do to manipulate the model (change textures, access sub models, animate...)

Could we collect a list of these use cases somewhere, in a bit more detail? I'm not sure I have a good enough handle on them to propose an API. A few comments on these specific items:

• change textures — this is definitely something users ask for a lot, but, i think many don't understand how UV wrapping works... if we give them this API, is it just going to be a footgun?
• access sub models — +1 for doing something here, although not sure yet whether it belongs in core.
• animate — i think this is all set? we could move animation-mixer into core if that helps, but note it doesn't work with COLLADALoader.

[feiss]

I consider accessing sub objects pretty important. It would allow users to pack all (or many) objects in just one scene, so it can be much more convenient both authoring the assets and loading them (with the additional benefit of having only one request). Also, I can imagine a lot of cases where you want to access to a subobject for changing its properties. Think for example, changing the color of a button of a controller, rotating a head, fading out a tooltip..)

A-Frame helps user to layout and customize basic objects very easily in a declarative way, but when it comes to full, complex or simple 3d models, it just load them and tell the user: "ok, now you are alone with this object and threejs, I won't help you any more". And I think that is a big barrier for many users.

I think we could help in two ways:

1. Adding a model-object component like the one suggested in this issue. Declarative, easy, very A-Framey.

2. Adding a lightweight abstraction API of threejs tightly coupled to the A-Frame component structure, introduced by @dmarcos in his lastest post. I initially thought of something like this:

get(selector)
setVisible(selector, visible)
hide(selector)
show(selector)
toggleVisible(selector)    
remove(selector)
setMaterial(selector, material)
setLambert(selector, params)
setPhong(selector, params)
setConstant(selector, params)
fadeTo(selector, to, duration, delay)
fadeIn(selector, duration, delay)
fadeOut(selector, duration, delay)
setMaterialParameters(selector, params)
setShape(selector, shape, weight, duration, delay) //shape keys
play(selector, animation, delay) //animations
stop(selector)
getMaterials(selector) //keyed array with of all materials

properties: 
.loaded // model is loaded
.materials[] // keyed array with all materials
.meshes[] // keyed array with all objects

• parameter selector can be a string or an array of strings, for selecting objects by name. Regexp could be an option too.

Example of use:

      AFRAME.registerComponent('mycomponent', {
        modelLoaded: function(evt) {
          this.setLambert('cube1', {color:'red'});
          this.setPhong('cube2', {color:'green'});
          this.setMaterialParameters('head', {map:'#facetex2'});
          this.setShape('face', 'mouthopen', 1, 2});
          this.fadeOut('sphere', 5, 10);
        },
        tick: function (time, delta) {
          if (!this.loaded) { return; }
          this.meshes['chair'].position.x = Math.sin(time / 100) * 0.1;
        }
      })

(of course this is not a very well thought implementation, take it as a simple and naive first approach)

[donmccurdy]

I consider accessing sub objects pretty important. It would allow users to pack all (or many) objects in just one scene, so it can be much more convenient both authoring the assets and loading them (with the additional benefit of having only one request).

+1 on this, I agree. Although, there are at least two use cases, that might require different implementations:

1. After loading a model, unpack all children into new (A-Frame) entities in the scene. User can then use raw JS, (or Blender custom properties?), to apply components to each.
2. Before loading a model, user declares entities that should pull specific things out of that model (for example, the original post in this thread).

If either should be in core, I think it would be (2). I've implemented (1) at times as well, but maybe that is more niche.

Adding a lightweight abstraction API of threejs tightly coupled to the A-Frame component structure

I'm more nervous about this. The three.js APIs are already a fairly high level abstraction around WebGL. And, I think it's a benefit not a drawback that you wind up writing three.js code (and using three.js docs + stack overflow) when working with A-Frame. Most of those things can be done fairly cleanly with three.js. The fadeIn/fadeOut methods are an exception, but I think that's a case for considering new imperative animation APIs rather than new abstraction on top of three.js.

[machenmusik]

Another way I was thinking of this was to make the model loader create A-frame entities with selector-friendly id values from the sub-object names, which is along the lines of loading a scene or hierarchy rather than object by object

Sorry just realized @donmccurdy said much the same, apologies for duplication

[machenmusik]

So maybe a hierarchical loader that is different, that would unpack into an entity tree... model-tree? Separate from single-entity loaders to avoid confusion.

[donmccurdy]

Good points. The distinction about re-creating the hierarchy is interesting. I've avoided that when trying this in the past (nesting is a problem with physics, which was my use case). But other users might want to preserve their model's structure.

[netpro2k]

Yeah I think preserving structure is important in many use cases, for example when you want to grab references to joints for a skinned mesh.

I like the idea of 2 components, 1 for cloning individual objects out of the model (the original thing proposed above) and 1 for making existing objects in the model hierarchy selectable and configurable with components.

For example:

<a-assets>
  <a-asset-item id="room" src="room.gltf"></a-asset-item>
  <a-asset-item id="avatar" src="avatar.gltf"></a-asset-item>
</a-assets>

<!-- clone the mesh named "chair" from room.gltf into the scene -->
<a-entity model-object="from:#room; obj:chair"></a-entity>

<!-- Make references to  the specific named nodes inside avatar.gltf available as aframe entities -->
<a-entity id="player" gltf-model="#avatar">
  <a-entity id="head" camera look-controls wasd-controls></a-entity>
  <a-entity id="left-hand" model-selector="left_hand_joint" hand-controls="left"></a-entity>
  <a-entity id="right-hand" model-selector="left_hand_joint" hand-controls="right"></a-entity>
</a-entity>

This is nicer than the approach of just inflating all entities as it both saves you the overhead of doing so and also allows you to declaritively configure the components to bind to parts of the model.

I see 2 tricky bits here:

1. The 1:1 mapping of the ThreeJS hierarchy to the aframe hierarchy is broken down a bit here, the model-selector entities actually represent some entities that might be deeply nested in the parent's structure but only show 1 level of nesting in aframe. Not sure how this would be done without adding some hacks to setObject3D.
2. Ideally there would be a way to defer initializing the other components on the entity until the model has loaded in a way that doesn't require those components to know they are dealing with entities inflated from a model file. Maybe this could be solved by doing something similar to aframe-template-component like:

<a-assets>
  <a-asset-item id="avatar" src="avatar.gltf">
  </a-asset-item>
  <script id="avatar-selector-tempalte">
    <a-entity id="head" camera look-controls wasd-controls></a-entity>
    <a-entity id="left-hand" model-selector="left_hand_joint" hand-controls="left"></a-entity>
    <a-entity id="right-hand" model-selector="left_hand_joint" hand-controls="right"></a-entity>
  </script>
</a-assets>

<a-entity id="player" gltf-model="src: #avatar selector-template: #avatar-selector-template"></a-entity>

[ngokevin]

I've done https://github.com/ngokevin/kframe/tree/master/components/gltf-part#aframe-gltf-part-component for my own needs

[netpro2k]

@ngokevin this is cool for handling the first case, where you want to clone individual objects out of a gltf file, but doesn't work for the case where you only want to reference nodes but keep them in their original hierarchy. My current usecase for this is referencing joints for skinned meshes and specific materials.

[ngokevin]

three.js: mesh.getObjectByName('whatever')

[netpro2k]

Yep, that's what I am doing now, would just be nice to have some declarative way to get at these so reusable components (with no knowledge that they are operating on some deeply nested node) could be added to them.

[ngokevin]

would be awkward if the hierarchies don't match up. So perhaps if you use my part component, and have another component that generates an entity for every single part. And then attach components afterward?

[donmccurdy]

I think there will be lots of different use cases for this, probably not possible to have one component that does it all... Seems best to just make components as the need comes up and figure out what works well.

[ngokevin]

https://github.com/supermedium/superframe/tree/master/components/gltf-part

or https://github.com/supermedium/gunters-of-oasis/blob/7d03d979e2a1dc40cf75957a4b1707df5a52b71f/src/components/sub-object.js

Or wait for model to load and manually fetch. If more API is needed, you can write your own component, and we can see if it's worthy later.