add 2d text (as the texture of a plane)

shensquared commented 6 years ago

Following the suggestions in https://github.com/rdeits/meshcat/pull/41 Here's an initial attempt

Still a lot to clean up but want to first check, does this look like what you had in mind?

rdeits commented 6 years ago

Yeah, awesome! So, one question we should resolve is how we want to represent this in the meshcat interface. For example, one way to do it would be to add a new kind of object (like the way _meshfile works) which includes the specifications for the plane and the text on it. A second option would be to just expose the relevant pieces: for example, we could add a Plane geometry and create a TextTexture, and then drawing text on a plane would consist of sending a Plane with a TextTexture from the Python side.

I think I prefer the second option: it requires a bit more logic on the Python side, but it means that we can write text on anything, rather than just planes. And we can create convenience functions in Python so that simple tasks like drawing text on a flat surface is still easy. What do you think?

shensquared commented 6 years ago

Agreed, option 2 is more versatile and python friendly, which I guess is the point.

I think here’s what’s probably necessary for the job:

Add Plane(Geometry) and TextTexture(Texture) in geometry.py

Add set_text() method for the Visualizer() class, which is then handled by a new SetText() class in command.py, and it looks kinda like:

class SetText:
__slots__ = ["object", "path"]
def __init__(self, text, geometry_or_object=None, material=None, path=[]):
    if geometry_or_object is None:
        geometry_or_object = Plane(width, height, h_segments, v_segments)
    text_texture = TextTexture(text)    
    material = MeshBasicMaterial(map=text_texture)
    # may also need a separate treatment if the object already has material 
    # and we are replacing the texture instead of initializing it like in here
    self.object = Mesh(geometry_or_object, material)
    self.path = path

def lower(self):
    return {
        u"type": u"set_object",
        u"object": self.object.lower(),
        u"path": self.path.lower()
    }

Either add a call of set_object at the end of set_text on python side, or in handle_command on JS side, change this line to include or =="set_text" https://github.com/rdeits/meshcat/blob/c7dcbca3af0e76f3c22573076a8d8e54b52dff67/src/index.js#L626

The only part I don’t immediately know how to do is actually creating TextTexture() in python. I know how to do it in JS now, but the tutorials I found all depend on creating a canvas in HTML first and then add the text via fillText. Not sure how this canvas business can be handled on the python side?

rdeits commented 6 years ago

Yeah, this sounds pretty good. Right, writing text into a canvas is possible in python, but I'd rather not bother with it. Instead, I think we can send a set_object command as you suggested, and we can make the TextTexture.lower() method return something like this:

{
    uuid: "<insert uuid here>",
    type: "_texttexture"
    text: "Hello world",
    font: "sans-serif",
    width: 200,
    height: 100,
    position: [10, 10],
}

Then on the Javascript side, we would need a little bit of logic to look for _texttexture textures and replace them with images by rendering the text into a Canvas using the provided arguments.

In that way, we can still rely on the browser's very convenient text rendering, while still staying very generic and composable.

Does that seem reasonable?

shensquared commented 6 years ago

Yep, sounds very good and doable. Will try it and update here.