Feature request for displaying 3D animation of gloss

[rem0g]

For this year we have a target to capture a certain amount of signs, it may be between 100-1000 depending on quality of the animations. Now we have about 200 animated signs on signCollect we would like to publish on Signbank.

We would need:

• An API to save/overwrite FBX file
• Display FBX file and showing the sign
• Interface to modify the FBX file, that would be: delete and add new FBX file

We already have 3D animation interface running on https://leffe.science.uva.nl:8043/jari/BabylonSignLab/LoadingAnimation/main.html?limit=10. This is developed by me and my colleague Jari. When implementing the interface on Signbank, Jari would be able to assist. Click on "Load animation sequence and play them", it will then load 10 animations and play them.

Let me know what is needed to implement such a feature. This is not a priority but deadline would be somewhere in oct/nov.

Also tagging @uklomp about this topic, and i would like to add Jari Andersen (https://github.com/J-Andersen-UvA) to this issue. Our github repo is: https://github.com/J-Andersen-UvA/BabylonSignLab

[susanodd]

Do you wish

• to display the animations within the Gloss Detail View ?
• a separate Animated Sign View / Panel ?
• a separate Search / List View for the animated signs ?

[rem0g]

@uklomp we should discuss this seperatedly with our colleagues.

Would it be possible to start ahead with backend code and discuss the integration with Jari, Susan?

[susanodd]

@uklomp we should discuss this seperatedly with our colleagues.

Would it be possible to start ahead with backend code and discuss the integration with Jari, Susan?

Yes, sure.

To connect the code to the rest of signbank, those are suggested "connections". Plus urls, of course.

[Woseseltops]

Cooool. Indeed, the backend could already be implemented, I guess it could be called something like GlossAnimation, similar to GlossVideo, with fbx files instead of mp4 files.

@rem0g , can you also display these files in a 'small windows'? I'm wondering about the UI, whether it could go in between the video and the image in the gloss detail view.

[rem0g]

It would the best to display in a seperate window, indeed. The UI is already busy at it is.

[Jetske]

Cool! If no one started on this yet, then I will.

[susanodd]

@rem0g does this work on Firefox / Chrome on Ubuntu? The example link https://leffe.science.uva.nl:8043/jari/BabylonSignLab/LoadingAnimation/main.html?limit=10 just seems to be a guy hanging in space with a pineapple. There isn't any signing going on.

There are some browser errors (on Firefox):

Uncaught ReferenceError: module is not defined
    <anonymous> https://leffe.science.uva.nl:8043/jari/BabylonSignLab/LoadingAnimation/SceneAndMeshLoader.js:311
[SceneAndMeshLoader.js:311:1](https://leffe.science.uva.nl:8043/jari/BabylonSignLab/LoadingAnimation/SceneAndMeshLoader.js)
Uncaught ReferenceError: module is not defined
    <anonymous> https://leffe.science.uva.nl:8043/jari/BabylonSignLab/LoadingAnimation/animFetchAndDestroy.js:165
[animFetchAndDestroy.js:165:1](https://leffe.science.uva.nl:8043/jari/BabylonSignLab/LoadingAnimation/animFetchAndDestroy.js)
Uncaught ReferenceError: module is not defined
    <anonymous> https://leffe.science.uva.nl:8043/jari/BabylonSignLab/LoadingAnimation/retargetAnims.js:260

What file formats would need to be on hand on signbank? On your repo, I find glb and fbx as well as json. Would the files be stored / organized with the gloss? [See new branch, I put the fbx file in class model GlossAnimation]

I created a model/folder "animation" analogous to "video".

Have you implemented any Python or is all the code in javascript and html? If it's under "animation" then it could be rather self-contained, but share models with the rest.

What about the json and glb files, do these need to be stored?

[susanodd]

If you look at the video/models.py file, you see the various classes related to storing the videos. You would need animation/models.py with the classes for the gloss animations. [See commit below, added this.] And the various attributes/fields of such a class. This involves making migrations to create the models in the database.

We will do this on a separate branch and keep "erasing" and "creating again" until the models meet your requirements.

I will set up a base structure.

[susanodd]

@rem0g do you want your display window (the link you sent with the pineapple), the code to be included in a signbank view, as it is? (But shrunk to fit.) Should this be fetched from your server? Or should the code live on the signbank site?

The CNGT scripts of @vanlummelhuizen are included in the requirements, which links to his GitHub repo.

[susanodd]

I created the first "upload new animation" page.

To keep it simple, there is a gloss lookahead for choosing the gloss to upload the animation for.

[susanodd]

@rem0g I put the babylon code that is referred to in the html window on your example into a folder on signbank. (See the commit and the pull request changed files.) I made a detail view that includes the javascript code. I modified the src references accordingly to where the files are.

[susanodd]

Okay, after much tweaking, I've got the pineapple.

[susanodd]

@rem0g the animation viewer page is here:

• https://signbank-test.cls.ru.nl/dictionary/animation/1

I created one animation using this page:

• https://signbank-test.cls.ru.nl/dictionary/animation/add/

for the gloss APPEL-A, with the fbx from your site. It's animation with id 1. The above "display" should actually be showing that for apple. But I don't know how to adjust the code yet.

• https://signbank-test.cls.ru.nl/dictionary/gloss/1462

There is an Animations tab for the Gloss Details now.

It's unclear why the demo code refers to glb files as well as fbx files, whereas this issue wants fbx files.

The Animations panel for the gloss detail now uses a one-liner display for babylon viewer. This needs a glb file so the pineapple is shown. (The demo code is still visible on the first link.)

The easiest solution would be to also include a matching glb file in the Gloss Animation model. There are OS command lines that convert between the two. The babylon viewer already exists.

The demo code (first link) seems to be making sequences of gloss videos, but this does not work (nor on your demo site).

[rem0g]

Looks ok! I want to park this for a bit to prioritize other issues.

[susanodd]

The branch that includes the UvA javascript animation code has been updated to include the current master code.

That branch is not on any server yet. It is waiting for either signbank-dev or signbank-test to be deployed on.