naver / dust3r

DUSt3R: Geometric 3D Vision Made Easy
https://dust3r.europe.naverlabs.com/
Other
5.46k stars 589 forks source link

Allow to export scene data as JSON for general use outside of Python #10

Open fangq opened 9 months ago

fangq commented 9 months ago

First, congrats to the developers. a powerful tool, found immediate use for my projects.

Because most of our tools are written in MATLAB/Octave, I found that the generated .glbfile is difficult to convert.

I just added a little bit code to allow the demo.py GUI to export data to a JSON/binary JSON construct (with the JMesh mesh data annotation) - which can be potentially parsed/shared in other environments (like JavaScript/Node/C++/MATLAB). I also added a drop down menu in the demo to let user choose output format.

Here are my commits

https://github.com/NeuroJSON/dust3r/commit/a13b18c0cde7496b83c86d95c2778a0e13d07c92 https://github.com/NeuroJSON/dust3r/commit/935ace7089e6b4c88fd86d0d4eb5d3bdbe3895c2

to export to JSON, only one extra dependency jdata (16 kb) is needed. To export to a binary JSON format (for smaller file sizes), another small package bjdata (65 kb).

loading the data in MATLAB/Octave

>> dat=loadjd('/tmp/scene.jmsh');
>> dat
dat = 
  struct with fields:

       images: [1×10 struct]
      cameras: [1×10 struct]
       meshes: [1×10 struct]
    transform: [4×4 double]

>> dat.meshes(1)
ans = 
  struct with fields:

    MeshVertex3: [196608×3 single]
       MeshTri3: [1×1 struct]

loading the data back to Python

import jdata as jd
dat=jd.load('/tmp/scene.jmsh');
>>> dat.keys()
dict_keys(['images', 'cameras', 'meshes', 'transform'])
>>> dat['meshes'][0].keys()
dict_keys(['MeshVertex3', 'MeshTri3'])
>>> dat['meshes'][0]['MeshTri3'].keys()
dict_keys(['Data', 'Properties'])
>>> dat['meshes'][0]['MeshTri3']['Data'].shape
(189024, 3)
>>> dat['meshes'][0]['MeshTri3']['Data'].dtype
dtype('int64')

in my test, the generated scene from 10 images took 43MB size in glb, 44MB in binary json (.bmsh) and 59MB for JSON (due to base64). No noticeable difference in loading/saving speed. For both JSON/binary JSON file, changing the compressor to 'lzma' could lead to much smaller file size.

just want to share this in case others see similar needs. I am also happy to create a PR if the developers are interested in adding this feature.

image

Mr-ZhuJun commented 8 months ago

Hello, I also have related needs, I hope you can add them