From Unity to Autodesk by FBX export

[BonHammer]

Hello,

The FBX export of the gaussian splatting unity project I produced is barely a few kb and failed to open on Autodesk Inventor. I guess the unity FBX export isn't finding any scene to export in Unity. (2022.3.10f1)

Any idea on how to Unity could export a gaussian splatting scene into any 3d CAD file? Or what must be changed to the unity object to ensure exporting?

I followed the user manual: https://docs.unity3d.com/Packages/com.unity.formats.fbx@4.2/manual/exporting.html

[andybak]

I'm sure @aras-p can explain better than me but "it's not that simple".

The whole point of Gaussian Splats that they encode the scene is a different way to conventional mesh-based representations. Each point is a quad that is always camera facing and renders in a way that is dependent on view direction.

You can capture this in a complex shader that has access to per-point data, but few there's no common interchange standard for this data so each app would need to understand the data being sent to it and do some fairly complex decoding and rendering. Fbx is based around a standard PBR (physically-based rendering) paradigm. You can probably jam in arbitrary metadata but each receiving app would need code to specifically extract that and manipulate the rendering accordingly.

[ThomasWarn]

It isn't impossible that you could import the data into autodesk inventor, but to do so without a massive overhaul or you would likely need to convert the splats back into points - and then the only unique feature of gaussian splatting to the best of my knowledge is the segment of the gaussian splatting training loop that generates new points off of the initial point cloud.

That's to say that at this point you may want to consider looking into more traditional photogrammetry methods.

[aras-p]

Yeah, the reason why FBX exporter does not export the splats, is because they are not meshes or any other typically found 3D scene representation (yet?). There probably exist some ways to extract a 3D mesh representation from a gaussian splatting scene, but someone has to invent that way first, and then develop software to do it. This repository is certainly not it.

[BonHammer]

I only mentioned FBX as its the official means of file exchange between Unity and Autodesk applications and since this is currently the only repository I found that could output to a 3D application. I had hoped I could create a 3D CAD export and almost any format could be used.

For reference, the complete file format list: https://help.autodesk.com/view/INVNTOR/2024/ENU/?guid=GUID-AF41FA87-7588-4698-9C41-756A01EBE7F4

Thanks for the more in dept explanation, I am no expert on this. I build overhead cranes. I tested the rendering and was impress by how accurate it can recreate digital twins of 3D environments with only a camera that would solve multiple industrial problems with all the other means of creating 3D mesh or point cloud environments.

I am trying to convince my colleagues that this technology could be a feasible way to take approximate measurements of old cranes to ease their discussions for maintenance and upgrades. I understand that this is a passion project for research purposes and I am looking forward to what you will develop next. I do not have the technical capacity to help on this technology, but I can navigate legal ways to share unique industrial environment gaussian-splatting scans if I ever get the ability to export it to a CAD software.

[aras-p]

@BonHammer the issue is not that "FBX is not supported", but rather that FBX (nor any other format right now) has a concept of what a "gaussian splatting scene" even is. FBX supports polygonal meshes, point clouds, curves, subivision surfaces and so on, but not "gaussian splats".

And export is just one side; the other one is that whatever application you'd import this into, has to also know how to render the gaussian splats. Again, pretty much 100% of all applications / DCCs / CAD tools right now can not do that. The paper that started this all only came out two months ago (well ok, not quite, the original "gaussian splatting" concept was introduced 20 years ago, but was largely ignored... until now).