Coordinate system mismatch between Bevy and glTF

[pkupper]

Bevy version

0.8.0

What you did

1. Spawn entity from glTF scene with recognizable forward direction (e.g. character, vehicle).
2. Transform the entity along the Transform::forward() direction.
3. Observe model moving backward.

What you expected

Model moves forward.

What went wrong

Both Bevy and glTF use a right-handed Y-up coordinate system. They do however disagree on what direction is forward and right:

• glTF defines forward as +Z and right as -X (see https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.pdf).
• Bevy's Transform defines forward as -Z and right as +X (see https://docs.rs/bevy/latest/bevy/prelude/struct.Transform.html#method.right and https://docs.rs/bevy/latest/bevy/prelude/struct.Transform.html#method.forward).

Possible solutions

1. glTF importer negates X and Z coordinates during import (breaking change for all bevy apps that use glTF import).
2. Bevy Transform is changed to follow the glTF convention (breaking change for all bevy apps that use one of the Transform's direction functions).
3. Bevy stops defining a forward, back, left, right direction and leaves it up to the user (e.g. by changing the forward, back, left, right functions on Transform to local_neg_z, local_z, local_x, local_neg_x).

In any case, Bevy should be consistent regarding its coordinate system. Currently, the look_at function follows the glTF convention and not the Transform's definition of forward and right (see https://github.com/bevyengine/bevy/issues/1153)

[superdump]

Interesting. I didn't know that glTF stated that if a model has a clear forward direction, that it should point along +z. I thought it was up to the modeller / team to define their convention.

The question of "what direction is forward?" came up in the rendering Discord channel a while ago. My answer back then was that whatever is the natural forward direction of the model defines what its forward should be, but also that for a camera, -z is forward because the identity transform leaves the camera looking along -z as per projection conventions for right-handed x-right, y-up, z-back.

glTF has a convention where forward is +z, but does that mean all glTF models have front pointing along +z? Also, I expect bevy will support importing from other formats through plugins or so, and I don't know if they have such conventions too.

My initial feeling was that I don't like proposal 1, and either 2 or 3 would be agreeable. But if one considers other formats having other conventions, or just that any imported model from any format, regardless of suggested conventions, may have a different direction for forward, maybe what we should do is make it easy to specify the forward direction of the model.

Perhaps @cart has already thought about this and has some opinions, I don't know. :)

[pkupper]

I actually prefer proposal 1 over the other two. My reasoning is as follows:

Ideally, the direction that is forward in the modelling application (e.g Blender has a front, top and right view which determine this) is preserved through the asset pipeline (export + possibly some intermediate conversions + import) such that in Bevy Transform::forward() aligns with this direction. Why drop this information somewhere along the way (proposal 3) if you have it and can make use of it.

Therefore for every conversion, when both the previous and the new format have such a convention, the conversion needs to preserve the forward direction (Blender certainly does so for it's export). This includes the import from a given model format into Bevy. I assume you agree that if Bevy were to support a Z-up file format, the loader should definitely do some coordinate transformation instead of just taking X, Y and Z as is. So it only makes sense for the glTF loader to transform from the glTF convention to the Bevy convention.

The question then is, what the Bevy convention should be. You presented a reason for -Z being forward in Bevy which already matches the current Transform implementation. I don't think there is any reason to mix conventions when it is avoidable.

Looking at https://github.com/bevyengine/bevy/issues/1153 it might however also require a change to the look_at implementation to be consistent there.

[nicopap]

I'd like to see how this is solved, it's relevant to:

• https://github.com/HeavyRain266/bevy_mod_fbx/issues/14

[pkupper]

Here's some discussion on this issue that happened in the Discord. Copying it here for discoverability.

norgate — 17.11.2022 23:52
Can I get some opinions on the way forward for https://github.com/bevyengine/bevy/issues/5670. Since the solution would likely be breaking for all Bevy games using glTF assets this should probably be settled sooner rather than later. I'd be happy to create a PR for any solution the community can agree on.

Alice 🌹 — 17.11.2022 23:54
I would merge 3 (rename forward method) pretty readily. 2 (change forward) might be better, even if it is annoying.

1 is way too implicit and janky for my taste

Griffin — 17.11.2022 23:57
I feel like 3 would hurt bevy ecosystem consistency/compatibility (or at least make it harder) 

François 🥷🏻 — 17.11.2022 23:58
2!

Griffin — gestern um 00:00 Uhr
2 feels a bit weird considering our infinite inverse -z. Would that change? Or would there just be a disconnect?

norgate — gestern um 00:01 Uhr
Something like 1 would always be necessary for importers of different file formats. Most obviously the ones that use z-up. The issue with 2 is that it solves the problem for glTF only.

François 🥷🏻 — gestern um 00:01 Uhr
ok maybe not 2. but for me the big issue is not with gltf, that doesn't matter that much. it's
Currently, the look_at function follows the glTF convention and not the Transform's definition of forward and right (see #1153)
so either change look_at or forward

Griffin — gestern um 00:01 Uhr
It looks like it's definable in both FBX and USD

norgate — gestern um 00:02 Uhr
@Griffin that is only relevant if you create your own assets.

Griffin — gestern um 00:03 Uhr
Can you elaborate?

norgate — gestern um 00:05 Uhr
I think you are suggesting that FBX and USD files can follow the Bevy convention to avoid this issue. But that is only possible to control if you create the file. Some asset downloaded from the internet might use a different convention.

İra — gestern um 00:21 Uhr
look_at is correct, all the 3d examples use it for the camera after all.
It's just that the code is a bit confusing because forward  seems to point away from the target, which does make sense since forwards in Bevy is towards the negative Z 

norgate — gestern um 00:36 Uhr
I want to understand your concern about coordinate transformation in the loader. Does 1 only seem janky to you in the case where up direction and handedness already match or would you also avoid coordinate transformations when that is not the case? So would it be fine if the model would otherwise end up sideways or mirrored instead of just rotated around the up axis? Or does this just concern glTF in particular as it has so far been Bevy's "native" format? Blenders exporters generally maintain the up, forward and right conventions by doing such transformations. I have not yet checked other engines importers but plan to do so.

Alice 🌹 — gestern um 00:40 Uhr
The main challenges here:

1. There's no great place to document this: do we put this in an example? On the Gltf type? On the asset loading?
2. New users will be confused by their coordinate system not matching what they had in blender.
3. Makes "blender as scene editor" patterns more cumbersome.

Fundamentally the choice seems arbitrary, so it feels like we should align with our native 3D format.
We could display an info of "flipping coordinates!" while loading, but that's going to be noisy fast and still introduces friction

norgate — gestern um 00:42 Uhr
Blender already has a different coordinate system than glTF. So when users look at the glTF file there is already a disconnect. So the confusion should already exist. 
The question is whether users value it more to have forward in Blender match forward in Bevy or to have +x in Blender match +x in Bevy. If it is the former than there needs to be no documentation as it would be the expected behavior. 

François 🥷🏻 — gestern um 00:46 Uhr
+x!

norgate — gestern um 01:09 Uhr
As of right now when going from Blender to glTF to Bevy the mapping is
+x -> +x
+y -> +z
+z -> +y
with the transformation being done by the Blender exporter.

What made me open this issue is that when experimenting with writing a flight simulator I found myself writing code like velocity += thrust * transform.backward() after using front/right view in Blender to create my model.

Alice 🌹 — gestern um 01:21 Uhr
Oh that's a pain... In that case I am fine with solution 1 too

Griffin — gestern um 03:09 Uhr
I was just pointing out that it's a common feature, and that our solutions will want to take that into account. I wouldn't want to assume that all content is generated specifically with bevy's convention, rather the opposite. I was more meaning to reinforce that idea that the conventions will be all over the place.
Am I missing something or does option 1 mean that models are mirrored? 
iirc in Unreal this is dealt with by often having the model be a child of the entity, and that local rotation is set to make forward whatever it needs to be. I think in bevy a similar hierarchy may already be in place when a gltf is imported. It seems like the mesh is always a child of the object.
We also badly need importer specialization/configuration 

JonahPlusPlus — gestern um 04:01 Uhr
Is there a way to handle special assets? For example, I want to load a cube map, which can be any sort of image. However, I can't make an AssetLoader for it, since that would conflict with the existing ImageTextureLoader. Is there a way to easily specify the loader or add a callback for editing the asset before it is marked loaded?

Griffin — gestern um 04:06 Uhr
If you find out, let me know 😅

JonahPlusPlus — gestern um 04:06 Uhr
Uh oh
I've been trying to find a way to make it so users can load static sky textures, but if there isn't a user friendly way of doing it, maybe it's not worth it.

Griffin — gestern um 04:09 Uhr
Ohh! I saw your help thread earlier and then forgot to take a look at it.

JonahPlusPlus — gestern um 04:10 Uhr
Yeah, I've been looking for a good solution for a while now 😅
It's fine, I can push that feature back to the next update. Maybe I'll take the time to create a AssetServer::load_with method. 

norgate — gestern um 08:22 Uhr
The coordinate transformation (negate x and z) would rotate the model by 180 degrees but not mirror it.

norgate — gestern um 08:23 Uhr
Thanks for the clarification. I agree this has to be taken into account in future loaders.

Diddykonga — heute um 19:13 Uhr
1 Is the only reasonable approach, the reason being that nowadays it really seems like there will be no consensus on coordinate systems, and a system that understands and supports converting from * to bevy is the most ergonomic.
2 would simply fix the gltf issue, but cause the same barrier for other formats.
3 would be a generalization of the API, reducing ergonomics.

Alice 🌹 — heute um 19:15 Uhr
Right, so you're saying we should automatically adapt all loaded assets from their native coordinate system into Bevy's native coordinate system?

Diddykonga — heute um 19:15 Uhr
On Import as part of pre-processing, yeah.

Alice 🌹 — heute um 19:15 Uhr
🤔 Yeah, that would allow us to quickly and easily import assets from wherever correctly
I think you're right

Diddykonga — heute um 19:16 Uhr
Then Editor Plugins could be loaded/supported to add support for new Assets that would require custom pre-processing. (Eventually) 

Gibonus — heute um 19:17 Uhr
Yeah, should be part of pre-processing
There should be some sort of ECS-native intermediate format with minimal cost to load
All scenes/formats should be converted into that format in a pre-processing step.
(having worked on an FBX loader, I think that's a perfectly fine way of handling it) 
(especially given the insane scene format in FBX)
What's a Cooking phase?

Diddykonga — heute um 19:21 Uhr
When you Release your game, you Cook your Assets to make them small, quick, and secure. Also, to only build/release relevant assets/resources, and ignore non-used ones. 

Gibonus — heute um 19:30 Uhr
I've more often came across "baking" than "cooking" for this specific meaning. 

Diddykonga — heute um 19:32 Uhr
Baking is Cooking :TE_peepoClown:

Gibonus — heute um 19:33 Uhr
fyi FBX allows arbitrary coordinate systems, but in practice mostly the maya one is used, which I think is concordant with the bevy one. But for the loader I setup the scene as child of an entity so that I can add to it arbitrary transforms to handle properly the change of coordinate systems

[cart]

Choosing -z as forward came from the following;

With right handed y-up and x-right, the "default" view (ex: the view in 2d and 3d) points in the negative z direction, with positive z coming out of the screen. In this context, if I move anything in 2d or 3d (a sprite, a 3d model, a UI element) "forward" (relative to the camera view) it is moving in the negative z direction.

This made sense to me. Treating z-positive as forward would mean a "default" camera (standard orthographic projection with identity transform matrix) would be facing "backward".

But if somebody does a survey of popular right handed y-up software in the gamedev space and finds a reasonably consistent z-positive-is-forward, I'm happy to revisit this. I'd also like to know how they handle default cameras in that context though. (ex: do they spawn them rotated 180 degrees around the y axis? do they just let them face backward by default?)

[superdump]

@cart - that response discusses axis conventions and camera forward conventions within those axes. But how do you feel about what we should do with assets that have a strong definition of what forward means, such as gltf that has the same axis conventions but +z is forwards?

[cart]

We definitely want to honor format intentions to the best of our abilities. If after we resolve the "z-forward" conversation we continue considering z-negative as forward, I think we should add a "flip z axis" setting to the GLTF importer, which defaults to true. (Note that importer settings have been added in my asset system rework, which I'll be submitting for discussion and review shortly)

[rlidwka]

(Note that importer settings have been added in my asset system rework, which I'll be submitting for discussion and review shortly)

@cart, can we have a generic "apply custom Transform" setting for GLTF importer?

I have 3d models from sketchfab that are wrongly rotated, moved or scaled. Sometimes I can't fix them due to "no-modification" license clause. I can of course add them as a child entity, but that seems like a needless burden for transform propagation system.

Suggestion: instead of flip_z_axis: true we could have apply_transform: Transform::from_scale(Vec3::new(1., 1., -1.)).

edit: argument could be made for similar setting for all loaded assets (e.g. sprites), but I haven't worked with these enough to see if it makes sense to generalize further

[cart]

This sounds like a good use case for the "Asset Transforms" we've been discussing in #8624, which would allow arbitrary transformations of assets after the initial load.

But I'm also open to an "apply transform" field in the GTLF importer.

[Adriaaaaan]

I would find it very beneficial to have an easy way of setting the coordinate system. We have a system that works cross platform/game engines (multiplayer) and as such went with a vendor neutral coordinate system that gltf uses. Would be great to use gltf for my use case

[viridia]

Here's a handy chart comparing the coordinate systems of various game engines and DCC apps: https://twitter.com/FreyaHolmer/status/644881436982575104/photo/1

[alice-i-cecile]

nickrussell: Ah yeah. Generally +Z forward is the rigging standard, which I argue is the area in game dev that 'suffers' the most from mismatched coordinates. Because it requires so many edge-case scripting and workarounds in practice. I'd def vote for +Z forward. Maya has set the industry standard, gltf follows it. And this will be good for Bevy to fit in existing rigging pipelines.

Opinion from game industry veteran from discussion.

[nickrart]

Choosing -z as forward came from the following;

With right handed y-up and x-right, the "default" view (ex: the view in 2d and 3d) points in the negative z direction, with positive z coming out of the screen. In this context, if I move anything in 2d or 3d (a sprite, a 3d model, a UI element) "forward" (relative to the camera view) it is moving in the negative z direction.

This made sense to me. Treating z-positive as forward would mean a "default" camera (standard orthographic projection with identity transform matrix) would be facing "backward".

But if somebody does a survey of popular right handed y-up software in the gamedev space and finds a reasonably consistent z-positive-is-forward, I'm happy to revisit this. I'd also like to know how they handle default cameras in that context though. (ex: do they spawn them rotated 180 degrees around the y axis? do they just let them face backward by default?)

The convention is that the camera is looking at the front of the asset. And for purpose of viewing and constructing assets this is pretty helpful. For the most part only 1st/3rd person player characters/entities would be accurately represented by a forward facing camera, and even then only by the owning player. For this reason I prefer +z forward.

[nickrart]

Generally I think it sounds like the proposed preprocessing will provide a really solid catch all for any asset pipeline and would be an upgrade to Unity and Unreal, where often tech artists are writing adhoc equivalents. However, mismatching coord systems is consistently a pain point for artists and non-eng in my experience.

[nickrart]

@cart what would a software survey look like? In my mind it'd be, "what DCC do you use, what rigging plugins, and what's your coord system before exporting"

[magras]

Actually gltf 2.0 has different requirements for assets:

glTF uses a right-handed coordinate system. glTF defines +Y as up, +Z as forward, and -X as right; the front of a glTF asset faces +Z.

and cameras:

The camera is defined such that the local +X axis is to the right, the “lens” looks towards the local -Z axis, and the top of the camera is aligned with the local +Y axis.

[nickrart]

Actually gltf 2.0 has different requirements for assets:

This is a great callout. I see some conflation of the viewer's space and world space for what the asset coordinate space should be. The biggest pain points in game production for coordinate spaces in my experience (and anybody I've talked with in the industry) come from rigged assets, i.e. Root Motion, runtime bone attachment. I understand that there are whole swaths of games that won't have to deal with these problems, but transforms are typically easier to solve for in those cases. And since there's a pretty solid standard for rig coordinate spaces across film and games I'd like to see that standard get more direct support in Bevy.