Feature request: multi-axis camera rotation

Summary

With Duality moving to a render pipeline that just passes through a view matrix it would be relatively straight forward to support rotating the camera along more than just the Z axis. This would make it easier for games to perform isometric rendering and setup the ground work for full 3d support without having to do it all in one step.

Analysis

I tried a while ago to make an isomorphic game in duality with mixed results. I ended up spending a lot of time mucking with the depthOffset field to get the rending order right. I ended up writing an algorithm to adjust the depth offset based on how close an image was to the bottom of the screen. This was not to bad but gets more complicated as you also have two types of images, those that represent the ground as well as those that represent things standing vertically on top of the ground. I could never get the depth algorithm to work quite right for these vertical images (as they are always above the ground but need to adjust their depth to one another based on an arbitrary "middle" point that is usually not the middle of the image). It is not uncommon for one of these images (say a tall building) to be closer to the bottom of the screen relative to another vertical image yet supposed to be behind that image. I tried a variety of solutions, manually setting objects to a given depth layer, defining the "middle point" of a given image and using that to set the depthOffset, splitting the image into pieces and setting those pieces depth offset's separately. Needless to say it was a constant pain.

In my current project I want to do isomorphic rendering again but this time around need to also have the horizontal terrain be height based. Reliving the pain of doing isomorphic rendering is unsettling but achievable but now adding height to the terrain is past the point of what I really want to be doing in a 2d engine. I am a huge fan of Duality (it lets me code in fsharp) and I saw your post on the rendering pipeline updates so was inspired to see if I could add support for this through a plugin.

Demo (using only existing plugin points)

I was able to add support for this through just using built in plugin points but it required a good amount of duplication of duality's internals.

2019_8_16_DemoOf3dRenderingInDuality

To make it work I essentially had to overwrite the view matrix being used by the default vertex shader and the default visibility strategy.

To do this I first had to capture my desired camera rotation inputs, since the camera and transform components are sealed I created a custom component to hold onto the camera orientation fields. To get these fields into the shader render pipeline I created a custom render setup that overrided the OnRenderScene function adding some logic to find each camera in the scene, extract the orientation fields from my custom component, construct the new view matrix (using rotated camera direction vectors and a calculateLookAtMatrix algorithm and finally added my new view matrix as an extra shader parameter (that I called actualViewMatrix). Then I updated all of my materials to use a duplicated version of the default minimal shader except that it now uses my passed in view matrix instead of the default.

With that, rendering in the game view (for both orthographic and projection modes) started working correctly except now culling was broken.

To fix culling I needed to update the device.IsSphereInView function used by DefaultRendererVisibilityStrategy.QueryVisibleRenderers. Since all of the fields in DefaultRendererVisibilityStrategy where private, to do that I ended up duplicating it into a custom version along with the IsSphereInView function. That unfortunately was not as simple as copy pasting the IsSphereInView method since it uses the combined view and projection matrices. Since the projection matrix is inaccessible I had to calculate it myself (by duplicating UpdateProjectionMatrix). I then was able to get at my view matrix by pulling it from the shader parameters where it was set by my custom render setup function. Now that I had a version of the IsSphereInView function using my custom view matrix, the culling behavior improved but was still not quite right when the camera was rotated around the x/y axis and a sprites z position ended up behind the camera. To fix that I had to adjust the IsSphereInView functions bounds check. Here is the new code (in Fsharp)

  member __.IsSphereInView(projectionMatrix: ProjectionMatrix, fullMatrix: Matrix4, worldPosition: Vector3, radius: float32) =
    // Transform coordinate into clip space
    let worldPositionFull = Vector4(worldPosition, 1.0f)
    let mutable clipPosition =
      Vector4.Transform(worldPositionFull, fullMatrix)

    // If the perspective divide is near zero or negative, we know it's something behind the camera. Discard.
    if clipPosition.W < 0.000000001f
    then
      false
    else
      // Apply the perspective divide
      let inverseOfClipW = 1.0f / clipPosition.W
      clipPosition.X <- clipPosition.X * inverseOfClipW
      clipPosition.Y <- clipPosition.Y * inverseOfClipW
      clipPosition.Z <- clipPosition.Z * inverseOfClipW
      let clipRadius =
        Vector3(
          radius * MathF.Abs(projectionMatrix.Value.Row0.X) * inverseOfClipW,
          radius * MathF.Abs(projectionMatrix.Value.Row1.Y) * inverseOfClipW,
          // this was added
          radius * MathF.Abs(projectionMatrix.Value.Row2.Z) * inverseOfClipW
        )

      // Check if the result would still be within valid device coordinates
      let result = 
        // the line below was changed from clipPosition.Z >= -1.0f
        clipPosition.Z >= -1.0f - clipRadius.Z
        // the line below was changed from -> && clipPosition.Z <= 1.0f
          && clipPosition.Z <= 1.0f + clipRadius.Z
          && clipPosition.X >= -1.0f - clipRadius.X
          && clipPosition.X <= 1.0f + clipRadius.X
          && clipPosition.Y >= -1.0f - clipRadius.Y
          && clipPosition.Y <= 1.0f + clipRadius.Y

      result

So to summarizes once I got past all of the access issues by duplicating code I only needed to make the following changes,

A way to specify the camera orientation
Update the view matrix to orient the camera in the specified camera orientation
Update the IsSphereInView function to do sphere bounds checking against the z axis as well

Demo Caveats

With my extension approach I could not get the scene editor's object movement UI to align with the objects renderer position. The CamView camera was especially bad since the axis where all misaligned, the scene view camera was a little better since the x/y axis aligned but the UI was still off centered. I believe that if camera orientation was built in then at least the CamView camera would be better since it would correctly update the view matrix to its own orientation (along the z axis looking down). I could not figure out how to hook into the editor's rendering pipe to do this from an extension point of view.

Implementation Approaches

In my summary above I described the three steps needed for basic support for this

A way to specify the camera orientation
Update the view matrix to orient the camera in the specified camera orientation
Update the IsSphereInView function to do sphere bounds checking against the z axis as well

2 and 3 are relatively straight forward, 1 on the other hand is more difficult. I see a couple of different approaches to specifying the camera orientation.

Update the Transform components angle to be a Vector3 representing the rotation around each axis. This is how other 3d engines do this but would mean that a lot more would need to be updated to make the x and y fields make sense for other component types. Personally, I don't think this is a good next step as it feels like it is almost equivalent to adding full 3d support. Initially it would mean updating existing components and editor tooling to respect the camera orientation, but also updating things like the 2D rigid body system to also be orientable. Would people also expect new built in resources and components for model rendering? Large undertaking that my guess is not a current priority.
Add an orientation Vector3 to the Camera. Simpler than 1 as only things impacted by the camera would need to be updated. I believe this mostly just includes the editor tooling. This solution is a little weird as the Camera today gets its zAxis orientation from its GameObj's Transform (Link the two together?). Would also mean that if you decide to move to solution 1 later then that would be a breaking change.

An alternative to adding support for specifying the camera orientation would be to instead add support for the RenderSetup to be able to adjust the view matrix used by a Camera (or some other plugin point). With this approach, would still need to update the IsSphereInView function to do the bounds check against the z-axis but I don't think we would need to completely fix the editor UI (I am fine with editing the view matrix putting some of the tools a little out-of-warranty). This approach would mean that in the demo above I would only have the camera orientation component and a custom render setup.

Curious to hear what you think and I am happy to put a pull request together. Just need to know what solution idea would be preferred.

First of all, great work on the on analysis, prototyping and issue description, great stuff 👍 It's good to see that you were able to get the basics up and running with existing infrastructure, and I think we can use what you learned to (a) incorporate some of it into the core and (b) extend core API to reduce pain points for similar extensions in the future.

So, 3D camera rotation: It's nothing I'd generally expect in a 2D game engine, but you make a great point that it could simplify isometric and similar 2D perspective rendering a lot, and while there might be some pieces missing beyond that, this is probably the most crucial one.

We should however be careful to not frame this feature in a way that suggests "full" 3D support, because, like you said, this is a very different and very unsupported field for Duality, and we should avoid any user expectations to leak over and cause trouble. With that in mind, I'd probably not call it "rotation" (suggesting full freedom), but "tilt" (suggesting a limited ability to view things from an angle) and implement this as a Camera property, like you suggested.

Add an orientation Vector3 to the Camera. Simpler than 1 as only things impacted by the camera would need to be updated. I believe this mostly just includes the editor tooling. This solution is a little weird as the Camera today gets its zAxis orientation from its GameObj's Transform (Link the two together?). Would also mean that if you decide to move to solution 1 later then that would be a breaking change.

I think we could make it a Vector2 for XY rotation, and still use the Z rotation from the regular Transform, so the camera still behaves as usual when attached to some potentially rotating object. It's not that weird when you think of this kind of rotation as a "camera tilt" that is configured and mostly left alone just like all other camera and rendering properties.

Edit: This would also give us some freedom to adjust behavior with the specific use case of 2D iso / other perspective in mind, since it would explicitly be a camera setting for tilted views, and not a transform rotation.

A way to specify the camera orientation

Update the view matrix to orient the camera in the specified camera orientation

Update the IsSphereInView function to do sphere bounds checking against the z axis as well

Would you be willing to draft a first core implementation of this in a Pull Request?

Unfortunately my response time is way up recently, and I'm currently focused on the NuGet / C# / .NET update, so the PR will probably just sit there for a while - but it could be a good starting point for later, and a reference implementation for others who could start to do early testing and get a feel for advantages and pitfalls.

Let me know if you need any more info on Duality internals, or someone to bounce ideas off of.

I could not figure out how to hook into the editor's rendering pipe to do this from an extension point of view.

Not sure if I got the context correctly, but you can select which RenderSetup a scene view uses for its own rendering in its toolbar, on the upper right. It does use an internal editor-only camera that is not part of any Scene though.

AdamsLair / duality