Possible issue regarding custom shaders and rotation of i3dm/b3dm tilesets

[martin-schilling]

Hi, thanks for your work on Cesium. I'm not entirely sure whether this is a bug or if I'm simply missing something here.

After support for loading i3dm tileset with model experimental was added just recently, I've been trying to use it in order to apply a custom vertex shader to an i3dm tileset. The idea is to scale the height of the individual instances of the tileset based on their attributes which, as far as I can tell, is only really possible with the new model experimental through using a custom vertex shader.

However, I quickly ran into a problem. It looks to me like the tileset's model coordinates are based on a coordinate system that is rotated differently than the world's coordinate system. Because of that, scaling the y factor of the vertices' coordinates causes a shearing effect. Instead, what I expected was that this would simply scale the height of the individual instances of each tile.

As I was curious whether this is caused by some mistake in the tileset I generated myself or whether this is a general thing, I looked into different tilesets from the sandcastle. It seems like most tilesets are affected by this, however, I managed to find one where this works just as I was hoping it would. Here are two examples, one that doesn't work and one that works.

What is the difference between these two tilesets that causes the different behaviors? Is this how it's supposed to be? Am I missing some configuration option or is there another way to achieve this?

Thanks in advance for your help.

[ptrgags]

@martin-schilling thank you for bringing this to our attention. I've noticed the same problem myself when I was responding to https://github.com/CesiumGS/cesium/issues/9937#issuecomment-990253220 a couple months ago where I was trying to translate and scale in the +Z direction.

To explain what's happening, many existing models in our Sandcastles were tiled in an earth-centered, earth-fixed (ECEF) coordinate system, which is relative to the center of the earth with z going through the north pole (see This sandcastle). This works fine for basic rendering, but for custom shaders, ideally model space is east-north-up (ENU) space so +z is "up" in a local area of the globe.

Adjusting the data to be in ENU space instead is certainly one way to fix this. Though in the long run, the discussion question becomes should this always be done at tiling time (possibly), or should custom shaders allow for converting to ENU space at runtime, at least in some cases?

In the latter case, should certainly be considered carefully, model matrix handling in ModelExperimental already needs a bit of refactoring.

[ptrgags]

A user mentioned this confusing behavior on the forum: https://community.cesium.com/t/what-is-the-model-positiommc-y-direction-in-custom-shader/17531

[martin-schilling]

@ptrgags thank you for your explanation. This does make sense to me now. I previously had the EAST_NORTH_UP flag activated and without it the scaling works as expected (altough the individual instances are then of course facing north).

However what I was wondering now is how exactly does one specifiy the position of the instances within ENU space instead? By also specifying a transform matrix in the tileset.json file that is then used to convert these ENU coordinates back to the geocentric coordinates that I'm currently using to specify the position of the instances?

I was looking into the sample tilesets generated by the samples-generator and came across the InstancedWithTransform tileset which, as far as I can tell, does exactly this. Consequently, I tried loading it with CesiumJS to see whether the vertex shader would work on it as I was hoping it would. While doing so I came across another issue that might be related to this. Loading the InstancedWithTransform tileset without model experimental works perfectly fine. However, after enabling model experimental, the model is suddenly not visible anymore.

Here are two screenshots. (Sorry, I would have provided sandcastle links, but I don't think that this model is available)

enableModelExperimental = false	enableModelExperimental = true

[onsummer]

Hi, I ask the similar quetion on the forum. What is the model positionMC.y direction in custom shader

The mostly entities we cared about are based on ENU space, and sometimes we want to modify them but lots of 3DTileset was stored by ECEF space (world coordinate system that +z is parallel to earth's axis).

So I wonder if it is possible to convert the dataset under ECEF space to ENU space, by this:

const toEcefMatrix = Cesium.Transforms.eastNorthUpToFixedFrame(interestPoint)
const inverse = Cesium.Matrix4.inverse(toEcefMatrix)

then use it in vertex shader:

new Cesium.CustomShader({
  uniforms: {
    u_toEcefMatrix: {
      value: toEcefMatrix,
      type: Cesium.UniformType.MAT4
    },
    u_inverse: {
      value: inverse,
      type: Cesium.UniformType.MAT4
    }
  },
  vertexShaderText: `
  void vertexMain(VertexInput vsInput, inout czm_modelVertexOutput vsOutput) {
    vec4 positionENU = u_inverse * vsInput.attributes.positionWC;
    // modify positionENU as output here, for example, scale z by 0.1
    vec4 modifiedPosition = positionENU * vec4(1.0, 1.0, 0.1, 1.0);
    vsOutput.positionMC = u_toEcefMatrix * modifiedPosition;
  }
  `,
  // ...
})

[ptrgags]

@martin-schilling

By also specifying a transform matrix in the tileset.json file that is then used to convert these ENU coordinates back to the geocentric coordinates that I'm currently using to specify the position of the instances?

Indeed, in the b3dm/i3dm/glTF, use a coordinate system where x is "east" and z is "up", and then apply either a transform in your tileset.json (or set tileset.modelMatrix at runtime) with the ENU matrix.

As far as the i3dm not appearing... I'll have to look into that in more detail. perhaps a matrix is being applied twice somewhere.

@onsummer

While your code snippet may work in some cases, in general you will likely notice jitter in the result due to precision issues in positionWC. The short explanation is you need 64-bit floats to represent global Cartesian values correctly, the long explanation is this article.

Ideally you would create a matrix from view space -> model space on the CPU and upload that to the shader uniform, bypassing world coordinates in the shader. Though this would require digging into some private API classes to get the model/view matrices. I suppose CustomShader (or at least the private API code that processes it) could do something like this, though not sure what the public API would look like. Would the user pass in the matrix to unapply? is this general enough to be worth adding? :thinking:

Are you producing your own data or are you using existing datasets? Right now the recommended approach is to produce data in ENU space as described above.