Upgraded New York City b3dm tiles, but data not rendered using CesiumJS v1.113

honglzhu commented 9 months ago

When using CesiumJS v1.113 for New York City b3dm dataset, I ran into this issue:

RuntimeError: Fragment shader failed to compile. Compile log: ERROR: 0:236: '=' : dimension mismatch ERROR: 0:236: 'assign' : cannot convert from 'highp 2-component vector of float' to 'highp 3-component vector of float'

Then I used this tool and upgraded the dataset with this command: npx ts-node src/main.ts upgrade -i ..\\NewYork\\tileset.json -o ..\\upgradedNY --targetVersion 1.1

The RuntimeError is gone, glb tiles are fetched, but no 3d buildings rendered on the map.

Below is my code (if you need the dataset, please let me know): ` <!DOCTYPE html>

`

javagl commented 9 months ago

If the data set has a "reasonable" size and can be shared, then this would be good (preferably the input data set with the B3DMs, to see where the upgrade went wrong).

But... the error message that you received with the original data might indicate that the data is "invalid" in one way or another.

(The message looks familar, and I think that I saw this in a case where normal data was stored in a strange way (quantized into two floats, but without information about the quantization itself), but that would have to be verified...)

honglzhu commented 9 months ago

Hi @javagl , I did the following:

I followed the instructions of this link to build the tool: https://github.com/CesiumGS/3d-tiles-tools?tab=readme-ov-file#developer-setup
Then ran this command to get the upgrade the old NYC b3dm to v1.1: npx ts-node src/main.ts upgrade -i ..\NewYork\tileset.json -o ..\upgradedNY --targetVersion 1.1
Tried to display the data using the latest CesiumJS v1.113, but no luck on both the old and new datasets.

Here are my two datasets:

Thank you for looking into this!

javagl commented 9 months ago

There are several issues coming together here.

First of all, the context is https://community.cesium.com/t/3dtiles-error-when-upgrade-cesium-to-1-97/20477/11 : CesiumJS did some internal changes that caused certain ... very old ... features of glTF 1.0 (!) to no longer be supported. This includes a special form of VEC2/BYTE oct-encoded normals. This eventually causes that

RuntimeError: Fragment shader failed to compile. ...

error.

The upgrade function of the 3d-tiles-tools can upgrade B3DM files to GLB. This usually mainly consists of extracting the GLB data from the B3DM. But there's the caveat that the B3DMs in the given data set contain glTF 1.0 (!) data. This data, in turn, is upgraded to glTF 2.0 with gltf-pipeline. (Afterwards, the 3d-tiles-tools add the metadata from the batch table, but that's a different story).

Now... the glTF 2.0 data that is created by gltf-pipeline still contains the VEC2/BYTE oct-encoded normals that causes the CesiumJS rendering error.

This can be fixed, with a special processing step. And I created a short script for that, locally. But... there's another caveat: There's a small bug in the B3DM metadata conversion. This is fixed via https://github.com/CesiumGS/3d-tiles-tools/pull/96

So the upgrade path for now is a bit clumsy:

Run the upgrade command based on the state from https://github.com/CesiumGS/3d-tiles-tools/pull/96
Run the "fixing" script (shown below)

After that, the data can be loaded and shown in CesiumJS:

Cesium NewNewYork

We'll consider

adding the functionality that is done in this "fixing" script to the main upgrade functionality (that could make sense and be relatively easy)
maybe offer the upgraded data set as a download (no promises here - sharing data is always a licensing issue)

There are some aspects that still have to be decided

This is the script for "fixing" the upgraded tileset, to no longer contain the invalid normals. This is only quickly written down. No warranties. Just added here for completeness.

import { Document } from "@gltf-transform/core";
import { Accessor } from "@gltf-transform/core";
import { prune } from "@gltf-transform/functions";

import { TileContentProcessing } from "@3d-tiles-tools/tools";
import { TileContentProcessorsGltfTransform } from "@3d-tiles-tools/tools";

// "Ported" from CesiumJS octDecode.glsl
function octDecode(encoded: number[], range: number): number[] {
  let x = encoded[0];
  let y = encoded[1];
  if (x === 0.0 && y === 0.0) {
    return [0.0, 0.0, 1.0]; // Unit-length
  }

  x = (x / range) * 2.0 - 1.0;
  y = (y / range) * 2.0 - 1.0;
  const v = [x, y, 1.0 - Math.abs(x) - Math.abs(y)];
  if (v[2] < 0.0) {
    v[0] = (1.0 - Math.abs(v[0])) * (v[0] >= 0.0 ? 1.0 : -1.0);
    v[1] = (1.0 - Math.abs(v[1])) * (v[1] >= 0.0 ? 1.0 : -1.0);
  }
  const len = Math.sqrt(v[0] * v[0] + v[1] * v[1] + v[2] * v[2]);
  const invLen = 1.0 / len;
  v[0] *= invLen;
  v[1] *= invLen;
  v[2] *= invLen;
  return v;
}

function octDecodeAccessor(
  document: Document,
  encodedAccessor: Accessor,
  range: number
) {
  const decodedData: number[] = [];

  const count = encodedAccessor.getCount();
  for (let i = 0; i < count; i++) {
    const encoded = [0, 0];
    encodedAccessor.getElement(i, encoded);
    const decoded = octDecode(encoded, range);
    decodedData.push(...decoded);
  }

  const decodedAccessor = document.createAccessor();
  decodedAccessor.setType("VEC3");
  decodedAccessor.setArray(new Float32Array(decodedData));
  return decodedAccessor;
}

async function runConversion() {
  const tilesetSourceName =
    "D:/Data/NewYork-upgraded/tileset.json";
  const tilesetTargetName =
    "D:/Data/NewYork-upgraded-fixed/tileset.json";
  const overwrite = true;

  const transform = (document: Document) => {
    const root = document.getRoot();
    const meshes = root.listMeshes();
    for (const mesh of meshes) {
      const primitives = mesh.listPrimitives();
      for (const primitive of primitives) {
        const normalAccessor = primitive.getAttribute("NORMAL");
        if (normalAccessor) {
          const type = normalAccessor.getType();
          const componentType = normalAccessor.getComponentType();
          const GL_BYTE = 5120;
          const GL_SHORT = 5122;
          if (type === "VEC2" && componentType === GL_BYTE) {
            console.log("Decoding BYTE normals...");
            const decodedNormalsAccessor = octDecodeAccessor(
              document,
              normalAccessor,
              255.0
            );
            primitive.setAttribute("NORMAL", decodedNormalsAccessor);
          } else if (type === "VEC2" && componentType === GL_SHORT) {
            console.log("Decoding SHORT normals...");
            const decodedNormalsAccessor = octDecodeAccessor(
              document,
              normalAccessor,
              65535.0
            );
            primitive.setAttribute("NORMAL", decodedNormalsAccessor);
          }
        }
      }
    }
  };
  const tileContentProcessor = TileContentProcessorsGltfTransform.create(
    transform,
    prune()
  );
  await TileContentProcessing.process(
    tilesetSourceName,
    tilesetTargetName,
    overwrite,
    tileContentProcessor
  );
}

runConversion();

honglzhu commented 9 months ago

Hi @javagl , thank you so much for the quick fix!

I tried to run your code after putting it into folder 3d-tiles-tools/src/fixup.ts (I had edited the in/out file paths accordingly), but got below error:

My commandline was: cd C:\projects\3d-tiles-tools C:\projects\3d-tiles-tools>npx ts-node src/fixup.ts

I must have missed something since I am not familiar with NodeJS/TS. Could you please tell me how to run your code to fix the upgraded data? Or you put the fixed data somewhere so I can download it?

Thank you for your assistance!

javagl commented 9 months ago

The script is based on the branch behind https://github.com/CesiumGS/3d-tiles-tools/pull/96

(An aside: There has been a refactoring for the 3D Tiles Tools, to break it into smaller packages. This is not merged yet, but should be merged in the next few days, so the bugfix in this PR is already based on this refactored state)

In order to check out this branch and use it locally, you'll have to do go into that directory with cd C:\projects\3d-tiles-tools, and then do a git checkout origin/fix-metadata-migration-null-strings to have this branch checked out. (It will complain about a "detached HEAD" - just ignore that for now). Then you'll have to do another npm install to have everything installed for the new state, and then npm run build

EDIT: NOTE That you'll also have to re-run the upgrade based on this state. The "fixup" uses the upgraded state as the input. So first, you'll have to do something like npx ts-node ./packages/cli/src/main.ts upgrade -i ..\\NewYork\\tileset.json -o ..\\upgradedNY-new --targetVersion 1.1

I just put the fixup.ts (which is called _FixNewYork.ts in my case :D ) into the root directory of the project (i.e. cd C:\projects\3d-tiles-tools in your case).

It should then not be necessary to edit any config files or so. Running npx ts-node ./fixup.ts should do it. (It will print a bunch of log messages for the updated accessors)

(If it doesn't work (then I'd be curious about the error messages, and), I'll try to upload the resulting data, but will have to ask internally where/how I may do that)

honglzhu commented 9 months ago

Thank you @javagl for the quick response. I got errors when running: npm run build

Below are some screenshots

honglzhu commented 9 months ago

Not sure why the npm run build got the above errors. Thanks.

javagl commented 9 months ago

Hm. I tried out out locally while writing that answer. I'll try it again and look into it. In the meantime, a very wild and very quick guess: Could you delete the node_modules folder and package-lock.json file, and do a fresh npm install afterwards?

javagl commented 9 months ago

I just tried out the process that you went through, namely doing the npm install on the current main state, and then switching to the branch, doing npm install and npm run build, and received the same errors.

Deleting the node_modules and package-lock.json and doing a fresh npm install and npm run build fixed it for me, hopefully that will resolve it for you as well 🤞

honglzhu commented 9 months ago

Yes, indeed it worked!

javagl commented 9 months ago

Great. Now, another small disclaimer: This "fixup" was a first shot. I tried the tileset in the latest version of CesiumJS, and it appeared to work, but if there is any problem with it (that I didn't notice yet), just drop a note here.

honglzhu commented 9 months ago

Yes, I will.

I will try to display the new dataset with CesiumJS v1.106.1 first; then upgrade our code to comply with CesiumJS v1.113 and then display the dataset.

(Using the latest CesiumJS version, i.e., v1.113, breaks our code, because CesiumJS v1.107 removed the readyPromise.

Will update you (either all works or I get questions).

Thank you @javagl for your prompt response and your professionalism is very impressive!!

honglzhu commented 9 months ago

Hi @javagl , this is to confirm that the upgraded dataset works like a charm in our App (using CesiumJS v1.106.1):

In a separate standalone test, it also works nicely using CesiumJS v1.113.

Thank you again for your great help!!

javagl commented 9 months ago

Great. I assume that this can be closed then. Feel free to reopen (or open a new issue) if necessary.

honglzhu commented 9 months ago

Yes, the issue is resolved! Thank you!

CesiumGS / 3d-tiles-tools

Upgraded New York City b3dm tiles, but data not rendered using CesiumJS v1.113 #95