Quantized-mesh output support

[ahuarte47]

Provides some new features:

• New quantized-mesh format ouput (-f Mesh), optionally you could add VertexNormals with -N option.
• New metadata file output.
• New cesium-friendly option to automatically create missing root tiles (level 0).
• Consider nodata values when creating tiles.
• Catch GDAL 'Integer overflow' error when creating tiles of low levels.
• Refactoring of code to easily write new output formats.

Details: https://www.linkedin.com/pulse/fast-cesium-terrain-rendering-new-quantized-mesh-output-alvaro-huarte/

Update 2018/08/08: Available Cesium Terrain Builder Docker thanks to @BWibo

[bjpirt]

I've just built this branch and generated quantized mesh tiles for some elevation data I've been working with and can confirm all seems to work well. Would be good to get it merged in

Thanks to @ahuarte47 for a quick fix and @markerikson for help in getting it built

[ahuarte47]

Thanks @bjpirt , my apologies to everybody, I wasn't able to reopen the original PR #55 to keep saved all comments already written

[LHolst]

i successfully build a terrain as Meshformat with Normals plus Cesiumfriendly option. Its working great @ahuarte47 , to use the normal information in cesium i had to set the "extensions": ["octvertexnormals"] in the layer.json

maybe that could be done by ctb-tile when the layer.json is generated by setting the -l option

[ahuarte47]

Thanks @LHolst for the feedback! I will add the "extensions" entry.

[LHolst]

i have noticed some inconsistency between tiles normals at the edges.

it could be an issue in cesium...

[LHolst]

The extensions entry works. Thanks @ahuarte47

When using the cesiumFriendly option, the missing toplevel 0.terrain file is copied, but not reflected in the layer.json by the first entry in available. Therefore cesium only loads half of the globe, because the startX and endX of z=0 misses the copied file.

Lastly, it would be nice when using 'Normal' and 'CesiumFriendly' together, the normals of the copied missing 0.terrain were flipped.

see wrong normals on the left, i modified the layer.json by setting startX=0

[ahuarte47]

Thanks @LHolst for your feedback and advices! About flipping normals in that copied missing root tile, it needs some more of work

[LHolst]

i really appreciate your help, root tiles working now :+1:

[ahuarte47]

Hi @LHolst, finally I have changed how CTB creates the missing root tile, now it is created from a temporary empty elevation that spatially overlaps the missing tile. I hope it works

[LHolst]

I am getting an bounds error Exception when using -l layer only option

ctb-tile -C -N -o . -f Mesh -l  /data/dtm_converted.tif                                                                                                                                                                                                     
0...10...20...30...40...50...60...70...80...90...100 - done.
0...10...20...30...40...50...60...70...80...90...100 - done.
terminate called after throwing an instance of 'ctb::CTBException'
  what():  The value is greater than the maximum X value
Aborted (core dumped)

gdalinfo /data/dtm_converted.tif
Driver: GTiff/GeoTIFF
Files: /data/dtm_converted.tif
       /data/dtm_converted.tif.aux.xml
Size is 8750, 6000
Coordinate System is:
PROJCS["CH1903+ / LV95",
    GEOGCS["CH1903+",
        DATUM["CH1903+",
            SPHEROID["Bessel 1841",6377397.155,299.1528128,
                AUTHORITY["EPSG","7004"]],
            TOWGS84[674.374,15.056,405.346,0,0,0,0],
            AUTHORITY["EPSG","6150"]],
        PRIMEM["Greenwich",0,
            AUTHORITY["EPSG","8901"]],
        UNIT["degree",0.0174532925199433,
            AUTHORITY["EPSG","9122"]],
        AUTHORITY["EPSG","4150"]],
    PROJECTION["Hotine_Oblique_Mercator_Azimuth_Center"],
    PARAMETER["latitude_of_center",46.95240555555556],
    PARAMETER["longitude_of_center",7.439583333333333],
    PARAMETER["azimuth",90],
    PARAMETER["rectified_grid_angle",90],
    PARAMETER["scale_factor",1],
    PARAMETER["false_easting",2600000],
    PARAMETER["false_northing",1200000],
    UNIT["metre",1,
        AUTHORITY["EPSG","9001"]],
    AXIS["Easting",EAST],
    AXIS["Northing",NORTH],
    AUTHORITY["EPSG","2056"]]
Origin = (2685625.000000000000000,1287000.000000000000000)
Pixel Size = (0.500000000000000,-0.500000000000000)
Metadata:
  AREA_OR_POINT=Area
Image Structure Metadata:
  INTERLEAVE=BAND
Corner Coordinates:
Upper Left  ( 2685625.000, 1287000.000) (  8d34'51.96"E, 47d43'45.82"N)
Lower Left  ( 2685625.000, 1284000.000) (  8d34'49.86"E, 47d42' 8.69"N)
Upper Right ( 2690000.000, 1287000.000) (  8d38'21.90"E, 47d43'43.70"N)
Lower Right ( 2690000.000, 1284000.000) (  8d38'19.69"E, 47d42' 6.58"N)
Center      ( 2687812.500, 1285500.000) (  8d36'35.85"E, 47d42'56.21"N)
Band 1 Block=8750x1 Type=Float32, ColorInterp=Gray
  Min=399.590 Max=679.660 
  Minimum=399.590, Maximum=679.660, Mean=547.999, StdDev=62.302
  NoData Value=-9999
  Metadata:
    STATISTICS_MAXIMUM=679,65997314453
    STATISTICS_MEAN=547,99864003316
    STATISTICS_MINIMUM=399,58999633789
    STATISTICS_STDDEV=62,30161802239

without the layers option tiles get generated is there a way to generate the layer.json along with the tiles?

[LHolst]

when i use my old layer.json, the normals are improved.

[ahuarte47]

Hi @LHolst, I fixed a bug, I hope it works now

[LHolst]

https://github.com/ahuarte47/cesium-terrain-builder/commit/4bb78ba2e2c13cb0b4631b355cafb1aa94676f65 generates layer.json correctly, thanks @ahuarte47 :+1: :tada:

also https://github.com/geo-data/cesium-terrain-builder/issues/15#issuecomment-390916918 did the trick to run gdal 2.2

[geoportallux]

for anyone who wants to test this PR using docker:

FROM debian
RUN apt-get update && apt-get -y install cmake build-essential gdal-bin git libgdal-dev
RUN cd /root && git clone https://github.com/ahuarte47/cesium-terrain-builder.git && cd cesium-terrain-builder && git checkout master-quantized-mesh && mkdir build && cd build && cmake .. && make install . && ldconfig

works great for us. many thanks

[winsento]

Generate terrain from dem.tif EPSG:4326

ctb-tile  -f Mesh -s 24 -e 0 -C -c 4 -l --output-dir /terrain /dem.tif

But when set CesiumTerrainProvider in Cesium - result is empty. Example here: https://jsfiddle.net/winsent/7kp7y9k2/6/

[LHolst]

@winsento you need to run the command without -l to generate the tiles, and with to generate the layer.json

[JoJoeGoing]

Failed to obtain terrain tile...... I do not know how to fix it.

[ahuarte47]

It would be useful more information to try help you. What command parameters did you used? or what raster file did you used?

[JoJoeGoing]

@ahuarte47 The raster file I used is .tif。And the command is : ctb-tile -o ./terrain -f Mesh -C -N -c 4 ./out.tif .（I have get the layer.json by this command ctb-tile -o ./terrain -f Mesh -C -N -c 4 -l ./out.tif). It works well and I get the terrain files . the gdal version is 2.1.1

[ahuarte47]

Ok I see, the error comes from Cesium no from CTB. Do you have defined the Content-Encoding: gzip content negotiation header?

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

[JoJoeGoing]

@ahuarte47 I really appreciate your help. It works well now。

[mrgloom]

Seems like Integer overflow problem is not gone https://github.com/geo-data/cesium-terrain-builder/issues/28

[xtassin]

Hi! Thank you for the great project.

I successfully processed SRTM 250m data from http://gisweb.ciat.cgiar.org/TRMM/SRTM_Resampled_250m/

However, the file for the western part of the world (http://gisweb.ciat.cgiar.org/TRMM/SRTM_Resampled_250m/SRTM_W_250m_TIF.rar) triggers this error: Error: The minimum X value is greater than the maximum X value

I believe the range of coordinate (< -180.0) is causing the issue. Any suggestion on how to solve this?

[ahuarte47]

I would use gdal_translate with -projwin parameter to clip the raster to (-180 / -29.98)

[xtassin]

Thanks for the suggestion! I did it with gdalwarp eventually.

[BWibo]

Hey there, today, I released a Docker image based on this pull request including a brief guide on how to create quantized-mesh terrain datasets. I am happy for any kind of feedback.

github.com/tum-gis/cesium-terrain-builder-docker

[xtassin]

I noticed that generated tiles have some elevations errors at the boundaries (see screenshot). Would you have an idea why?

[ahuarte47]

Do those walls match the limits of your input raster? If you do not have data outside that borders you have to set the nodata flag of these raster cells in order to skip them when creating the tiles. You could also add some height to that cells to avoid a so big break with the real terrain.

[xtassin]

I assumed the walls match the raster tiles but I would have to confirm this. There is definitely data outside of the border as I am using a global SRTM source. These artifacts are visible on most tile boundaries.

On Thu, Sep 6, 2018 at 6:55 PM, Alvaro Huarte notifications@github.com wrote:

Do those walls match the limits of your input raster? If you do not have data outside that borders you have to set the nodata flag of these raster cells in order to skip them when creating the tiles. You could also add some height to that cells to avoid a so big break with the real terrain.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/geo-data/cesium-terrain-builder/pull/64#issuecomment-419166325, or mute the thread https://github.com/notifications/unsubscribe-auth/AASGfGFsF2QJQS5TYwfZm8df0tA9IifTks5uYVN5gaJpZM4T7BGN .

[84ace]

Does anyone have the issue with small area terrain tile generation causing the tiles to be somehow misplaced and overscaled? I have commented on issue #72 with the commit (https://github.com/geo-data/cesium-terrain-builder/commit/29e0ba8dfd626eab485a9addf150fc4ce3135cf3) that first caused the issue and a link to a tiff that can reproduce the problem in that commit or any commit after that. If anyone can shed light on it i'd be grateful. Thanks!

[SunBlack]

Adjustments to command line help is missing in this PR. Could you add it?

[ahuarte47]

Hi @SunBlack, I have updated the README file with all missing new commands. Thanks!

[taozywu]

Thanks for the suggestion! I did it with gdalwarp eventually.

@xtassin Hi, i have the same error. Error: The minimum X value is greater than the maximum X value

I want to know how you use this command gdalwarp, Looking forward to your reply!

[sevvie]

I've been talking about this in the MBTiles support PR (#61) but I think the problem I've reached actually has to do with what I was working with here.

I've used @ahuarte47's branch as a base and merged in @a4chet's adaptation of the MBTiles code, but I'm getting the same results whether I output the Mesh as a directory of .terrain tiles or an MBTiles archive.

The GDEM I'm using as a source is the NASA ASTER GDEM-10km-colorized.tif they offer at https://asterweb.jpl.nasa.gov/gdem.asp. Whether I use gdalwarp, gdal_translate, or anything else to prepare the DEM for processing, once I test the resulting file/files produced by ctb-tile, I get the following results:

As I zoom in, tiles at the north pole (or south, if I rotate the globe in that direction) begin to disappear, until I reach zoom level 5, the highest I generated for this test, at which point I'm just zooming in past the planet; all tiles disappear.

The ctb-tile command I'm using is ctb-tile -b terrain -o . -f Mesh -s 5 -e 0 -N -C GDEM-10km-colorized.tif.

(The -b terrain comes from a4chet's MBTiles work, wherein the fileformat is set to MBTiles and the file is given the name "terrain.mbtiles". The results are, however, the same when I do it just as terrain files.)

[sevvie]

Associated thread because debugging this is a nightmare. https://github.com/AnalyticalGraphicsInc/cesium/issues/8340

[sevvie]

The problem I was having was either the source data or just using full GDEMs, because I was able to avoid the problem for now by using a specific bounding box defined by a VRT. However, my terrain doesn't appear to render at all, and I don't have the foggiest idea where to begin debugging it.

It's trying to load the terrain but for whatever reason everything just remains flat. No errors (except for tiles that don't exist because of the limited tileset). I've changed datasets to a mosaic from the 30m ASTER GDEM data, but that doesn't seem to have made a difference either.

[BWibo]

Today I created an Alpine Linux based Docker image: tum-gis/cesium-terrain-builder-docker#5

• Alpine v3.10
• Gdal: v2.4.2

Get it here: docker pull tumgis/ctb-quantized-mesh:alpine

This has not been properly tested jet. I'm happy for any feedback.

[bibuzz]

@ahuarte47

I have used your Quantized-mesh builder to created tiles of the world tiff file. But the world tiff file contains bathymetery data and on viewing the tiles on Cesium using new Cesium.CesiumTerrainProvider , the land is below the oceans, like in a trench. ( the land 3d works well ) I used the command ctb-tile -s 1 -e 0 /backup/data/tilesets/terrain/world /backup/global_gebco.tif

[ ]

I asked in cesium forum and they mentioned that nothing can be done from the cesium end and it has to be changed at how the data is generated.

https://github.com/CesiumGS/cesium/issues/9642

I saw in your limitations and To-Do that you consider all tiles as land. Is that why the ocean is above? How is the positive altitude coming from?

[adam-ce]

are you still working on this? I was trying to get it to work, but i'm getting segfaults. I was able to track it down to out of bounds access. The following (new) assert is triggered:

index 9bab8ff..390cca9 100644
--- a/src/HeightFieldChunker.hpp
+++ b/src/HeightFieldChunker.hpp
@@ -1,6 +1,8 @@
 #ifndef HEIGHTFIELDCHUNKER_HPP
 #define HEIGHTFIELDCHUNKER_HPP

+#include <cassert>
+
 /*******************************************************************************
  * Copyright 2018 GeoData <geodata@soton.ac.uk>
  *
@@ -226,6 +228,7 @@ public:

   /// Return the array-index of specified coordinate, row order by default.
   virtual int indexOfGridCoordinate(int x, int y) const {
+    assert((y * m_size) + x < m_size * m_size);
     return (y * m_size) + x;
   }
   /// Return the height of specified coordinate.

m_size is 256, y is 256, but the array the code is reading and writing is only 256x256. so clearly, the addresses are out of bounds. I can imagine, that this is something simple (increasing the allocated memory and change the addressing?), but without understanding the algorithm and what's going on, i can't be confident in how to fix it. Any hints?

edit: I was able to fix it. the branch in my profile appears to work for me. In addition, it contains some code modernisation, and I ran it through address sanitizer. i'm still having trouble with rendering as tiles disappear when using a perspective camera (but not when using an orthographic one). i'm looking into it..

edit2: fixed the disappearing tiles. the quantised mesh branch was definitely broken here, but probably previous versions of cesium didn't care. fixed in my branch as well. still needs a bit of cleaning though..

[v8gx]

i have noticed some inconsistency between tiles normals at the edges.

it could be an issue in cesium...

Hi! Was the normal seam issue ever solved? I am processing terrain using the tum-gis docker image and getting seams like this:

[v8gx]

I have looked into the normal issue a bit and at level 0 already, the normals between the 2 "half-globes" already fail:

[BWibo]

@ahuarte47 @ShawayL I just reflected your changes in the https://github.com/tum-gis/cesium-terrain-builder-docker Docker image.

[SHIHUAMarryMe]

wow! great!!!!!!!!!!!

[gfrontera]

I have looked into the normal issue a bit and at level 0 already, the normals between the 2 "half-globes" already fail:

I believe that the problems with normals at the perimeter of tiles are intrinsic to how the tiles are built. The normal for each vertex is computed as a weighted average of the normals of all the triangles containing such vertex. That can be computed for all vertices in the tile except for those vertices in the perimeter, as you can only access some of the triangles that contain them.

One solution would involve generating the tiles without the per-vertex normal extension, and then on a second stage (when the vertices for all tiles have been computed), compute and append the normals for each tile. Does this sound right, @ahuarte47?

[ahuarte47]

One solution would involve generating the tiles without the per-vertex normal extension, and then on a second stage (when the vertices for all tiles have been computed), compute and append the normals for each tile.

Maybe the issue is related with what neighbour vertices are used to calculate the normals. I think that row of wrong normals can be located in 0 longitude. Normals of triangles located in the left and in the right side of that X/lon were calculated with a different subset of vertices. It would need work to validate this theory and a fix to resolve the issue.

Unfortunatelly I can not maintain this effort, any help is welcome.

[gfrontera]

Is this issue what is preventing this MR to be merged?

[mhaberler]

the fix by https://github.com/geo-data/cesium-terrain-builder/pull/64#issuecomment-986036186 is really important - without it --profile mercator crashes with the same symptoms described above

same goes for https://github.com/tum-gis/cesium-terrain-builder-docker

is this project still maintained?

[ahuarte47]

Hi @mhaberler I am so sorry, I am not maintaining this branch, maybe other fork is beeing improved.

[BWibo]

Hey @mhaberler, I am still maintaining tum-gis/cesium-terrain-builder-docker, but only the Docker part. I can't do anything about CTB itself. The image is built from https://github.com/ahuarte47/cesium-terrain-builder/tree/master-quantized-mesh. So any commits to that branch end up in the image.