Translations

[Tekn1ck]

I have a quick question for you that perhaps you can assist with.

I am working in EPSG:27700 + ODN height and we have applied an offset of X=-535000 and y=-180000 (we have been doing this for a long time to get objects that are in Ordnance Survey space into something more friendly for 3d editing due to floating point errors - although I am open to change this concept)

I am a bit confused about how I should write this in the manifest so that the model just pops in exactly where it should do in Unity and what to write in the VTS map make local script.

I managed to get pretty close by translating the point in EPSG:27700 origin point -535000, -180000 to WGS coords like below in QGIS (units in meters)

"srs": "enu lat0=51.502820844 lon0=-0.056331546 h0=0.00",

Then I put this in the manifest

"srs": "enu lat0=51.502820844 lon0=-0.056331546 h0=0.00",

in Unity I put these same numbers into the make map local which brings it pretty close, but the height is off - and when I take a close look, if I correct the height in one location on the dataset and then check another location then the height differences vary. Perhaps more worrying than this, there also appears to be a slight horizontal shift perhaps 30 cm or so from the reference data.

So my questions are

1. • What is the correct way to do this?
2. • Is it possible to just change the srs in the tileset.conf file after rather than reprocessing the VEF to VTS again?
3. • During the VEF to VTS process are the meshes distorted to fit any ellipsoid or is it simply slicing the mesh and retexturing where required for optimised streaming?

[vaclavblazek]

1. You have to tell ve2vts the correct SRS your data are. You have your dataset in EPSG:27700 which should be equivalent to +proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +units=m +no_defs. So, it's a Transversal Mercator projection. ENU is a local cartesian system defined by a plane tangential to the ellipsoid at the given point. These systems can be pretty close but I'm not that sure. Please, use correct SRS. As you said, your data are localized around some arbitrary origin. You can use a global transformation matrix i.e. the toplevel trafo array in VEF's manifest.json. Just add your local origin to the forth column of a unit 4x4 matrix, i.e. something like "trafo": { 1, 0, 0, X, 0, 1, 0, Y, 0, 0, 1, Z, 0, 0, 0, 1 }. You need to use lastest vts-tools if you installed from a package repo (v. 1.90) for this feature.

2. A tileset uses SRS from reference frame, all data in the tileset adhere to these SRS. You cannot change it by any configuration. Data are not converted between different spatial reference systems when rendering.

3. As in 2), all data in tileset are in reference frame's SRS, meshes are in physicalSRS. In case of melown2015 reference frame, physicalSRS is WGS84 geocentric system. I.e. +proj=geocent +datum=WGS84 +units=m +no_defs. So yes, data are transformed and cut in predefined grid.

[Tekn1ck]

Hi again,

Thanks for this that makes a lot of sense, the thing that was stopping me was that I did not understand where to put the coordinates in the Trafo command. I have followed this out now and done a quick test and it calculates fine from VEF to VTS fine.

However, now when I load the data into Unity it is not really anywhere near where I would like to be unless I press the "default position" checkbox in the "vts map make local script" , but this rotates the model and shifts it too.

I have tried quite a few combinations namely

In the manifest to apply the two below offsets since I am unsure if I am moving the data or defining the center of the data in this offset. Trafo offset of -535000,-180000 Trafo offset of 535000,180000

In Unity- "vts map make local script" left the lat long at 0,0 tried the lat long as before the coordinate transformed from WGS84 (-0.056331546,51.502820844) tried the coordinates from WGS84 geocentric system (3978400.988459764,-3911.449518747)

Nothing gets close - if I manually try and adjust the lat long I am getting very strange numbers to bring it close and it is not correct anyway.

I wonder also whether there is an additional fundamental problem here if the data is being reprojected into the WGS84 geocentric system since it no longer matches to all the survey data and architectural drawings/3d that we input which are always in EPSG:27700. I could be wrong, but I think we need to make a new reference frame that means we do not reproject the data away from EPSG:27700. The problem is Unity - it is a cartesian system so when we put 3d models in they are alway xyz and geographic transformations are complex and not what architects use.

What are your thoughts on how to approach this? Ideally we would just work in EPSG:27700 from source through to final rendering in Unity

[Tekn1ck]

I have done a bit more research and it seems that strange number is not so strange after all - it is pretty close to the origin of the EPSG:27700 crs and now when I add this into the make map local it is close to my manual guess - but still not close enough.

The grey model is my reference model - the VTS model should sit precisely over the top.

I am a bit stumped now. What to do next!?

[ladislavhorky]

Hi, I may be totally out of context but lets see what we can do.

First: start from a correct VEF - no shifts or ad-hoc transformations should be then needed. In case there is something special about SRS handling in UNITY, I am cc ing @malytomas .

VEF: a) Are your meshes in really in EPSG: 27700 (then they will contain rather big numbers) or localized? Then you would need to use the global trafo (shift) mentioned by @vaclavblazek. That may be part of your issue.., if the model is somehow localized (not reflected in VEF manifest), you are feeding vef2vts wrong data that then get transformed, possibly turning a simple shift into something non-trivial that you are then trying to fix.

b) The 27700 is above different ellipsoid than WGS84, so your proj4 string in VEF should definitely contain +towgs84 param (7-rametric transformation between ellipsoids). Looking at https://epsg.io/27700, the correct string may be (contrary to the one mentioned by @vaclavblazek):

+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +towgs84=446.448,-125.157,542.06,0.15,0.247,0.842,-20.489 +units=m +no_defs

Good news is this might be responsible for the shifts you see, bad news it may not be the one towgs84 key you need - these are constructed so that the transformation has some reasonable error over some area. I case of whole SRS area of use, you might not get lower that few meters on average. Then you would need to use grids and possibly SW from Ordnanace Survey (see remarks on epsg.io link) or search for different towgs84 that would work better for you. We have similar troubles with Czech Krovak, but the (painstakingly found) towgs84 works well for our use-cases.

After fixing the VEF, everything else should just work.

Hope this helps (yes, SRSs are tough, especially if you work in local one)

[Tekn1ck]

Hi,

So A) They originate in EPSG:27700 so yes they have some pretty massive numbers in them, hence why we translate these to a local origin by shifting the output by -535000,-180000. I have done @vaclavblazek trafo - and that seems fine to me, I am pretty certain that I am feeding the correct trafo data in. "trafo": [ 1, 0, 0, -535000, 0, 1, 0, -180000, 0, 0, 1, 0, 0, 0, 0, 1 ],

b) Had a go with that proj4 string and it is close, like before but now has a height difference.

This is proving quite complex. If I were to try to create a reference frame that is EPSG:27700 so I am not reprojecting - how would I do this?

[vaclavblazek]

Custom RF? Now we are talkin'.

Before we begin: If you want to have your RF in EPSG:27700 you're gonna end up with a bit of "flat Earth" which is only locally usable. I guess it's not a problem here.

NB: we use proj4 strings exclusively because frontends (especially web) lack WKT or EPSG support.

1) Spatial references, reference frames etc. are in the /opt/vts/etc/registry directory. You may either tweam files there or make a copy and instruct VTS tools to use a different registry directory via --registry cmdline option. NB: unity does not need this since mapConfig.json serves as it's own registry.

2) You may get inspired in our prehistoric ppspace reference frame. Do not use vertical adjustmend, tho.

3) Add new SRS in srs.json, based on our utm33n. Just your name it accordingly, use proper proj4 string, and do not forget to say it's a projected system.

4) In referenceframes.json, make a copy of ppspace reference frame (choose some name]. Use your SRS as model.physicalSrs and model.navigationSrs.

5) Then update ranges properly. nodes[0].extents is a 2D area of your root tile -- rather make it square, ll stands for lower-left corner, ur stands for upper-right corner.

6) Box your world in division.extents. Use the same 2D bounds as in 2c, Z axis is minimum/maximum height value your world supports.

7) You may want to restart mapproxy with your updated registry and have a look at system spheroid in your new reference frame dir. It should show a flat square.

8) Convert your dataset to your RF. Try it in a web browser first.

[malytomas]

First, let me repeat this: always test your mapconfig in webbrowser (vts-browser-js) or vts-browser-desktop before unity!

Second, RF (reference frame) in mapconfig defines 3 separate srs:

Physical srs is what is used for meshes. VTS is designed to handle large numbers in physical srs.

Another srs in RF is navigation. It is used for camera controls in general.

Unity itself does not handle large numbers well, this is what the map make local is for. It will transform the space from navigation srs into a completely new local system - the one that suits unity - it aligns up vector and north.

Note that map make local is not srs conversion. It just rotates forward towards north and up vector out of the center of earth.

The third srs is public and it is not used in unity.

[Tekn1ck]

Happy new year - will be trying a new reference frame in the coming weeks!

[Tekn1ck]

Hi all,

So I finally had a chance to try this and I think I am pretty close, but there is still something funny going on. I have managed to make it all process but it will not load in the browser window.

I reprocessed a few tiles without the trafo on since that was not working at all. So now I am starting with a load of tiles in epsg27700 base.

In the SRS file I added this.

 "epsg27700": {
    "comment": "Projected, Ordnance Survey (epsg27700)",
    "srsDef": "+proj=tmerc +lat_0=49 +lon_0=-2 +k=0.9996012717 +x_0=400000 +y_0=-100000 +ellps=airy +towgs84=446.448,-125.157,542.06,0.15,0.247,0.842,-20.489 +units=m +no_defs",
    "type": "projected"
},

In the referenceframes I added this

{
    "version" : 1,
    "id": "epsg27700",
    "description": "Ordnance Survey Planar Reference Frame",
    "body": "Earth",

    "model": {
        "physicalSrs": "epsg27700",
        "navigationSrs": "epsg27700",
        "publicSrs": "geographic-wgs84-egm96"
    },

    "division": {
        "extents" : {
            "ll": [-90619.29,10097.13, -500], "ur": [612435.55,1234954.16,7000] },
        "heightRange": [-500,7000],
        "nodes": [
            {
                "id": { "lod": 0, "position": [0,0] },
                "srs": "epsg27700",
                "externalTexture": true,
                "extents": {
                    "ll": [-90619.29,10097.13], "ur": [612435.55,1234954.16] },
                "partitioning": "bisection"
            }
        ]
    },

Then I ran the CLI and used the new epsg27700 frame and the calc completed.

vef2vts --input /project/EPSG27700/City_4_VTS --output /project/epsg27700 --tilesetId epsg27700 --referenceFrame epsg27700 --overwrite --keepTmpset

The problem is that when I browse to it, the browser will not load the map when I first click on the output folder as it usually does and if I click on the index file it just hangs.

These are some of the last lines of the console output - it looked like it went through with no errors ;)

2021-02-26 10:41:08 I3 [467(tile:16-58063-56383)]: Generated tile #2018 of 2038 ( 99.02 % done): 16-58063-56383 (epsg27700, extents: 532266.958397,181146.500400:532277.686162,181165.190235) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58052-56374)]: Generated tile #2019 of 2038 ( 99.07 % done): 16-58052-56374 (epsg27700, extents: 532148.952976,181314.708916:532159.680742,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58061-56379)]: Generated tile #2020 of 2038 ( 99.12 % done): 16-58061-56379 (epsg27700, extents: 532245.502866,181221.259741:532256.230631,181239.949576) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58058-56380)]: Generated tile #2021 of 2038 ( 99.17 % done): 16-58058-56380 (epsg27700, extents: 532213.319569,181202.569906:532224.047335,181221.259741) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58059-56381)]: Generated tile #2022 of 2038 ( 99.21 % done): 16-58059-56381 (epsg27700, extents: 532224.047335,181183.880070:532234.775100,181202.569906) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58053-56374)]: Generated tile #2023 of 2038 ( 99.26 % done): 16-58053-56374 (epsg27700, extents: 532159.680742,181314.708916:532170.408507,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58049-56374)]: Generated tile #2024 of 2038 ( 99.31 % done): 16-58049-56374 (epsg27700, extents: 532116.769680,181314.708916:532127.497445,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58060-56383)]: Generated tile #2025 of 2038 ( 99.36 % done): 16-58060-56383 (epsg27700, extents: 532234.775100,181146.500400:532245.502866,181165.190235) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58054-56374)]: Generated tile #2026 of 2038 ( 99.41 % done): 16-58054-56374 (epsg27700, extents: 532170.408507,181314.708916:532181.136273,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58061-56383)]: Generated tile #2027 of 2038 ( 99.46 % done): 16-58061-56383 (epsg27700, extents: 532245.502866,181146.500400:532256.230631,181165.190235) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58050-56374)]: Generated tile #2028 of 2038 ( 99.51 % done): 16-58050-56374 (epsg27700, extents: 532127.497445,181314.708916:532138.225211,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58049-56375)]: Generated tile #2029 of 2038 ( 99.56 % done): 16-58049-56375 (epsg27700, extents: 532116.769680,181296.019081:532127.497445,181314.708916) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58054-56375)]: Generated tile #2030 of 2038 ( 99.61 % done): 16-58054-56375 (epsg27700, extents: 532170.408507,181296.019081:532181.136273,181314.708916) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58051-56375)]: Generated tile #2031 of 2038 ( 99.66 % done): 16-58051-56375 (epsg27700, extents: 532138.225211,181296.019081:532148.952976,181314.708916) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:08 I3 [467(tile:16-58055-56374)]: Generated tile #2032 of 2038 ( 99.71 % done): 16-58055-56374 (epsg27700, extents: 532181.136273,181314.708916:532191.864038,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:09 I3 [467(tile:16-58062-56383)]: Generated tile #2033 of 2038 ( 99.75 % done): 16-58062-56383 (epsg27700, extents: 532256.230631,181146.500400:532266.958397,181165.190235) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:09 I3 [467(tile:16-58051-56374)]: Generated tile #2034 of 2038 ( 99.80 % done): 16-58051-56374 (epsg27700, extents: 532138.225211,181314.708916:532148.952976,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:09 I3 [467(tile:16-58055-56375)]: Generated tile #2035 of 2038 ( 99.85 % done): 16-58055-56375 (epsg27700, extents: 532181.136273,181296.019081:532191.864038,181314.708916) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:09 I3 [467(tile:16-58048-56375)]: Generated tile #2036 of 2038 ( 99.90 % done): 16-58048-56375 (epsg27700, extents: 532106.041914,181296.019081:532116.769680,181314.708916) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:09 I3 [467(tile:16-58050-56375)]: Generated tile #2037 of 2038 ( 99.95 % done): 16-58050-56375 (epsg27700, extents: 532127.497445,181296.019081:532138.225211,181314.708916) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:09 I3 [467(tile:16-58048-56374)]: Generated tile #2038 of 2038 (100.00 % done): 16-58048-56374 (epsg27700, extents: 532106.041914,181314.708916:532116.769680,181333.398751) [mesh,atlas]. {encoder.cpp:process():304} 2021-02-26 10:41:09 I3 [467(Tile-136-101-1-1)]: VTS Encoder: generated. Finishing. {encoder.cpp:run():134} 2021-02-26 10:41:09 I3 [467(Tile-136-101-1-1)]: Generating DTM from heightmap (1786x2551 pixels). {heightmap.cpp:dtmize():273} 2021-02-26 10:41:09 I3 [467(Tile-136-101-1-1)]: Extracting navtiles. {ntgenerator.cpp:generate():358} 2021-02-26 10:41:09 I3 [467(Tile-136-101-1-1)]: Setting navtiles at lod 11. {ntgenerator.cpp:generate():365} 2021-02-26 10:41:09 I3 [467(Tile-136-101-1-1)]: Navtiles extracted. {ntgenerator.cpp:generate():390} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: Computing surrogates at lod 16 from heightmap 223x318. {ntgenerator.cpp:fillSurrogate():263} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: Computing surrogates at lod 15 from heightmap 111x159. {ntgenerator.cpp:fillSurrogate():263} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: Computing surrogates at lod 14 from heightmap 55x79. {ntgenerator.cpp:fillSurrogate():263} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: Computing surrogates at lod 13 from heightmap 27x39. {ntgenerator.cpp:fillSurrogate():263} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: Computing surrogates at lod 12 from heightmap 13x19. {ntgenerator.cpp:fillSurrogate():263} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: Computing surrogates at lod 11 from heightmap 6x9. {ntgenerator.cpp:fillSurrogate():263} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: VTS Encoder: Flushing. {encoder.cpp:run():141} 2021-02-26 10:41:10 I3 [467(Tile-136-101-1-1)]: VTS Encoder: done. {encoder.cpp:run():145} 2021-02-26 10:41:10 I4 [467(Tile-136-101-1-1)]: All done. {vef2vts.cpp:run():1238}

Have I made any mistakes?

[Tekn1ck]

Oh to add - I was not sure what to do with the z height bounds of the world, but I figured it would not do any harm to just leave the bounds the same as the ppspace example.

[Tekn1ck]

@vaclavblazek Sorry to bother you, but would you be able to take a quick look at my posts above and suggest where I may have made a mistake?

[vaclavblazek]

All seems legit. Could you please try debugger.html instead of index.html? It should show tiles (with their masks) in 2D Leaflet browser. Green corners mark which child is valid.

[Tekn1ck]

Can you give me an example of how to do this? I am currently just loading up the output folders and clicking on the name in the browser window when I navigate to my local server http://192.158.0.10:8070/store/output/ inside these output folders there are no html files.

[vaclavblazek]

Use http://192.158.0.10:8070/store/output/debugger.html . (FYI, .../ is the same page as .../index.html)

[Tekn1ck]

ok - so testing on one that works and the debugger shows up a nice array of squares with references in. but the epsg output juts gives me a 500 Internal Server Error

[vaclavblazek]

OK, could you pls look inside vtsd log (/var/log/vts/vtsd.log) why it returns internal server error?

[Tekn1ck]

ok - so when I try to open the epsg output in the browser the last few lines in the log are:

2021-03-04 10:54:42 I2 [1415(cache:4)]: Removing driver for /var/vts/store/output/epsg27700. {cache.cpp:operator()():561} 2021-03-04 10:59:56 I2 [1415(shttp:3)]: [conn:1146238] HTTP "GET /output/epsg27700/ HTTP/1.0". {http.cpp:prelogAndProcess():313} 2021-03-04 10:59:56 I2 [1415(cache:9)]: Opening driver for "/var/vts/store/output/epsg27700". {main.cpp:operator()():175} 2021-03-04 10:59:56 I3 [1415(shttp:7)]: [conn:1146238] HTTP "GET /output/epsg27700/ HTTP/1.0" 500 312 [ is not known reference frame. @{stringdict.hpp:get():165}]. {http.cpp:postLog():333}

So it looks maybe like the reference frame is not loading in the browser?

[vaclavblazek]

No, its all backend. Paste me /var/vts/store/output/epsg27700/tileset.json here, pls.

[Tekn1ck]

Attached are the two json files and the folder structure lists for your reference.

Thanks

list_directory.txt list_folder_structure.txt

tileset.zip

[vaclavblazek]

1. is your new reference frame in the /opt/vts/etc/registry/referenceframes.json file on the machine where vtsd runs?
2. did you restart vtsd since you updated it?

To restart vtsd, please use systemctl restart vts-backend-vtsd .

[Tekn1ck]

looks like that did the trick. Thanks - will try in unity tomorrow!

[vaclavblazek]

Great! All our tools read registry into memory on start. It's never checked for updates.

[Tekn1ck]

yeah - turned out it was my stupidity - we have two machines running the backend - one for processing and one for serving - I had not updated the reference frame and srs on the serving computer. silly me!

[Tekn1ck]

Hi Vaclav,

We made this work fine and are just replacing our first model with the new EPSG27700 one. I am now running into a different problem connected to Unity. Previously we had been forcing the model to load an not be culled using { "traverseModeSurfaces":"stable", "traverseModeGeodata":"none" , "cullingOffsetDistance":25000, "lodBlending":2} on the camera and it worked very well (we have some very big graphics cards). We do this because reloading the culled tiles every time the camera orbits around a point really slows the frame rate down.

Our problem is that now I have replaced the model it is just running incredibly slowly and not actually loading the model unless I drop the cullingoffsetdistance to something below 500 or so. If I switch back and forth between the two models I can see that there is a difference in the way they are reacting to the camerasettings. I wonder if it could be connected to the origin shift and the way the camera reads, but I honestly am stuck here.

In a browser it is fast on both models.

So I am wondering if there is something I can do to troubleshoot this?

[malytomas]

The cullingOffsetDistance was designed to avoid culling shadow casters. It extends the culling frustum around the camera in all directions. It is in meters (specifically, in physical space units). I have no idea how it could possibly work with 25 000, that really seems too much.

The green lines in the picture above represent the original culling frustum of the camera (which is in the red circle). You can see how the tiles around the camera are not culled. There is over 2000 tiles being rendered with the culling distance set to 500. There was under 500 tiles with the culling distance set to zero.

[Tekn1ck]

Hi Tomas,

So yes, I know how it works - the reason we have increased it so much is because if you are flying forward - fine no worries it works well - but if you are orbiting a point or panning from left to right in a large model the camera culling actually creates drag in the frame rate because it is constantly having to reload the tiles. In our previous model it works fine to incerase it this much- I am wondering if there could be anything connected to origins that could cause a big slow down?

The reason I ask is because it appears to be creating prefabs in the cameraobjects, but not displaying them.

[malytomas]

Can you check how many tiles/prefabs is instantiated for both models? Does it differ?

The vts camera component is showing statistics with number of tiles for each lod.