[Suggestion] Countering the seemingly inaccurate coordinates

[SkyLicks]

Because overviewer coordinates are based off sea level, coordinates for areas of high elevation, areas of depression, and underground seem to be inaccurate.

Having a togglable display grid that can visually clarify this will be very helpful! If not, perhaps overview can be configured so that it also displays y=coordinate. Really, anything that visually helps clarify where the true x and y are will be helpful.

Example of what I am talking about: https://grandtheftmc.net/map/#/-10/64/-345/max/MineSantos/View%20Angle%20%5BNorth-West%5D (Toggle warps display on the right-hand side and then click the warp at the top of the mountain to see the real coordinates. Notice how off overviewer appears to be.)

[enaut]

If I'm not completely wrong this is because the leaflet shows images and can't know anything about the block structure and elevation. The coordinates are calculated for a mouse position over a 2D image. - so basically no way to do this without a complete rewrite of overviewer and a complete change in how overviewer works.

[SkyLicks]

Then perhaps this rewrite is worth it. What is the point of having a coordinate system if the coordinates aren't even accurate for anything other then sea level?

[enaut]

Well as overviewer is more in maintenance mode rather than active thriving development you are free to do the rewriting ;) but bear in mind that - at least as it seems to me - this requires a significant amount of manpower and there are way more pressing things that could be fixed before that.

[CounterPillow]

One idea is to leverage the heightmaps Minecraft writes into the .mca files. As far as I can tell, we have heightmaps for each column of chunks. The heightmaps appear to be 288 bytes in size, and because chunks are 16*16, this would mean we have 9 bits of height data per surface-block.

We could calculate the median height per chunk column from this data, and then write it out into an acceptable format into the world directory so that fromLatLngToWorld can pick height out from e.g. a certain pixel representing a chunk height in a PNG heightmap or something along those lines.

[CounterPillow]

I wrote a small script to parse heightmap data that confirms my suspicions that this is indeed packed 9 bit values:

#!/usr/bin/env python3
import sys
import numpy

def packed_longarray_to_shorts(long_array, n):
    bits_per_value = (len(long_array) * 64) / n
    if bits_per_value < 4 or 12 < bits_per_value:
        raise Exception("u dum")
    b = numpy.frombuffer(long_array, dtype=numpy.uint8)
    # give room for work, later
    b = b.astype(numpy.uint16)
    if bits_per_value == 8:
        return b

    result = numpy.zeros((n,), dtype=numpy.uint16)
    if bits_per_value == 4:
        result[0::2] =  b & 0x0f
        result[1::2] = (b & 0xf0) >> 4
    elif bits_per_value == 5:
        result[0::8] =   b[0::5] & 0x1f
        result[1::8] = ((b[1::5] & 0x03) << 3) | ((b[0::5] & 0xe0) >> 5)
        result[2::8] =  (b[1::5] & 0x7c) >> 2
        result[3::8] = ((b[2::5] & 0x0f) << 1) | ((b[1::5] & 0x80) >> 7)
        result[4::8] = ((b[3::5] & 0x01) << 4) | ((b[2::5] & 0xf0) >> 4)
        result[5::8] =  (b[3::5] & 0x3e) >> 1
        result[6::8] = ((b[4::5] & 0x07) << 2) | ((b[3::5] & 0xc0) >> 6)
        result[7::8] =  (b[4::5] & 0xf8) >> 3
    elif bits_per_value == 6:
        result[0::4] =   b[0::3] & 0x3f
        result[1::4] = ((b[1::3] & 0x0f) << 2) | ((b[0::3] & 0xc0) >> 6)
        result[2::4] = ((b[2::3] & 0x03) << 4) | ((b[1::3] & 0xf0) >> 4)
        result[3::4] =  (b[2::3] & 0xfc) >> 2
    elif bits_per_value == 7:
        result[0::8] =   b[0::7] & 0x7f
        result[1::8] = ((b[1::7] & 0x3f) << 1) | ((b[0::7] & 0x80) >> 7)
        result[2::8] = ((b[2::7] & 0x1f) << 2) | ((b[1::7] & 0xc0) >> 6)
        result[3::8] = ((b[3::7] & 0x0f) << 3) | ((b[2::7] & 0xe0) >> 5)
        result[4::8] = ((b[4::7] & 0x07) << 4) | ((b[3::7] & 0xf0) >> 4)
        result[5::8] = ((b[5::7] & 0x03) << 5) | ((b[4::7] & 0xf8) >> 3)
        result[6::8] = ((b[6::7] & 0x01) << 6) | ((b[5::7] & 0xfc) >> 2)
        result[7::8] =  (b[6::7] & 0xfc) >> 1
    # bits_per_value == 8 is handled above
    elif bits_per_value == 9:
        result[0::8] = ((b[1::9] & 0x01) << 8) |   b[0::9]
        result[1::8] = ((b[2::9] & 0x03) << 7) | ((b[1::9] & 0xfe) >> 1)
        result[2::8] = ((b[3::9] & 0x07) << 6) | ((b[2::9] & 0xfc) >> 2)
        result[3::8] = ((b[4::9] & 0x0f) << 5) | ((b[3::9] & 0xf8) >> 3)
        result[4::8] = ((b[5::9] & 0x1f) << 4) | ((b[4::9] & 0xf0) >> 4)
        result[5::8] = ((b[6::9] & 0x3f) << 3) | ((b[5::9] & 0xe0) >> 5)
        result[6::8] = ((b[7::9] & 0x7f) << 2) | ((b[6::9] & 0xc0) >> 6)
        result[7::8] = ( b[8::9]         << 1) | ((b[7::9] & 0x80) >> 7)
    elif bits_per_value == 10:
        result[0::4] = ((b[1::5] & 0x03) << 8) |   b[0::5]
        result[1::4] = ((b[2::5] & 0x0f) << 6) | ((b[1::5] & 0xfc) >> 2)
        result[2::4] = ((b[3::5] & 0x3f) << 4) | ((b[2::5] & 0xf0) >> 4)
        result[3::4] = ( b[4::5]         << 2) | ((b[3::5] & 0xc0) >> 6)
    elif bits_per_value == 11:
        result[0::8] = ((b[ 1::11] & 0x07) << 8 ) |   b[ 0::11]
        result[1::8] = ((b[ 2::11] & 0x3f) << 5 ) | ((b[ 1::11] & 0xf8) >> 3 )
        result[2::8] = ((b[ 4::11] & 0x01) << 10) | ( b[ 3::11]         << 2 ) | ((b[ 2::11] & 0xc0) >> 6 )
        result[3::8] = ((b[ 5::11] & 0x0f) << 7 ) | ((b[ 4::11] & 0xfe) >> 1 )
        result[4::8] = ((b[ 6::11] & 0x7f) << 4 ) | ((b[ 5::11] & 0xf0) >> 4 )
        result[5::8] = ((b[ 8::11] & 0x03) << 9 ) | ( b[ 7::11]         << 1 ) | ((b[ 6::11] & 0x80) >> 7 )
        result[6::8] = ((b[ 9::11] & 0x1f) << 2 ) | ((b[ 8::11] & 0xfc) >> 2 )
        result[7::8] = ( b[10::11]         << 3 ) | ((b[ 9::11] & 0xe0) >> 5 )
    elif bits_per_value == 12:
        result[0::2] = ((b[1::3] & 0x0f) << 8) |   b[0::3]
        result[1::2] = ( b[2::3]         << 4) | ((b[1::3] & 0xf0) >> 4)

    return result

def main():
    if len(sys.argv) != 2:
        print("Usage: {} FILE".format(sys.argv[0]), file=sys.stderr)
        sys.exit(1)
    with open(sys.argv[1], "rb") as f:
        heightmap = numpy.fromfile(f, dtype=numpy.uint64)
    heights = packed_longarray_to_shorts(heightmap, 16*16)
    print("Median: {}".format(numpy.median(heights)))
    heights = heights.reshape((16, 16))
    numpy.savetxt(sys.stdout, heights, fmt="%3d")

if __name__ == '__main__':
    main()

Here's its output with a WORLD_SURFACE heightmap of a flat world:

./unpack_heightmap.py /tmp/height2.bin 
Median: 4.0
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4
  4   4   4   4   4   4   4   4   4   4   4   4   4   4   4   4

So my suggestion of using the median chunk column height to get a better, rough approximation for fromLatLngToWorld seems to be on good track. I'll do more investigating in the future.