cannot get accurate position using meter offset in scatterplot

[CrokinoleMaster]

I used the below to plot using METER_OFFSET. Results are quite good until the points are very far from the Origin, where they start moving away from their positions plotted using LNGLAT. Any idea to get it more accurate with the lnglat positions?

class Layer extends ScatterplotLayer {
    constructor(props) {
        super(props)
    }

    calculateInstancePositions(attribute) {
        const { data, getPosition, projectionMode, positionOrigin } = this.props
        const { viewport } = this.context
        const { value } = attribute
        let i = 0
        for (const point of data) {
            let position = getPosition(point)
            if (projectionMode === COORDINATE_SYSTEM.METER_OFFSETS) {
                const lngLatDelta = [
                    position[0] - positionOrigin[0],
                    position[1] - positionOrigin[1]
                ]
                position = viewport.lngLatDeltaToMeters(lngLatDelta)
                if (
                    (lngLatDelta[0] < 0 && position[0] > 0) ||
                    (lngLatDelta[0] > 0 && position[0] < 0)
                ) {
                    position[0] *= -1
                }
                if (
                    (lngLatDelta[1] < 0 && position[1] < 0) ||
                    (lngLatDelta[1] > 0 && position[1] > 0)
                ) {
                    position[1] *= -1
                }
            }
            value[i++] = get(position, 0)
            value[i++] = get(position, 1)
            value[i++] = get(position, 2) || 0
        }
    }
}

[gnavvy]

Hi @huaruiwu METER_OFFSET will only give you high precision for positions near the center. It is used when you have a center LATLNG with relative coordinates in meters for the rest of the points. If all your coordinates are in LNGLAT, you can try the LATLNG mode and enable fp64. (if you don't have a lot of points, performance should not be an issue with fp64)

[howtimeflies0]

This is expected behavior of the METER_OFFSET mode, as the mercator projection is non-linear. Just curious, how large an area does your data cover? a few thousand kilometers?

[CrokinoleMaster]

Thanks for the quick response! It's an area of 333,000km2, and over 500,000 points so I cannot use fp64.

[gnavvy]

333,000km2

~600km * 600km, this is larger than a regular city range.

may I ask if you need to visualize every single point at a low zoom level? would you consider doing some pre-aggregation / sampling when viewing a larger geo-range, and filtering points out of bound when zooming in to get a closer view?

[howtimeflies0]

@huaruiwu In that case, projectionPixelsPerUnit needs to be an attribute. Depending on the distance between the center point, each vertex would need to be projected using a different projectionPixelsPerUnit

Currently, projectionPixelsPerUnit is calculated around the center point and the same for all points.

[CrokinoleMaster]

@gnavvy I would like to visualize each point at a radius of 1 at low zoom levels. @shaojingli That makes sense. I will try that and see what happens!

[ibgreen]

This is a case (huge amount of lng lat points over a wide area) where pre-projecting the points (either in JavaScript or using transform feedback) would make a lot of sense. We don't have support for that mode yet but it would be an interesting addition.

[ibgreen]

By pre-projecting I mean pre-resolving the flat non-linear mercator projection.

[CrokinoleMaster]

Tried to just add projectionPixelsPerUnit as an attribute but it's difficult to get it working because it also depends on the zoom.

[Pessimistress]

You can also try dividing the area into horizontal strips and render each slice on a scatterplotlayer with its own projection center. The error should be smaller that way.

[Pessimistress]

Just looked at the uniform calculation. We are using the viewport center instead of projection center to calculate the scales. This should be a bug? @ibgreen @shaojingli

[howtimeflies0]

It's always been like that... this has been mentioned by @heshan0131 before the launch of v4. I agree that the center for pixelperunit calculation should be the positionOrigin instead of map center. But again what is we don't have positionOrigin (in lat/lon mode)

[ibgreen]

Just looked at the uniform calculation. We are using the viewport center instead of projection center to calculate the scales. This should be a bug?

There are (at least) two aspects to this:

1. (regardless of projection mode) there is the general idea that we should be able to work in "meters" in shaders - the ability to specify relative sizes of the geometry in meters.
   • To do that we need to provide the shaders with a linear meter scale. The "best" linear approximation of a meter we can give at any given time is the linear distance scale of the mercator viewport is the scale at the viewport center.
   • Changing this would mean the size of a meter would be dependent on whether positionOrigin is supplied or not, and would be increasingly wrong as the user panned away from positionOrigin.
2. Then there is METER_OFFSETS mode. It is suppose to be used only for small scales where the meter approximation stays correct.
   • I agree that this mode should benefit from a fixing the meter scale, although it has never been a visible issue in small scale visualizations we do.

Preliminary conclusion, it is either a tradeoff, or we need to pass two meter scales to the shaders.

It is all really complicated. I think it would be really valuable to do a full spec of the project shader module before starting to patch it, or we risk getting into a game of "whack-a-mole" where fixing one bug causes another to pop up on the side.

[howtimeflies0]

Changing this would mean the size of a meter would be dependent on whether positionOrigin is supplied or not, and would be increasingly wrong as the user panned away from positionOrigin.

@ibgreen, calculating the size of a meter dependent on positionOrigin sounds like a reasonable approximation. But even so, how to get this value for data provided in lat/lon mode is the problem, we won't have positionOrigin in that mode. I believe it's not a good idea to have lat/lon mode and meter offset mode behave differently with regard to our meters calculation

[ibgreen]

I believe it's not a good idea to have lat/lon mode and meter offset mode behave differently with regard to our meters calculation

Yes agreed, I share this concern, it would make the system even more complicated and surprising for the user.

Also, I believe that any changes we make here will not help the original issue. If you have data points that are far enough apart the linear distance scale approximation will not work and the question of which center we use is moot. It will be difficult to make a METER_OFFSET solution that works well for larger scales.

I have been playing with a LNGLAT_OFFSETS mode idea for some time that I think could work and provide big performance improvements for this case (it would use 64 bit calculations only for initial offset subtractions, but not for matrix multiplication). Just a matter of finding the time to implement it.

Then we could start recommending the METER_OFFSET mode only for small data sets that are actually specified in meters.

[CrokinoleMaster]

Great ideas. I think a LNGLAT_OFFSETS with good performance would be really useful.

[tehwalris]

I'm currently abusing the METER_OFFSETS mode in way which may be of interest here. I perform the non-linear part of the projection once in JS to effectively get coordinates on the zoom-0 Mapbox tile. I then use METER_OFFSETS as projection mode with one small change. In the draw call of the layer, I override projectionPixelsPerUnit with a value that only depends on zoom, not position of the viewport.

This is only a few lines as is, but maybe we could turn this into a LINEAR_OFFSETS mode as a feature, with the scale factor passed as a prop. In my case I need to project the points myself anyway, so having access to a direct mode like this would be useful. Maybe I'm missing something and this is also achievable with IDENTITY and web-mercator-viewport.

The modified non-linear projection code: (from web-mercator-viewport)

const PI = Math.PI
const PI_4 = PI / 4
const DEGREES_TO_RADIANS = PI / 180

export default function projectFlatUnscaled(lng, lat) {
  const lambda2 = lng * DEGREES_TO_RADIANS
  const phi2 = lat * DEGREES_TO_RADIANS
  const x = (lambda2 + PI) / (2 * PI)
  const y = (PI - Math.log(Math.tan(PI_4 + phi2 * 0.5))) / (2 * PI)
  return [x, y] // This is passed as position to deck.gl layer
}

Uniform override example: (based on point-cloud-layer)

  draw({ uniforms }) {
    const { radiusPixels } = this.props
    const p = uniforms.projectionScale * WORLD_SCALE
    const projectionPixelsPerUnit = [p, p, p]
    this.state.model.render(Object.assign({}, uniforms, {
      radiusPixels,
      projectionPixelsPerUnit,
    }))
  }

[CrokinoleMaster]

Interesting. I haven't had a chance to try this. I just ended up dividing the area into horizontal strips like @Pessimistress suggested

[ibgreen]

This is only a few lines as is, but maybe we could turn this into a LINEAR_OFFSETS mode as a feature,

@tehwalris Apologies for slow replies, looks like I missed your remark here at the end of this long issue. Your use is interesting and similar to something I have been thinking about:

• Add a COORDINATE_SYSTEM.WEB_MERCATOR_TILE_ZERO mode. It would allow the app to preproject lng/lats in JavaScript to numbers between 0-512. The shader already knows the scale so it could apply it without the app having to modify distance scales.

Implementing this mode is easy. I mainly want to make sure there is a use case for these modes before we add more complexity to the framework.

The main advantage I see would be somewhat faster rendering (no need for vertex shader to reproject lng/lats every frame).

But note that the precision advantages only comes when we use offsets instead of absolute coordinates, so this mode would need to support fp64.

Maybe we would also add a COORDINATE_SYSTEM.WEB_MERCATOR_TILE_ZERO_OFFSETS mode? But that makes things ever more complex to describe...

[CrokinoleMaster]

Only way I found that got around this whole problem was to bin the data. This was not enough though, because WebMercatorViewport uses latitude and longitude to calculate distance scales. So I had to also add a projectionOrigin prop to WebMercatorViewport.

WebMercatorViewport :

       const distanceScales = calculateDistanceScales({
            latitude: projectionOrigin ? projectionOrigin[1] : latitude,
            longitude: projectionOrigin ? projectionOrigin[0] : longitude,
            scale
        })

Bin data :

                const aoiCenter = turf.center(
                    wellknown(nextProps.selectedAOI.get('boundary_wkt'))
                )
                let bins = {}
                measuredGeometry.features.forEach(f => {
                    const lat = f.geometry.coordinates[1]
                    const key = lat.toFixed(2)
                    if (!bins[key]) {
                        bins[key] = [f]
                    } else {
                        bins[key].push(f)
                    }
                })
                const scatterplotLayers = Object.keys(bins).map(key => {
                    const bin = bins[key]
                    return {
                        id: 'measurements points ' + key,
                        data: bin,
                        opacity: 1,
                        radiusMinPixels: 1,
                        getPosition: d => {
                            return d.geometry.coordinates
                        },
                        getRadius: d => 0.5,
                        getColor: d => [OI_GREEN.r, OI_GREEN.g, OI_GREEN.b],
                        fp64: false,
                        projectionMode: COORDINATE_SYSTEM.METER_OFFSETS,
                        positionOrigin: [
                            aoiCenter.geometry.coordinates[0],
                            Number(key + '5')
                        ]
                    }
                })

[tehwalris]

@ibgreen

But note that the precision advantages only comes when we use offsets instead of absolute coordinates, so this mode would need to support fp64.

That depends. Preprojecting does reduce "jiggle" for me. It's somewhere between fp32 and fp64 shader quality. As far as I understand, the improvement is because sin/cos then use JS numbers - so better than fp32 shader sin/cos.

[howtimeflies0]

Need to evaluate the use case.

[ibgreen]

Collecting early thoughts on improved projection modes in this RFC