Point locations seem visually "shifted"

[makella]

I'm not exactly sure how to explain this, but looking through some of the examples on the dev center, I'm noticing a shift in points.

For example, see how the points in this map look like they are piled on top of each other?

vs. that same map in this gif from Locations where I don't see the piling: GIF: https://docs.google.com/presentation/d/1JfvroO1eejtpi8S1KRc-vZnHpx-yJptEKTFZHM8DTI8/edit#slide=id.g38262f344b_0_900

Similarly, the points in this gif from Locations time:

To now:

https://cartodb.github.io/carto-vl/examples/editor/index.html#eyJhIjoic3BlbmRfZGF0YSIsImIiOiIiLCJjIjoiY2FydG92bCIsImQiOiJodHRwczovL3t1c2VyfS5jYXJ0by5jb20iLCJlIjoid2lkdGg6IDVcbmNvbG9yOiByZ2IoMjA0LDAsMClcbnN0cm9rZUNvbG9yOiBibGFjayIsImYiOnsibG5nIjoyLjE4NTQ0OTUyNDEyNjczODcsImxhdCI6NDEuMzkzOTA5NDIxNjU3Mjd9LCJnIjoxMSwiaCI6IkRhcmtNYXR0ZXIiLCJpIjoiZGF0YXNldCJ9

I know that this dataset is dense with points, but it is a significant visual difference from before and I'm not exactly sure how to explain it.

Did the default resolution change?

cc @Jesus89 @davidmanzanares

[rochoa]

I noticed the same while working with the master branch yesterday.

[makella]

ohhh good! glad that you noticed it too :)

[Jesus89]

Yes, I noticed that too. I think there was a change in the ordering algorithm that generates this weird order from left to right. I noticed also the tiles limits in the points:

[makella]

for some reason when i see the points like this i get the creeps!! it freaks me out like bugs or something haha...

[davidmanzanares]

Weird, we should try to isolate the problem to CARTO VL (frontend) and (in that case) to a particular commit or PR.

[davidmanzanares]

I made a quick test and I discovered important rendering differences between master and v0.5.0. The main cause is that we are requesting more tiles (at one more zoom level), but even by forcing the same tiles the result is not the same (although it's more similar).

I used this example, which I think it's a good one for this: file:///home/dmanzanares/github/renderer-prototype/examples/editor/index.html#eyJhIjoic3BlbmRfZGF0YSIsImIiOiIiLCJjIjoiY2FydG92bCIsImQiOiJodHRwczovL3t1c2VyfS5jYXJ0by5jb20iLCJlIjoid2lkdGg6IHNxcnQoY2x1c3RlclN1bSgkYW1vdW50KS81MDAwMCkqMjAqKHpvb20oKS80MDAwKzAuMDEpKjMuNVxuY29sb3I6IHJhbXAoY2x1c3Rlck1vZGUoJGNhdGVnb3J5KSwgUFJJU00pXG5zdHJva2VDb2xvcjogICAgICAgcmdiYSgwLDAsMCwwLjcpXG5zdHJva2VXaWR0aDogICAgICAyKnpvb20oKS81MDAwMCIsImYiOnsibG5nIjoyLjE3LCJsYXQiOjQxLjM4fSwiZyI6MTMsImgiOiJEYXJrTWF0dGVyIiwiaSI6ImRhdGFzZXQifQ==

This will certainly require more effort, and the number of tiles we ask should be reviewed.

[makella]

but even by forcing the same tiles the result is not the same (although it's more similar)

as @Jesus89 commented has the draw order of features changed?

I think there was a change in the ordering algorithm that generates this weird order from left to right.

[davidmanzanares]

Within CARTO VL we don't change feature order unless you use order:, in @Jesus89 image the issue shown is caused by tiling. Tiling could be more apparent due to the change I mentioned of downloading tiles of the next level, but that problem should be in every CARTO VL Maps API, regardless of the version.

Regarding the left to right issue, as I said, the ordering within a tile is (unless overridden) determined by the backend. I don't know of any changes there, PG10 adoption could have changed that for sure, but I lack knowledge here, maybe @Algunenano knows more.

The tiling problem (the border of the tile issue, not the left to right issue) could be mitigated/fixed by the almost just one tile optimization.

The left to right issue could be mitigated if we randomized the order with something like order: random (unimplemented), although I think that looking at the source first would be the best approach.

[makella]

ok! let me know if there is anything i can do to help to get this figured out. for me, this is one of those big ones cause maps are looking really "weird".

[Algunenano]

Regarding the left to right issue, as I said, the ordering within a tile is (unless overridden) determined by the backend. I don't know of any changes there, PG10 adoption could have changed that for sure, but I lack knowledge here, maybe @Algunenano knows more.

The order of the features in the MVTs isn't guaranteed by the backend, even if you add ORDER BY to the sql. That ordering isn't taken into account when the final encoding is done, and it would be highly complex to add that feature, specially in St_AsMVT.

If you are not using order by, some of the things that could change this order are different query plans (affected by DB release, DB parameters, table metadata, sharding...) or a migration (since it can change how they are stored and retrieved from the disks). Even if you use order by there is other circumstances that can, and will, influence in the final order, such as parallel plans in St_AsMVT, how the values are stored and retrieved from the intermediate hash table, etc. And this is without considering possible table alteration (data update).

In short, if you require consistent ordering across time, DB releases, tiler configuration, etc. you should never trust the order from the backend and do it in the client, in this case CartoVL. Since this might be expensive I'd recommend it to be optional (maybe when you declare order).

[rochoa]

Thanks for the detailed response, @Algunenano.

Now, it's clear what we need if we want to guarantee perfect ordering. However, we are not gonna work on it.

[makella]

is there a happy medium somewhere? the visual shift is super jarring and i can see it affecting the appearance of point maps overall.