[Bug] Accuracy issues in ClipExtension

[felixpalmer]

Description

ClipExtension clips points which should remain visible due to numerical accuracy in shader code. The behavior is only present when using clipByInstance: true.

Flavors

• [ ] Script tag
• [ ] React
• [ ] Python/Jupyter notebook
• [ ] MapboxOverlay
• [ ] GoogleMapsOverlay
• [ ] CartoLayer
• [ ] ArcGIS

Expected Behavior

No response

Steps to Reproduce

https://jsfiddle.net/felixp/ngt2oz4e/

Environment

• Framework version: deck.gl@9.0.24
• Browser: Chrome/Firefox
• OS: MacOS 14.5

Logs

No response

[felixpalmer]

@Pessimistress Some ideas I tried:

• Passing the bounds as center and halfSize for better accuracy (didn't work)
• Using sub_fp64 to perform the operation (didn't work)