Write a `particles` visualization demo using point-in-polygon

[thomcom]

We've been planning for a while to build a pip demo that uses index buffer objects and a viewport specified as a polygon for pip culling. This issue describes that project:

A web application is loaded that queries the api-server GPU backend to load a dataset onto GPU.

The api-server uses @rapidsai/cudf to load a large number of x,y point coordinates and metadata into a cudf.DataFrame.

The client app then queries for the points that contained in the current viewport and displays them.

It receives mouse events for scrolling and dragging that change the perspective on the displayed points, with a background image (or map) that translates and scales to match the GL geometry.

Whenever the perspective changes, the client requests an updated set of points from the original datasource, passing the current viewport coordinates to be used as a polygon.

api-server uses the viewport coordinates to cull the points with point-in-polygon before returning them over arrow.

https://github.com/rapidsai/node/issues/439#tasklist-block-78cf1f58-34c6-4e7b-a261-11ff5477a1f8

[thomcom]

The particles demo is displaying a NAD database of points on top of a Google Earth background image as texture. The app requests the data points from api-server and displays them.

• Loading the 67m NAD points CSV file, which has been stripped down to four columns: State, ZIP, Lon, and Lat, takes ~2 seconds in the api-server.
• Transferring all 67m points to the client, which includes striping the data into a WebGL appropriate buffer, takes 12s in the api-server. This suggests that only 93k points can be streamed per frame. It is a very interesting proposition to modify the client to receive a packet of points per frame and render them as they are available.
• The client needs work: The world view and camera views are not perfectly correlated yet. The last place I was before moving back to cuspatial was working on that correspondence via solving the Aw=s projective correspondence: the projection matrix from world coordinates to screen coordinates. In order to achieve this I am planning on building the matrix A when the app loads, using the size of the browser canvas as screen coordinates and a set of manually identified Lon/Lat values that correspond to the map I created. This approach has issues. The screen coordinates are fine, and reliable, and fixed, but the map is not very useful. When I return I want to:
  • Get an actual cartesian map of the United States using a real data source, producing it manually if necessary with GeoPandas. This will enable me to perfectly identify the lon/lat reference coordinates that will be used to produce A.
  • I also need to consider the aspect ratio of the screen versus the aspect ratio of the map - the map will have a fixed aspect ratio but the client aspect ratio will change, which affects which values should be used to compute A. The best way to deal with this is probably to convert the screen reference coordinates to the same aspect ratio as the map before computing A.

Once i am happy with the reference coordinate systems, the app will be modified to hit a new api-server endpoint whenever there is a re-render event: after a zoom event and after a drag event. It was necessary to fix the coordinate system reference points before doing this because the new screen viewport will be projected into world coordinates, then those world coordinates will be passed to the server to request the new set of viewport-culled display points.

Then I will add the viewport culling api endpoint to the server which will use a new Polygon produced from the lon/lat viewport coordinates, call point-in-polygon on the resident DataFrame, and return 93k or a larger number of points to the client for display.

[thomcom]

An important item I remembered during our weekly meeting is that the server presently has a terrible memory leak. If I use the full dataset of 67m points the server crashes after only 2-3 restarts.