heavyai / heavydb

HeavyDB (formerly OmniSciDB)
https://heavy.ai
Apache License 2.0
2.95k stars 449 forks source link

display a large amount of geometry #835

Open shyoon87 opened 3 months ago

shyoon87 commented 3 months ago

It's very interesting to manage geometry using gpu.

Is there any way to display a large amount of geometry except to convert it to "tf_geo_rasterize" and into a Raster image?

Is there a way to convert it to VectorTile and display it on Mapbox?

cdessanti commented 3 months ago

It's very interesting to manage geometry using gpu.

Is there any way to display a large amount of geometry except to convert it to "tf_geo_rasterize" and into a Raster image?

Is there a way to convert it to VectorTile and display it on Mapbox?

Right now out software (not the os version) and draw millions of linestrings, polygons and multipolygons or points in less than a second.

If you have a ratser it's can be loaded and going to be converted into points that can be drawn quite easily. I loaded an Esa dataset containing 8 billions points being able to analyze data with the single grace hopper.

BTW I am not the beta person to reply this question so I am asking for the this question internally

mflaxman10 commented 3 months ago

There are multiple questions in here, so let’s separate out a bit. In general, we’ve got the fastest geo rendering architecture on the market, offering orders of magnitude higher performance than conventional. This works by avoiding raw data movement across the network, and we can render complex maps with literally billions of features in a few hundred milliseconds.

So this avoids a whole series of workarounds required in conventional GIS and BI, most notably vector tiles and separating visualization formats from analytic ones. Vector tiles work by simplifying out and indexing vector fragments by scale. None of that is needed in our system, since the source of truth is near instantly available. You can apply cartographic scale generalization if you want for visualization purposes but don’t need it for performance. Also no reason to rasterize for such purposes.

Independent of all of that, we do have raster data handling and rasterization as you’ve discovered. That can be used for analytics typically for things like weighted overlays. We continue to invest in improving this, at last release with HeavyConnect Raster which supports linking to and managing huge raster file data stores. At next release (8.1) we’ve added push down bounding box filtering for further efficiency when dealing with slower external storage sources like s3. Upcoming releases will include enhancements to color management and raster RBG (aka true color) rendering.

Lastly, if you have some data already in vector tile and lack access to the original full fidelity data, our SQL version of COPY FROM supports all of the built in GDAL drivers. I’ve lost track of which version of GDAL supports reading of vector tiles, so feel free to post again on that specifically if you need. We’ve mostly been recommending “flat geo buffer” as a high efficiency geo transfer format when needed. That works today in Immerse. We are tracking rapid advances in geoparquet, and expect further announcements there soon.