Integrating geospatial data like geopandas does

[roger120981]

In python there is the geopandas library that allows working with geospatial data within a dataframe (geodataframe) and series (geoseries), there is some possibility of integrating them in this project like in that library?

[cigrainger]

Oh very interesting! This is far outside my wheelhouse but absolutely seems like a good idea. My understanding is that it might be worthwhile to implement the Simple Features Standard. https://en.wikipedia.org/wiki/Simple_Features.

This is where the reliance on a Rust library wholesale becomes quite hairy. I think it would need to be approached way upstream with Polars, with respect to maybe the GeoRust ecosystem.

I'll give this some thought.

[roger120981]

I have been investigating the georust ecosystem a bit and it seems to me a good option, to carry geopandas you need management of projections, reading of different raster and vector formats and conversions between them, management of geometries and all those functionalities exist within their libraries, It would only be necessary to integrate them into the Polars or maybe create a packages in elixir as a first step.

[philss]

I know this is not related to the feature request, but I want to add this just as reference for people that want to work with Geospatial data in the Elixir ecosystem today: if you are using Livebook, you can work with GeoJSON and TopoJSON (a subset of GeoJSON) through VegaLite.

The only problem is that you may need to convert the data from other formats to these formats.

I made a small study using those tools here: https://github.com/philss/brazil-in-notebooks/pull/1 :point_right: Run in Livebook

[roger120981]

@philss

Excellent work!,

I am trying to integrate some functionalities of the georust ecosystem into an elixir NIF library in order to be used with spatial data, especially related to the gdal library that would allow working with a great variety of geographic data, both raster and vector.

Maybe you don't like the idea of using Rust very much, but there are more advances in this field than in elixir, in addition to the performance in some operations that demand calculation power

[philss]

@roger120981 thank you!

I am trying to integrate some functionalities of the georust ecosystem into an elixir NIF library in order to be used with spatial data, especially related to the gdal library that would allow working with a great variety of geographic data, both raster and vector.

Cool! Please share with us when you have something.

Maybe you don't like the idea of using Rust very much, but there are more advances in this field than in elixir, in addition to the performance in some operations that demand calculation power

I prefer to use Elixir when possible, but I agree that a NIF in Rust for this is a good idea. Specially if they have a complete set of tools for this field. Although we have some tools in Elixir like Geo that can be used for some tasks without problems.

Just be aware of some limitations of NIFs. Even if they are written in Rust, you can have some drawbacks.

[roger120981]

@philss Thanks for the observation.

[roger120981]

@philss @cigrainger After much research and cooperation from community members, steps are beginning to be taken to resolve this issue. Initially I opened this discussion, https://github.com/pola-rs/polars/issues/1830 , but finally Kyle Barron shed light on how to incorporate geospatial analysis into polars, here is a new library that promises if we support it and that we can integrate it into explorer in the future. It is based on georust ecosystem libraries and algorithms are perfectly integrated https://github.com/kylebarron/geopolars

[josevalim]

Just to update everyone, we are still several months away from this. There is work happening on bringing geo formats to Arrow, then these need to be surfaced in Polars. Only then we can surface them to Explorer.

[roger120981]

There is currently a project well advanced in this direction, https://github.com/geoarrow/geoarrow-rs, lead by @kylebarron, was used as core for this project,, https://developmentseed.org/lonboard, and in the future, it must be the core of this project, https://github.com/geopolars/geopolars

[roger120981]

This project serves as a complement to visualize millions of data in web environments https://github.com/geoarrow/deck.gl-layers

[roger120981]

@josevalim It would be advisable to look as it has bindings for Python and Javascript, and it would be a good starting point to integrate it as a Nif library in the future.