search

ibis-project / ibis

the portable Python dataframe library
https://ibis-project.org
Apache License 2.0
4.95k stars 580 forks source link

docs(blog/how to): Ibis + duckdb spatial + geoparquet + lonboard #9598

Open ncclementi opened 1 month ago

ncclementi commented 1 month ago

Please describe the issue

After this support added to duckdb to read geoparquet (still on nightly as there is no new release), and this PR on Ibis https://github.com/ibis-project/ibis/pull/9549, we should now be able to plot on lonboard directly from an Ibis table.

Write a blog/how to about it. Not sure what's the best option, I don't have a strong opinion.

Code of Conduct

kylebarron commented 1 month ago

With #9549 it should work to pass lonboard.viz(ibis_table) or lonboard.ScatterplotLayer(table=ibis_table) directly. Let me know if it doesn't

ncclementi commented 1 month ago

@kylebarron Confirmed, I can plot directly from an Ibis table both with viz or a map with multiple Layers.

The only caveat is that if the data is not on WGS84, because duckdb doesn't have info of the CRS then it errors when trying to plot. Transforming to the right projection does the trick.

I'll try to put a doc, blog or something up at some point this week.

ncclementi commented 1 month ago

I wonder if we should wait for the next duckdb patch release or give instructions on how to install from nightly. Happy to give instructions and then update once released.

ncclementi commented 1 month ago

@kylebarron Idea: Would you be interested in co-writing a blog? I think it would be great to have more context around how lonboard works/integrates with Ibis. Also open to plot the any of the data sets in your examples, or something new that we think could be interesting to showcase.

I can get the initial PR going with code and wording, but I'm thinking in having sections like "What is lonboard", "How it works (brief) and "Why you should use it", that would definitely benefit from your review/authoring.

Thoughts?

kylebarron commented 1 month ago

Yeah that sounds great! I always struggle to find compelling datasets, so if you have one you like, I'm happy to use that one

ncclementi commented 1 month ago

I've been trying to play around with some more interesting data from overture maps, and while at it I landed in a few bugs hence the delay. xref: https://github.com/ibis-project/ibis/issues/9662 xref: https://github.com/duckdb/duckdb/issues/13120 (this one is a problem as a nightly version is needed to be able to get the geoparquet support)

ncclementi commented 4 weeks ago

The fix was in for the latest duckdb nightly, but there is a still a bug, a different one though. We are now blocked on https://github.com/duckdb/duckdb/issues/13285