search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.39k stars 3.97k forks source link

Source JDBC: Support for spatial data types #25910

Open cazitouni opened 1 year ago

cazitouni commented 1 year ago

Tell us about the problem you're trying to solve

I would like to do import and export of spatial datas between different databases but actually spatial types are automaticaly casted into strings. ( at least for Oracle and Postgis)

Describe the solution you’d like

I would like to do the exact same thing I do actually for non spatial data in my postgres/oracle databases but with the support of the geometry type.

Describe the alternative you’ve considered or used

No clear alternative for me at this point

Are you willing to submit a PR?

Maybe, but I will have to get into the project first.

mahic commented 1 year ago

@cazitouni +1 Not quite the same case as you, but I'm working with massive amounts of spatial data stored in Oracle in columns of type SDO_GEOMETRY which we want to export to Parquet, or optimally geoparquet as our team is doing analytics on top of spatial data with Spark and Sedona.

What I've currently done as a workaround is to make views in Oracle with SDO_UTIL.TO_WKTGEOMETRY(GEOM) that I connect Airbyte to in order to be able to export it to Parquet.

I guess one of the potential issues with Airbyte supporting this is that (at least for SDO_GEOMETRY ) you don't only have the actual data but might also have metadata such as SRID associated with the type/column.

andrewshrout commented 1 year ago

Hoping this gets done as well. +1

Also working with parquet and spatial data.