Open matthewmturner opened 2 years ago
@houqp i imagine youll have a view on this.
I think you should be able to add deltalake support to datafusion-tui by leveraging the existing table provider directly without touching datafusion core, see: https://github.com/delta-io/delta-rs/blob/main/rust/src/delta_datafusion.rs.
There is no need for adding new filetype or file format because deltalake only uses parquet, which we already support in datafusion.
Interesting, ok. I thought there was some more magic going on under the hood (but I hadn't really had chance to look into it) but maybe that only comes into play with some of delta lakes more advanced features like time travel which I don't think is doable without sql extensions.
I'll try it out and get back to you. Thanks!
I think @matthewmturner is on to something. The SQL in this issue is straightforward and makes sense from an intuitive user perspective. Why it doesn't work seems like a limitation of DataFusion:
Our use-case is running a Ballista server, with delta-rs
compiled in, with the intention of allowing users to register tables in locations we can't know at compile time. Unfortunately, I think the way FileFormat
s work currently doesn't make this possible?
Is your feature request related to a problem or challenge? Please describe what you are trying to do. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and why for this feature, in addition to the what)
I would like to be able to register a delta lake deltatable as a table from SQL as part of working on datafusion-tui. For example:
From what ive seen this would require adding a
FileType
andFileFormat
fordeltatable
underdeltalake
feature, similar to how there isavro
feature.While I understand a delta table isnt exactly a file type / format - i think for the purposes of what were doing with those it meets the definition. Ive played with querying delta tables before and they use
register_table
as opposed toregister_listing_table
. So i think we would just need to match based onFileType
and then for delta table useregister_table
instead.Describe the solution you'd like A clear and concise description of what you want to happen. Enable deltatable
FileFormat
andFileType
as features underdeltalake
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.