make some datasource listing helper functions public?

[samuelcolvin]

Is your feature request related to a problem or challenge?

I'm trying to implement a custom variant of ListingTable, and I'm running into an issue that the following three helper functions are currently not public:

https://github.com/apache/datafusion/blob/cf0cba72204e54c8eb05156d372dc5bab248e4ea/datafusion/core/src/datasource/listing/helpers.rs#L313

https://github.com/apache/datafusion/blob/cf0cba72204e54c8eb05156d372dc5bab248e4ea/datafusion/core/src/datasource/listing/helpers.rs#L125

https://github.com/apache/datafusion/blob/cf0cba72204e54c8eb05156d372dc5bab248e4ea/datafusion/core/src/datasource/listing/helpers.rs#L51

That alone prevents me from implementing my own ListingTable with an otherwise reasonably level of complexity.

Describe the solution you'd like

I'd like either:

• to make those three functions public
• or, to provide a way to customise ListingTable so I can:
  • apply extra pruning based on column equivalence, see #10399
  • and, use file lists and file statistics that are stored somewhere other than the object store

Describe alternatives you've considered

I'd rather not copy and paste the whole of /datafusion/core/src/datasource/listing/helpers.rs, but that seems to be the only way around this.

Additional context

No response

[alamb]

I think making the functions public would be fine.

In general I think it would be super valuable to pull the ListingTable code out of the core crate (into datafusion-listing-table for example), as that would help make sure the APIs that are needed to use it are all publically exposed / documented. LIstingTable as implemented today is non trivial to reuse

However I think it would take concerted effort to pull LIstingTable out

Here is a related discussion https://github.com/apache/datafusion/issues/8345 where others have a similar idea