MarcoGorelli
commented
4 months ago Hi @benrutter !
If you're looking to use this in soda-core, you might be interested in taking a look at Narwhals, which is intended exactly for this kind of use case
Feel free to join our Discord (link in README) to have a chat, and/or book some time at https://calendly.com/marcogorelli
Regarding this repo:
- The interchange protocol: this is more about having a standardised way of converting between libraries. I think it was a good project, but unfortunately it seems to have lost steam (there's been no activity on it for quite a while). Personally, I'm hopeful that we can revive it
- The dataframe api standard: I've been told it's been discontinued and that an exit-blog post will be published at some point
benrutter
Hey- I love this project! Feels like if dataframe libraries all implemented this, it would be an absolute dream to build libraries!
Just take something like soda-core which is a test library that supports lots of dataframe types. It has a bunch of code for every ttpe, would be amazing just to have one.
I wanted to reach out and ask a question, this isn't a real issue or anything so feel free to ignore!
Did you (or I guess would you) consider something along the lines of fsspec? If you haven't come across it before, fsspec implements a standard glue layer across lots of file system types (s3, dropbox, sftp etc) so that they can all be accessed the same way.
Something like "df-spec" would be a standard api for accessing multille dataframe libraries, with an implementation for each. Obviously not as ideal as them all having the same api, but probably more feasible to make happen (I can't see polars or spark adopting a new api soon).
Don't worry, I'm not some crazy person asking you to build this! I'm just curious if you considered it as an option, or what you think of it as an idea?