Closed datajoely closed 3 months ago
Yeah only some operations actually require pandas. So I could localize those imports and then tell the user to install pandas to use those functions. Obviously that introduces some friction though. So it is a tradeoff here between the gain of minimizing the required dependencies and lowering friction for users.
Yeah it's not something that you need to worry about today, but it would be great later down the line. Thanks again for the fantastic work I'm already 90% of the way done building a Kedro dataset for it sqlframe 💪
In case you're interested here is my Kedro implementation https://github.com/kedro-org/kedro-plugins/pull/694
Very cool thanks for sharing! 🙌
Super cool work - very exciting project between this and Ibis I'm so excited about the future of composable, flexible systems where syntax is decoupled from execution.
I noticed in your setup.py that pandas is a required dependency for every back-end. Thinking out loud - I get why you need this to match the Spark API 1:1.
It's no a massive footprint these days, I think the risk is more for older brownfield projects that have complicated existing dependencies. Part of the appeal for this project is that existing Spark codebases can migrate to say duckdb without any pain, this may crop up over time.
pip-compile
on pandas with no extras returns the following, the footprint doesn't look super risky, but conflicts may crop up: