Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
2.34k stars 164 forks source link

Support for rolling joins and other special joins #2911

Open GitHunter0 opened 1 month ago

GitHunter0 commented 1 month ago

Hi folks, Daft is on track to become something really great.

Do you have plans to support special types of join, like rolling joins and others described in the links below? These kind of operations are very important in some fields.

Thank you.

https://medium.com/analytics-vidhya/r-data-table-joins-48f00b46ce29
https://duckdb.org/docs/guides/sql_features/asof_join.html
https://docs.pola.rs/api/python/stable/reference/dataframe/api/polars.DataFrame.join_asof.html

jaychia commented 1 month ago

The team really appreciates the kind words!

Definitely open to adding new types of joins. Some of the more stateful ones are interesting because they're quite difficult to implement in a distributed fashion.

Looking at the current suite of functionality in Daft, these joins seem to stand out to me as TODOs:

Is there anything we're missing? We'll try to get the team together next week to sit down and prioritize some of these things.

GitHunter0 commented 1 month ago

Hey @jaychia , I appreciate the feedback and the fact that Daft is committed to look into these issues.

Is there anything we're missing?

Another important feature in my opinion is this: https://github.com/Eventual-Inc/Daft/issues/2108

One of the coolest innovations of Daft is the multimodal type-system to handle all kinds of complex objects (not only pythons'), I'm really excited about that.

Keep the great work.

Cheers