search

weld-project / weld

High-performance runtime for data analytics applications
https://www.weld.rs
BSD 3-Clause "New" or "Revised" License
2.99k stars 260 forks source link

Standardizing Weld's IR code #516

Open espetro opened 3 years ago

espetro commented 3 years ago

Hi there,

I have a growing interest in this project and, as it's asked in other issues, I don't know if this project is still active or not. That being said, I've seen that the current implementation seizes the power of Rust, but what about using IR frameworks such as MLIR ?

sppalkia commented 3 years ago

Good question, MLIR is indeed interesting for Weld's use case. We haven't tried it before (and we're currently not working on Weld actively), but if someone were to take a stab at using it as Weld's backend, I'd be happy to provide reviews/suggestions.

The Weld frontend currently doesn't have too many optimizations around n-ary arrays (and in fact, does not yet support a tensor type since the primary use case for now has been more 'slice and dice' ETL-like workloads in libraries like Pandas), so that may be a good starting point: if the frontend is capable of applying some interesting optimizations over tensors, the availability of these types in the backend IR will become much more compelling.