search

weld-project / weld

High-performance runtime for data analytics applications
https://www.weld.rs
BSD 3-Clause "New" or "Revised" License
3k stars 257 forks source link

Is weld still under research/development? #515

Open crystaldust opened 4 years ago

crystaldust commented 4 years ago

Hi guys, @sppalkia @deepakn94 @mateiz , we're considering to found a numeric computation WG in rust community and just wondering if weld project is still under research/development since there is no update for a few months? Maybe we can point out some new directions? Guess weld would be an important role in math computation field.

surechen commented 3 years ago

Hello everyone, I am also very interested in weld. I read two papers related to weld and ran several test cases using WeldNumpy. I did see performance improvements in some scenarios. During my use, I found that the current version does not support python3. I made some minor modifications to make it support python3. I noticed that some operations need to be added to WeldNumpy, and I hope to contribute some code. I am also very interested in the potential of weld in the parallel development of multiple machines. I hope to do some work for the weld community. Looking forward to your reply.

sppalkia commented 3 years ago

Thanks for your interest in Weld! Currently, none of us are actively working on the project, though I hope to return to it as a side project at some point.

When I was last working on it, I was "modernizing" the Python bindings and integrations. The python directory contains the old code with hacky NumPy and Pandas integrations that only work with Python 2.7 (plan is to eventually delete this). The weld-python directory contains the new code and a new version of Grizzly that works only with Python 3.x, and also contains many more tests, documentation on the functions, etc. This is where all the active development was last happening, but it does not have a NumPy integration yet.

hesingh commented 8 months ago

I am exploring Weld for gpu to optimize LLM models written in Pytorch. But first, I need gpu support in Weld. Please see https://www.modular.com/ and their Mojo lang to speed up ML workloads. I think Weld is a better tool than Modular.

hesingh commented 8 months ago

In our private repo of Weld, my company has modified Weld as follows.

  1. Refactored hardcoded llvm backend so that common backend code sits in the codegen directory and the llvm2 directory contains only LLVM specific code. So now, new backends can be added with ease.
  2. Modified Weld to use LLVM 14.0.
  3. Modified Weld to build and test on Apple Mac machine.

A question, what app do folks want to run on gpu?