[WIP] Various new features: Enhanced JIT Compiler targeted to multiple devices, ONNX Support, et al. - Githubissues

hikettei / cl-waffe2

[Experimental] Graph and Tensor Abstraction for Deep Learning all in Common Lisp

https://hikettei.github.io/cl-waffe2/

MIT License

122 stars 5 forks source link

[WIP] Various new features: Enhanced JIT Compiler targeted to multiple devices, ONNX Support, et al. #154

Closed hikettei closed 3 weeks ago

hikettei commented 1 month ago

Refactors

[ ] Reduce code to less than 10000 lines.
[ ] Remove cl-waffe2-simd, with keeping AVX/Neon/SLEEF intrinsic supports.
[x] Remove JITCPUTensor instead of adding backends/aten.

AbstractNodes

[x] Changes on base-impl
- Unified and simplified arithmetic operations using !view.
  - cl-waffe2 uses broadcast+arithmetic ops instead of removing ScalarAdd/ScalarSub/ScalarMul/ScalarDiv/InverseTensorNode.
- Rename: !inv -> !reciprocal
- Fix typo: !leaky-relu
  - [x] keepdims option for !max/!min etc

Command Line Tool

[x] Uses Fiveam -> Rove as a testing tool.
[x] Make all tests hardware-independent
[x] Added a command line tool at ./roswell/waffe2.ros
[ ] From ONNX to C Interpreter mode (waffe2 codegen xxx)
- It generates the minimum C interpreter (calling CUDA/Metal functions), minimizing the dependencies.

ShapeTracker

[ ] [Ehhancement] [ ] to express ScalarTensor, instead of using [scal] and out-scalar-p=t.

Frontend

[ ] Implement from onnx to cl-waffe2 IR mode.

Backend

[x] Ready for implementing AbstractTensor backend
- [ ] Fast Conv2d kernel (reduces memory-latency, winograd)
- [ ] Ready for implementing GPU Supports, including Triton, Metal, CUDA, etc.
[ ] Remove cl-waffe2-simd
[ ] Remove obsolete SIMD dependencies
[x] Merge the aten-runtime branch.

APIs

[ ] Implement AbstractListTensor
- It basically behaves (Shape) Tensor.
- but when called with forward, it wraps λ function w/ loop.
- When called with !concatenate, it creates (LazyXXX ~ Shape) where LazyXXX = (length x) tensor
- It enables implementing kv-cache for transformer models.
[ ] Control-FLow
- [ ] IfNode IfNode ~ EndIfNode
- [ ] LoopNode LoopNode ~ EndLoopNode
- [ ] POV: Is wf2IR turing-complete?

hikettei commented 3 weeks ago

I will merge the changes once as it is unlikely to get much development time for a while T_T. (plus, the IR specifications I have formulated are so bad that it is difficult to add new features. I need to take enough time and redevelop the entire back-end from scratch. but is it worth it?...)