The entire graph compilation and exporter, including Transformer, Tokenizer, Data Loader, weight parser, etc...

[hikettei]

Goal:

• [ ] Concept: Caten/workflow enabling the entire graph compilation (including tokenizer, data loader, token sampler etc...)
• [ ] Enable the entire graph abstraction using Caten
• [ ] Implement SentencePiece (compile to Render-Graph, and C)
• [ ] Generate xxx.dylib
• [ ] proceed in the script (shape information is available everywhere)

[hikettei]

something like huggingface trainer (but compilable)

# Workflow of gpt-2 inference
      Input
        |
 [Action: Tokenizer]
        |
  |-[Action:Run]-|
  | Transformer  | <-------------------------------|
  |--------------|                                 |
        |                                          | x N
        |----[Action: Logits Argmax Concatenate] --|
        |    
    [Output]
        |

=>

(defworkflow LLM-Codegen (inputs model)
  ...
   )

[hikettei]

Requirements on Action?

• [ ] Each action is CLOS class
• [x] Compilable to C or other languages using the faculty of caten/ajit (i.e.: FUNCALL, IF, ELSE, ENDIF, FOR, expr is available)
  • [x] i.e.: Consider implementing Lisp-Like DSL into render-graph
• [ ] Action is able to invoke Module(compiled graph)

[hikettei]

WIP: From Common Lisp to Render-Graph Lowerer

(defaction TestFunc (n)
  (declare (type :int64 n))
  (let ((m (* 10 n)))
    (if (> m 1)
        (let ((s (* m 10)))
          s)
        n)))

(defaction Test (x i k)
  (declare (type (:array :row (i k) :float) x)
           (type :int32 i k))
  (setf (aref x 2 2) 2.0)
  (let ((a x))
    (setf (aref a 3 2) 1.0)
    a))

(defaction Matmul (x y z m n k)
  (declare (type (:array :row (m n) :float) x)
           (type (:array :row (n k) :float) y)
           (type (:array :row (m k) :float) z)
           (type :int32 m n k))
  (dotimes (mm m)
    (dotimes (nn n)
      (dotimes (kk k)
        (setf (aref z mm kk) (+ (aref z mm kk) (* (aref x mm nn) (aref y nn kk))))))))