search

vmware-archive / sql-to-dbsp-compiler

Compiler translating SQL view definitions into DBSP circuits (https://github.com/vmware/database-stream-processor)
Other
28 stars 5 forks source link

New JSON representation for JIT compilation #123

Closed mihaibudiu closed 1 year ago

mihaibudiu commented 1 year ago

Instead of serializing the Java representation as JSON, we actually generate a custom JSON representation of a pipeline to match the expectations of the JIT compiler, which is defined as code here and in the related files: https://github.com/vmware/database-stream-processor/blob/dataflow-jit/crates/dataflow-jit/src/ir

Currently most of the programs would fail to compile to JSON; we have tested Map and Filter operators successfully. There are many TODOs in the code, which will be filled as I get more understanding of the exact JIT constraints. But this commit allows the JIT compiler to progress and provides test cases for it. When that compiler is mature enough we will run side-by-side Rust and JITted code to compare the results.

mihaibudiu commented 1 year ago

Rebasing this on main becomes harder and harder as the number of commits grows, so I will resolve the conflicts and merge this branch into main if everything works fine.