vmware / differential-datalog

DDlog is a programming language for incremental computation. It is well suited for writing programs that continuously update their output in response to input changes. A DDlog programmer does not write incremental algorithms; instead they specify the desired input-output mapping in a declarative manner.

MIT License

1.38k stars 119 forks source link

Speed up compilation, reduce disk footprint of multiple datalog programs #58

Open ryzhyk opened 6 years ago

ryzhyk commented 6 years ago

Currently every test in tests/datalog_tests downloads and compiles its own Rust dependencies, as well as makes its own copy of differential_dataflow, taking a couple of gigabytes of space per test. There must be a way to share common parts across all tests.

d-e-s-o commented 5 years ago

I think one way to increase sharing would be to somehow use a Cargo workspace as we have done for the crates below rust/template. However, I am not sure if nesting of workspaces is possible and how to keep the workspace up-to-date with newly added tests to begin with. So it does not sound like an appealing way to go about that to me, but at the same time it's the only thought that came to mind.

ryzhyk commented 4 years ago

501 made significant progress towards resolving this issue. Remaining TODOs:

[ ] Move differential-datalog crate out of the template and into a shared location, e.g., $DDLOG_HOME, so we only need to compile it once and not for each DDlog program. This way we should be able to run more tests.
[ ] See if we can speed-up recompilation of large programs even more, e.g., by placing each value type in a separate module that will only be recompiled when that type changes.
[ ] Smarter memory allocation strategy that would allocate multiple values at once using domain-specific information from DD.

ryzhyk commented 4 years ago

More improvements in #512 and #520.

ryzhyk commented 4 years ago

It appears that a lot of remaining overhead comes from auto-derived trait implementations. So the next step would be to see if we can do better by getting DDlog to generate these trait implementations instead.