hammerlab / ketrew

Keep Track of Experimental Workflows
http://www.hammerlab.org/docs/ketrew/master/index.html
Apache License 2.0
74 stars 10 forks source link

Build with `solvuu-build` instead of`omake` #529

Closed smondet closed 7 years ago

smondet commented 7 years ago

And now, ladies and gentlemen, Ketrew's 4th build-system!

ihodes commented 7 years ago

LGTM, this is much easier to follow than OMake!

agarwal commented 7 years ago

I'm curious how much using -j helps for a project like this. Any numbers?

smondet commented 7 years ago

@agarwal not really:

$ time bash -c 'make clean ; WITH_TESTS=true make JOBS=1'
Makefile:4: _build/project.mk: No such file or directory
ocamlbuild -j 1 -use-ocamlfind -plugin-tag "package(solvuu-build,nonstd)" project.mk
...

real    0m56.334s
user    0m42.083s
sys     0m11.864s

Vs

$ time bash -c 'make clean ; WITH_TESTS=true make JOBS=4'
Makefile:4: _build/project.mk: No such file or directory
ocamlbuild -j 4 -use-ocamlfind -plugin-tag "package(solvuu-build,nonstd)" project.mk
...

real    0m53.459s
user    0m40.596s
sys     0m11.012s

ocamlbuild is not famous for its parallelization of tasks…

Note also that Ketrew's build DAG has one big bottleneck: the call to js_of_ocaml that takes about 20 seconds, and on which the ketrew.lwt_unix library and hence all the apps/tests depend.

ihodes commented 7 years ago

On my machine, comparing solvuu-build-powered ocamlbuild with the "OMake way":

Solvvu-build

> make clean; time bash -c  'WITH_TESTS=true  make JOBS=1'
real    0m48.909s
user    0m35.815s
sys 0m6.615s

> make clean; time bash -c  'WITH_TESTS=true  make JOBS=4'
real    0m45.550s
user    0m34.469s
sys 0m6.407s

OMake:

> omake clean; time bash -c 'omake'
real    0m35.365s
user    0m40.963s
sys 0m9.134s

> time bash -c 'omake -j 4'
real    0m31.287s
user    0m39.913s
sys 0m8.551s
agarwal commented 7 years ago

Thanks for the numbers. There is still much room for improvement. One piece of that would be implementing a new rule engine, which would be a fair bit of work. Another piece is understanding compilation rules more deeply; note comment in solvuu/solvuu-build#14.