ztellman / automat

better automata through combinators
588 stars 50 forks source link

Make FSMs serializable #58

Closed emlyn closed 7 years ago

emlyn commented 7 years ago

I'm using automat with Apache Spark. In order to avoid repeatedly recompiling my FSM, I compile it on the driver then send the compiled FSM to the executors, which requires them to be serializable.

It would be great if you are willing to merge this, so that I (and anyone else in a similar situation) don't have to maintain a separate fork.

I was only able to get :base compiled automatons to work, but not :eval, because deserialization requires the class to be available in the jar, and the eval one doesn't generate a class file when aot compiled.

This works as long as I specify :backend :base when compiling the FSM, but I was wondering what are the consequences of this? Presumably the the :eval backend is some sort of optimisation? What are the differences between the backends?

ztellman commented 7 years ago

The :eval backend is significantly faster than the :base backend. In practice, I'd suggest that you'd want to serialize the non-compiled FSMs, and then compile them locally using :eval. Is there a reason this wouldn't work for you?

emlyn commented 7 years ago

It's simpler to do the compilation once and serialize that, but if it's at a significant cost to performance I can also do it on the executors. I just have to memoize the compilation to avoid recompiling for every task the executor runs. In that case I'd just need the first of the two commits in this PR.

emlyn commented 7 years ago

I've made the change to compile the FSM on the executors (with memoisation), and that seems to work fine. So I've removed the unneeded commit from this PR, and left just the one making uncompiled FSMs serializable. Would you be willing to merge this?

ztellman commented 7 years ago

Merged, thanks for the contribution.