WallarooLabs / wally

Distributed Stream Processing
https://www.wallaroolabs.com
Apache License 2.0
1.48k stars 69 forks source link

Revisit Python-pony interface (Machida) #2351

Open nisanharamati opened 6 years ago

nisanharamati commented 6 years ago

Now that we have >1 year of experience with the Pony-Python interface under our belt, I think it's a good time to think about what's we would like to improve going forward (especially as the Python3 API approaches).

Some things worth looking at (feel free to edit this issue to add your own!)

/cc @WallarooLabs/wallaroo-labs

caj-larsson commented 6 years ago

I like the idea of removing the computation class and moving that structure into pony instead. But after trying out pickle on a function with added attributes I found that they don't get pickled. I guess that encode/decode functions can solve that if we also move this meta information into pony.

I've added a check to make sure that the users encode function either returns a bytes() or a unicode() compatible value. More type and value checks are definitely warranted in the action list parser.

We could also integrate with the optional typing in higher versions of python to disallow an explicit mismatch. However in runtime there is of course no way of making sure that we get the correct python types.