WaveBeans / wavebeans

Audio Processing. On scale.
https://wavebeans.io
Apache License 2.0
24 stars 0 forks source link

Mediums are a mess #30

Open asubb opened 4 years ago

asubb commented 4 years ago

There is a package inside exe project io.wavebeans.execution.medium which covers serialization issues for execution. It is complete mess by itself with self-written serialization.

The point of the medium is to provide functionality for effective transferring data for pods.

Current limitations:

  1. (for framework developer) It is very hard to add a new medium type onto the list, hence it is hard to add new types into the lib and have support of it during execution.
  2. (for framework user) It is impossible to add custom type if any, however lib functionality allows to do this.
  3. Despite the fact that current implementation is kinda fast (it's dumb though), it's definitely way faster than built-in java serialization, it is completely unsupportable.

What needs to be added:

  1. Transparent way for framework developers to extend the medium list without pain. Perhaps using 3rd party libraries. What should take into consideration: current implementation heavily reuses primitive arrays, otherwise that's a lot of pressure on GC (seems so).
  2. A way for framework users to add their own types and provide medium for this if that's required. Or better provide some generic medium by default.