numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io/
Apache License 2.0
1.09k stars 111 forks source link

All wire formats should be protobuf #1780

Open vigith opened 3 months ago

vigith commented 3 months ago

Summary

We use encoding/binary for serde. We need to standardize on a wire format that is scalable and also supports backward compatibility. encoding/binary encourages using gob or protobuf; I think protobuf has more benefits than gob because it is language agnostic and streaming is not implicit.

Use Cases

Affected components

### Components to change
- [ ] #1768 
- [ ] #1779 
- [ ] WAL storage serde using protobuf

Message from the maintainers:

If you wish to see this enhancement implemented please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.