search

numaproj / numaflow

Kubernetes-native platform to run massively parallel data/streaming jobs
https://numaflow.numaproj.io/
Apache License 2.0
1.1k stars 112 forks source link

structured concurrency is lost in actor pattern #2151

Open vigith opened 4 days ago

vigith commented 4 days ago

spawn handlers are lost in https://github.com/numaproj/numaflow/blob/main/rust/numaflow-core/src/source.rs#L111-L126

vigith commented 4 days ago

the moment we add handler to struct, Clone will no longer be applicable

BulkBeing commented 4 days ago

I think we should rely on message passing to shutdown tasks. Another possible option is to return join handles in a wrapper struct and abort tasks within the Drop implementation of it.