d2iq-archive / serenity

Intel:Mesosphere oversubscription technologies for Apache Mesos
Other
69 stars 21 forks source link

Added Event Bus for pipeline interconnection messages and integrated it with ValveFilter #98

Closed bwplotka closed 9 years ago

bwplotka commented 9 years ago

Notes about the bus (from doc)

message SomeEventEnvelope {
    message SomeEvent {
        optional bool first_event_param = 1;
        optional bool second_event_param = 2;
        (...)
     }
     optional SomeEvent message = 1;
 }
  1. Create subscriber. Derive from ProtobufProcess.
  2. Install endpoint and handler for SomeEventEnvelope in Subscriber.
  3. Register subscriber: EventBus::subscribe(subscriberPID)
  4. When event is ready just do EventBus::publish(msg)
    • Publisher does not need to be an libprocess actor!

Based on Szymon's bus: https://github.com/mesosphere/serenity/pull/91

This PR includes integration with ValveFilter.

Tested in real flow.

skonefal commented 9 years ago

I don't like the way current bus demands changes in filter's code (it should not add demand if/elses). Also it MIGHT be better if it implements Akka-style topic-based subscription (message class subscription could be too wide...) and before we start to use it, we should move all our filters to libprocess, so they could be easily connected to bus... Do we want it now?

Some of the issues (filters code change) could be solved by having multiple instances of bus, or topic-based subscription, but we are not there yet.

bwplotka commented 9 years ago
  1. Your "dislikes" can be easily changed by adding topic string classifier to this EventBus. (:
  2. Currently we need this bus only for mentioned kind of messages to Valve. Do we have any other use cases right now? (Valve is a libprocess actor right now).
skonefal commented 9 years ago

I dislike both buses :) - each from different reasons. I think we shouldn't add topic yet - we are able to live without it now (and I'm a little afraid of it's possible drawbacks eg. hardcoding dependencies between filters inside filter code) - but it would be lovely if we carefully designed whole transition from Producer-Consumer to Topic-based bus pipelines. To avoid too much code tanglement, we should use bus only for inter-pipeline communication (it should be stated in doc). Please fix the mutlithreading parts, and we should be good to go.

bwplotka commented 9 years ago

Rdy for 2nd review @skonefal

bwplotka commented 9 years ago

Last nits repaired.

skonefal commented 9 years ago

LGTM

skonefal commented 9 years ago

LGTM