Paper 5a: Real-Time Middlewarte for Cyber-Physical Event Processing

[ericwendt]

Please add feedback on the paper

[AkinoriKahata]

Reviewer: Akinori Kahata Review type: Comprehensive

1. The problem being solved

   • As cyber-physical systems connected with many sensors and actuators, the systems receive many inputs and processes many events concurrently based on the priorities of each process.
   • Because the priorities and processes of each event are diverse, event processing management of cyber-physical systems is difficult but important for the quality of the systems’ output.
   • Existing stream processing software cannot provide prioritized processing and share with time consistency; then, the authors try to develop the software which can manage the event processes based on priorities of each process.

2. The main contributions

   • The paper introduces new type software for event processing, “CPEP”, which can manage the event with four features; configurable processing operations, processing prioritization and sharing, enforcement of temporal validity and shedding, and efficient concurrent processing.
   • As a result of the empirical experiments, CPEP with TAO software combination is superior to Apache Flink software from perspectives; the total timely-event latency, thorough put, effectiveness of using memory.

3. Questions

   • What is happen when the high-quality event comes to the event processor a few milliseconds after the middle-quality event occurred? I mean, some processes should be processed concurrently, but the start timing of the event is not the same. I cannot understand the algorithm on how to decide the real priority.
   • The differences between CPEP and existing software, such as Apache Flink, are not apparent for me. In §2, there is an explanation about Apache Flink has not featured of time consistency. However, I cannot understand other differences. Then my question is, what is the most significant difference with Apache Flink. In other words, what is the most difficult challenge of CPEP development?

[lrshpak]

Reviewer: Lily Shpak Reviewer Type: Critical

Main Problem Being Solved

For systems that have both hardware and software components, real time data processing is essential. It is important for data to be processed in real time so it can be actuated correctly. Another issue they are attempting to solve is making sure that new and old data do not get processed together.

Main Contributions

The authors solve this issue by creating a system that they call Cyber-physical Event Processing (CPEP). A main attribute of this system is that it separates the event processing into three levels; low, medium, and high priority. This separation allows events to be processed concurrently.

Questions

1. The authors implemented this system within TAO, which they say is a widely used middleware, if someone using this system wanted to use something other than TAO would they be able to?
2. They bring up related work, but what sets apart their research from research that has already been done?
3. They process different priorities separately but then how do they make sure that after these events are processed the low priority processes can run at the same time as high processes?

   Critiques

4. I think that they could have explained the related work at the end of the paper. It confused me a bit because they introduce their research and then they jump right into related work instead of fully explaining their work.
5. I do like that they have all these experiments that they ran to test their system. I think it is good that they laid out each experiment and what they were testing.

[searri]

Reviewer: Rick Sear Review Type: Comprehension

Problem being solved

The authors say there are 4 issues to do with cyber-physical systems:

1. Many little events that build together into complex operations
2. Event prioritization
3. Time consistency and enforcing time validity of events
4. Concurrent data processing while maintaining efficiency

Important contributions

The paper introduces the CPEP middleware system which attempts to be customizable depending on use case and provide a good (space-efficient, fast, accurate) solution to all four of these issues.

Questions

• The paper runs performance metrics against Apache Flink. Not knowing much about middleware, is this a fair comparison? Are there other middleware platforms besides Flink which might outperform CPEP?
• CPEP is nice because it's very configurable. Experiment Set 4 compares some combinations of its cross-priority sharing option and its event shedding option. I'm wondering what examples of actual contexts might need these different combinations? Is one much more prevalent than the others?
• How reliant is CPEP on using the TAO software (i.e. could I easily implement it in something other than TAO if I wanted to)?

[grahamschock]

Reviewer: Graham Schock Review Type: Comprehensive

Problem being solved Constraints are important in cyber physical systems. It is vital to the success of a cyber physical system that we are able to process events efficiently, safely and within the time constraints necessary to the application of the physical system. The paper gives an example of an electric grid that requires a latency time to be less than 50ms. To make things worse multiple sensor integration is required for certain applications which adds more stages and layers of complexity.

Main Contributions In order to meet this need the paper introduces CPEP which stands for cyber physical event processing. This CPEP system allows for configurable processing operations (which allow for simple and complex event processing), a level of prioritization and resource sharing which allow for important events to be processed sooner, an enforcement of constraints and removal of outdated data, and concurrent processing which leads to more efficiency.

Questions

1. How do we determine what is high priority in a system like this? It seems the middle ware relies on a configuration file for reliable information. Can this configuration file always be trusted?

2. It seems security was not discussed much. I wonder how we can add to our middle ware to ensure that everything is safe and secure and what trade offs that has? _

3. In figure 2 it talks about how we can generate priority through event processing. It also talks about "operators" and how execution of an operator produces an event. What is an "operator" and how does it create an event?

[mjhegarty]

Reviewer: Michael Hegarty Review Type: Comprehensive

Problem Being Solved

Cyber-Physical systems are becoming more and more complicated in terms of large numbers of inputs and outputs, and the data that needs to be processed often has complex operations that might involve multiple inputs such as sensor combination. In addition some inputs have higher priorities than others and should be processed before lower priority ones. In addition events have absolute timing constraints they need to meet(the real world system needs to respond to an event within a certain amount of time otherwise the data isn't relevant anymore) as well as relative timing constraints(if 2 inputs are used together to decide something they need to happen at around the same time).

Main Contributions

The authors create a middle ware to solve these problems called CPEP. It is implemented off of a middle ware called TAO to create a configurable system capable of processing events with different levels of priority(maintained via preemption) while maximizing concurrency of processing. In addition the system strives to maintain absolute and relative time consistency between events and can validate if these constraints are met or not.

Questions

1. I understand the concept of a system with multiple inputs and outputs but I was wondering what real world examples of this would look like? Is this something that would be on an embedded linux system?
2. The system abstracts the idea of sampling the input data sources by having suppliers that give it the data. Are there trade offs to a system that does this instead of one that includes the timing requirements to sample these data steams at their various frequencies?
3. The idea of checking the timing validity of an event and marking/discarding it if doesn't make those requirements is interesting to me. Is this level of unreliability of acceptable in most IoT systems?

[hjaensch7]

Reviewer: Henry Jaensch

Review Type: Comprehension Review

Problem Being Solved

Cyber physical systems pose unique challenge in the diversity of components that need to work together to achieve a task. Physical sensors all generate different kinds of data at different periods. A CPS is also usually responsible for completing more than one task. The more complexity added means that more work needs to be done to guarantee any constriant. A wide variety of sensor events and actuator requirements paired with variable task prioritization can result in a complex system with many points of failure.

Main Contributions

This paper proposes a system that prioritizes event based processing called CPEP. This system uses the idea of an event to compartmentalize processing and assign priority. Events can vary greatly in complexity and timliness. CPEP attempts to maximize concurrency between events to improve the efficiency of the system while still meeting deadlines and maintaining proper prioritization.

Questions

1. The priority section discusses handling cross-priority sharing and how individual threads are assigned adjacent priorities, but how does this system handle concurrency issues created by different processes of differing priorities interact with the same memory?

2. The system was built on top of TAO middleware and designed to emulate some of the existing TAO mechanisms. Without the constraints of attempting backwards compatability would CPEP be implemented in a different way?

[ratnadeepb]

Reviewer: Ratnadeep Bhattacharya Review Type: Critical

Problem Being Solved: In real time event processing for cyber-physical systems (CPS), events need to be processed in a timely manner and with regard to temporal validity of the data being used. This work implements a middleware for the same that reduces redundant work, maintains temporal validity of data and provides a way for the system to accommodate user level priorities.

Contributions The main contribution of this work is that it provides a middleware that can combine simple operations to build up more complex workflows. It improves over message passing systems like Kafka, which does not provide end-user defined priority based processing, over Data Distribution Service, which does not process events and over Flink, which does not provide temporal consistency.

Figure 2, 3 and 4 in the paper provide an excellent overview of the design of the system. The overall design is a directed acyclic graph of operators. Any operator might receive events from multiple operators (upstream). Any operator can be scheduled once all of its input events have arrived (upstream operators have completed their job).

Equation 1 describes the event's lifetime. Equations 2 and 3 describe absolute validity of internal events (events produced by upstream operators and not suppliers). These flow down from the event interval constraints of the supplier events. Similarly, relative time consistency is defined by Equations 4 and 5.

The workers itself can be divided into multiple priorities. Priority level of operators flow in the reverse direction than that of timing consistencies. That is, consumers provide priority levels for each end operator. Any upstream operator is given the highest priority amongst the priorities of its downstream operators.

Each priority level has its own event processor which in turn has a bunch of worker threads and a bunch of mover threads. There are also three queues - InputQ -> buffers all supplier events at the same priority level, PendingQ -> holds events for the downstream operators, MovingQ -> for cross priority sharing. When cross-priority sharing is required, the worker thread that produced the event places it in the PendingQ and any free mover thread places it in the MovingQ.

Questions

1. The system does not bind threads to CPUs. This is good for resource utilization but how much context switching is happening?
2. This is a multi-threaded event processing system with a bunch of data structures per process. How does it scale? What kind of coordination is required and how does it slow the system down when using multiple CPUs?

[pcodes]

Reviewer: Pat Cody Review Type: Comprehensive

Problem Being Solved

Cyber-physical event processing is difficult because applications may have different temporal and latency requirements. Furthermore, event processing operations have varying degrees of complexity, and having these operations work concurrently is challenging.

Main Contributions

This paper introduces CPEP, a middleware for cyber-physical event processing designed to address the 4 problems described in the introduction. CPEP provides relative and absolute time consistency for events, and schedules based on event priorities.

Questions

• If the system is designed to handle multiple applications, what kind of security/isolation does this provide, if any?
• How do the authors envision deploying this system to the real-world?
• The authors provide some base discussion on making the system distributed. How/when do they envision this being useful, and what kind of tradeoffs would this present?

[tuhinadasgupta]

Reviewer: Tuhina Dasgupta Review Type: Comprehension

Problem:
The author identifies four primary issues with cyber-physical systems (event prioritization, time validity, concurrent data operations, and complexity of the system) Contribution:
They propose a solution, called CPEP, which is a customizable middleware system that can solve all of the above-identified issues while having good space & time complexity by maximizing concurrency. Questions:

1. After doing some basic research (Googling) there seems to be a vast number of types of middleware systems. The only comparison made in the paper is against Apache Flink. What this because Apache is the most comparable or for some other reason?
2. Surprisingly, this paper is one of the few so far to not address security and privacy issues. Are there any with this system?
3. I understand that the system is multi-threaded but don't fully understand the cost of switching between flows of execution. What's the overhead like?

[ericwendt]

• @AkinoriKahata, Akinori Kahata, Comprehension:

  • Summary: Aki questions about the scheduling algorithm as a whole and how it handles scheduling high and low priority tasks. He also inquires about the usefulness in comparing CPEP to Apache Flink. Aki shows clear understanding of the benefits that CPEP has over other real-time middle-wares, such as prioritization with time scheduling.
  • As far as I’m able to gather, this system simply provides a conduit for task scheduling, such as the supplier, worker, and mover threads. It does not reinvent any core scheduling properties and uses a fixed priority\algorithm. I believe that a higher-priority event that comes later than a lower-priority event would still be treated as more important. Such high priority events tend to be time-sensitive where missed deadlines are unacceptable. CPEP gives me every reason to believe that the system will still function as such.

• @lrshpak, Lily Shpak, Critical:

  • Lily brings up the middleware TAO and the complexity of moving to a different middleware framework. She asks about how CPEP handles processes running at the same time in different priority levels. Lily shows an understanding of the benefits of a real-time middleware such as CPEP and Flink, and brings up valid concerns over its integrity.
  • The data structures used in CPEP are native to C, which is a very versatile language that can run miles on embedded, low memory, and power constrained systems. TAO is an open-source software, probably a good motivation as to why CPEP decided to work with it, but the beauty of what CPEP outlines is its core concepts, which seem independent of platform. The threading and hierarchy model used in CPEP should be portable to other technologies. Still, it would have been interesting to see speed comparisons using another middleware, although presents another huge layer of complexity.
  • Each Event Level has separate worker threads that can run concurrently, and furthermore multiple worker threads within these levels can be initialized.

• @searri, Rick Sear, Comprehension

  • Rick brings up the four main features of a real-time data streaming middleware, as well as many valid concerns for how the platform is managed. How CPEP falls into TAO was another concern brought up here.
  • Apache Flink is one of the leading data streaming platforms, providing high throughput at super low latencies. This is absolutely a fair comparison, as it seems CPEP was built to directly compete with this platform. If CPEP can really perform as well as its graphs indicate, it could provide quite a useful service.

• @grahamschock, Graham Schock, Comprehension

  • Graham takes a unique mindset on this paper, bringing up the time constraints of CPEP. The time constraints are one of the most important factors that a middleware needs to address, and Graham nails it on the head when he says that multiple sensors causes greater complexity.
  • For large scale data streaming applications that would want to utilize CPEP or Apache Flink, it is assumed that a hierarchy of these priorities are thoroughly specified by the administrator. I would assume these would be well thought out, but CPEP is completely reliant on this to run properly.
  • Good question. Right off the bat, a huge security concern for any scheduling platform is making sure that priorities aren’t starved. Low priority does not mean “run if possible.” All events need to be serviced. Unfortunately, this paper does little to clarify this issue, and makes use of Fixed Priority Scheduling, which by nature does not protect against starvation.
  • I see “operator” as an extension of “event.” Imagine an operator is a thread running Fourier Transform on a set of data. This operator gets passed data, and starts going to work on one of the worker threads. This then becomes and “event,” running a process on said data. Operators are separated by types of actions occurring, and events are literally the process. Make sense?

• @mjhegarty, Michael Hegarty, Comprehension

  • Michael focuses on the timing portion of this paper, such as the relative and absolute constraints for meeting a deadline. He also wonders about the real-world applications of needing such a scheduling system.
  • Good question! Any embedded system that needs fasting-acting processing and low latency can make use of CPEP essentially. Image processing, sensor readings of various kinds. All of these readings are likely coming from distributed devices that work together to make some kind of decision. Having faster throughput and higher microprocessor efficiency can make these decisions happen faster based on a hierarchy of priorities. It may take a memory hit when sharing events between priorities (although many of their graphs seem to indicate otherwise astonishingly, not sold on this one), but often times this is the price of speed.
  • Supplying timing boundaries is still included in CPEP’s design. It makes decisions to schedule these processes more efficiently to produce lower latency, much like Apache Flink, but it shouldn’t compromise the core principles of Fixed priority scheduling.
  • If data is so old that processing it would require input from sensors that are miles ahead, then it stands to reason that it was either low priority or that a ton of readings came in at the same time. It’s important that the entire system be acting in real-time, otherwise the data is useless, and could potentially result in an accident if old. Fantastic questions, really giving me some cool stuff to think about.

• @hjaensch7, Henry Jaensch, Comprehension

  • Henry brings up the deadline constraint problem that CPEP attempts to mitigate. The interesting part of his discussion is the notion of failure, that is, the system CPEP sets up is not designed to solve the problem of scheduling, it simply gets closer to an optimal CPU efficiency, and expects some data to be shedded/failed.
  • Great observation! Standard C data structures are used in these threads, and as you might expect, CPEP uses built in thread locking mechanisms as well.
  • Yes. I assume this was done to cut down on production time to get a working product. It stands to reason that sufficient effort would be required to port this to another framework, but the concept of what CPEP does is platform independent as long as they provide the correct concurrency structures.

• @ratnadeepb, Ratnadeep Bhattacharya, Critical

  • Deep shows a fundamental understanding of how CPEP works overall, calling in example of the thread design and how events are moved between Event Processors and how operators "trickle down" to each other to avoid operators working with expired data. He also inquires about the overhead incurred from designing such a system that does not assign processes to threads.
  • Woah, interesting. I assume it’s going to be huge amount of the time. If you were to assign worker threads to cores, this would sufficiently cut down on the number of switches, but I think it would fundamentally oppose what the middleware is trying to achieve, which is CPU efficiency. Fantastic observation.

• @pcodes, Pat Cody, Comprehension

  • Pat shows understanding of how CPEP works at a high level, citing its core features. He also questions about the real-world utility of this system and how the company envisions using this at scale.
  • Locking mechanisms are used when editing data structures. This can be all be handled within CPEP. However, just like normal processing, there’s no guarantee that a programmer writes code that stops. Still very possible for one thread to be waiting on a broken sensor that never sends required data. Certain guards can be put in place but a level of unpredictability is expected.
  • If we scale this system, I believe a certain amount of latency will be incurred. Also, if this is going to be deployed on a network of sensors, safeguards against broken systems need to be put in place. As long as events are correctly taking this into account and have conditions for this, the scheduling of CPEP should take care of the rest. Certain suppliers will have higher priorities than others, but CPEP should be able to handle this regardless of user specifications.

• @tuhinadasgupta, Tuhina Dasgupta, Comprehension

  • Tuhina brings up the main contributions CPEP has to real-time processing, and inquires about the security concerns of this system as well as middle-ware competitors.
  • The cost of context switching will definitely be high here, especially since worker threads are not assigned to a single CPU. This is compounded even further when event sharing is not present, as more memory is needed to compensate for Event Processors not distributing workloads efficiently.

[jacobcannizzaro]

Reviewer: Jacob Cannizzaro Reviewer Type: Critical

Main Problem Being Solved

Cyber-physical systems absolutely need to be able to make strict deadlines. On constrained systems this can be hard to do as there are other difficulties such as starvation for threads of different priority as well as validity of data that must be kept timely. It is important to meet these other demands for all the complex events being operated on while not losing the ability to stay within low latency requirements.

Main Contributions

This paper introduces the real-time middleware for cyber-physical event processing (CPEP). This system helps to integrate the complex event processing while allowing process prioritization to effectively meet high priority thread needs while not starving lower priority ones. It also helps to maintain temporal validtity of data so that the system is not acting on outdated event information.

Questions

1. Can we go over the event processing stream diagrams in figure 7? I do not understand the difference between (a) and (b) and why the high priority threads are potentially jumping all the way down and appear to stay at a low priority.
2. Is there a more comparable framework to CPEP than Apache Flink?

Critiques

1. I don't know much about Apache Flink but they say that it is an open source stream processing framework. They don't really say that it is meant for middleware purposes. Looking them up, I see it is meant for event driven applications but doesn't mention edge computing or middleware, or even embedded devices. Maybe there isn't anything out there exactly like CPEP but then I think that they should have expressed that and expaned upon that as their contribution. This just seems like too easy of a comparison.
2. I do not have much experience with the exact subject here, but think the paper is just poorly written, as if it needed one or two more draft cycles. One blaring example is the repeated pattern of listing a related project without really describing its goals, saying what it can't do, and then: "In Contrast, CPEP" (which appears 4 times in the text). Maybe I'm being overly critical, but three of them are on the same page.