[KIEKER-500] Kieker.Analysis: Configuration Meta Model and Pipe&Filter Framework

[rju]

JIRA Issue: KIEKER-500 Kieker.Analysis: Configuration Meta Model and Pipe&Filter Framework Original Reporter: Nils Christian Ehmke

---

As some of you may know we changed the plan and decided to develop at first a meta-model for the analysis and implement a corresponding interpreter in order to configure the analysis before starting with the web interface.
Everything I mention in the following points can be found in the branch "1.5-analysis".

1) I modified the meta-model in order to add some more semantics and possibilities. For various reasons I also add the "Reader" again as a class.

2) Currently the delivering of the monitoring records is done partially with the help of the analysis controller. With the new (planned) connection, it would work slightly different:
Every plugin (reader + analysis plugin) has output ports but only the analysis plugins have input ports. The connection between the single components (for example: The output of Reader X is connected with the Analysis Plugins Y and Z) is done by the AnalysisController.
The plugins themselves have to register their input and output ports (in other words: They have to create them and register them with a name).
The plugins decide what they do once data is sent via the input ports, but they also have to sent their new data via the output ports (if they want). Theoretically every kind of data (as long as it's still a Java class) can be send via the ports.

Once the AnalysisController is started, it notifies the reader to start reading. The reader itself is responsible for delivering the data via the port.

3) A simple example is available. If you want to use it, you have to use the method "loadFromFile" of the AnalysisController in order to deliver the file "../1.5-analysis/model/example/My.analysismetamodel" (Important note: The init string for the reader within this file has to be modified. It should point to the directory in the examples which contains test data. Also: Ignore the fact that some parts of Kieker cannot be compiled currently. You also may have to add the Plugin dependencies in the buildpath). You can then take the first (and only)
instance from the returned map and run it. This simple example reads the testdata (via a FSReader) and delivers the records to the DummyRecordConsumer which writes the records to the standard output.

4) If you take a look at the sources, the Readers have now to inherit from kieker.analysis.plugin.AbstractMonitoringReader (The package will be corrected later) and the Analysis Plugins from kieker.analysis.plugin.AbstractAnalysisPlugin. As mentioned, the registering of the new ports and their distinct behavior has to be implemented by the person who wants to implement an own plugin. That means also that every plugin and every reader would have to be modified in order to suit this new "connection system". Furthermore all plugins would need a default constructor.

Feel free to share your thoughts.

[rju]

author Jan Waller -- Wed, 2 Nov 2011 15:32:01 +0100

Some minor problems with the genModel.

Currently the Eclipse-Project Names used as the target for code generation are hardcoded. So regardless of the project, the model is in, the target code will always land in the project named "Kieker". Simialr with .test .edit etc...

Furthermore, wouldn't it be better to include the generated source code into the project (svn)? (or perhaps they are already...)

The new lib dependencies should be added to the lib folder and included in the generated .classpath sample.

[rju]

author Jan Waller -- Wed, 2 Nov 2011 15:44:30 +0100

KIEKER-27 Done should probably be fixed in the context of this new feature set.

But be careful with too many changes in the FSReader context. (Perhaps the results of KIEKER-332 Done could be transfered here?)

[rju]

author Jan Waller -- Wed, 2 Nov 2011 15:55:22 +0100

Generated code should land in a new folder "src-gen".
The Kieker default naming schema should be used:

• Interfaces start with I -> I!{0}
• Normal classes without Impl -> !{0}
• etc.

[rju]

author Jan Waller -- Wed, 2 Nov 2011 16:01:11 +0100

Shouldn't the class Class be the Javaclass Class?

[rju]

author nils-christian -- Wed, 2 Nov 2011 16:34:28 +0100

Replying to [jwa|comment:3]:
> Generated code should land in a new folder "src-gen".
> The Kieker default naming schema should be used:
> * Interfaces start with I -> I!{0}
> * Normal classes without Impl -> !{0}
> * etc.
>

Corrected in Revision 2239 (More precisely: The code is generated in the source-folder src-gen/analysis). Does the etc. contain something else not listed?

[rju]

author Jan Waller -- Thu, 3 Nov 2011 17:08:13 +0100

Nein, das war das was mir eingefallen ist.
Man kann sich noch überlegen, ob man einen extra Order impl braucht, aber das ist mir eigentlich gleich.

Danke für die schnellen fixes

[rju]

author nils-christian -- Thu, 3 Nov 2011 22:01:42 +0100

Replying to [jwa|comment:4]:
> Shouldn't the class Class be the Javaclass Class?

Good point! I replaced the whole EClass and the classnames of the plugins (formerly EStrings) with EJavaClass<T> [java.lang.Class] (Revision 2245). Now it is a little bit difficult to use the editor of the generateable Kieker.editor, as the editor doesn't know the kieker classes, but the files can be modified manually (as the classes are just stored as the full qualified class names).
The example file has been modified to suit this new model.

[rju]

author Jan Waller -- Fri, 4 Nov 2011 10:42:46 +0100

An important point (for me) is:
> The new lib dependencies should be added to the lib folder and included in the generated .classpath sample.
If you could fix this, it would really be helpful. Thanks

[rju]

author nils-christian -- Fri, 4 Nov 2011 15:24:32 +0100

Replying to [jwa|comment:8]:
> An important point (for me) is:
> > The new lib dependencies should be added to the lib folder and included in the generated .classpath sample.
> If you could fix this, it would really be helpful. Thanks

Done in Revision 2246. I didn't add anything to the lib folder (I am not sure if this is necessary in the first place), but the plugin dependency and the src-gen/analysis folder are now listed in the classpath-file.
Can you confirm that the file works now?

[rju]

author Jan Waller -- Fri, 4 Nov 2011 15:34:28 +0100

It still doesn't find the required jars: "The import org.eclipse.emf cannot be resolved".

EMF creates compile time and runtime dependencies, thus we will need the jars in any case, if we don't want to restrict all further compiling and execution to the Eclispe IDE (imho).

[rju]

author nils-christian -- Fri, 4 Nov 2011 15:41:25 +0100

Replying to [jwa|comment:10]:
> It still doesn't find the required jars: "The import org.eclipse.emf cannot be resolved".
>
> EMF creates compile time and runtime dependencies, thus we will need the jars in any case, if we don't want to restrict all further compiling and execution to the Eclispe IDE (imho).

Yes, I can do this, but I am not sure whether one can still generate the code from the ecore-models without eclipse (and the necessary plugins) or not.

[rju]

author Jan Waller -- Fri, 4 Nov 2011 15:42:42 +0100

External code generation is not important (for now), but one has to be able to compile and execute the generated code.

[rju]

author nils-christian -- Fri, 4 Nov 2011 15:55:07 +0100

Replying to [jwa|comment:12]:
> External code generation is not important (for now), but one has to be able to compile and execute the generated code.

I added the dependencies which should be necessary according to Eclipse. Are there any more dependencies which still cannot be solved?

[rju]

author Jan Waller -- Sat, 5 Nov 2011 10:43:41 +0100

see also KIEKER-648 Open

[rju]

author Jan Waller -- Mon, 7 Nov 2011 17:30:28 +0100

final protected void registerOutputPort(String name, IOutputPort port)}}}
Why the name? Port already has a name, why not simply use the name of port?

Code currently would look like this (double "out"):
{{{
  this.outputPort = new OutputPort("out");
  super.registerOutputPort("out", outputPort);

[rju]

author André van Hoorn -- Mon, 7 Nov 2011 17:55:46 +0100

Replying to [jwa|comment:15]:
> final protected void registerOutputPort(String name, IOutputPort port)
> Why the name? Port already has a name, why not simply use the name of port?
>
> Code currently would look like this (double "out"):
> {{

{ > this.outputPort = new OutputPort("out"); > super.registerOutputPort("out", outputPort); > }

}}

As indicated by the parameter name, the string passed to the OutputPort should be a decription (describing the semantics of the output) rather than a (unique) name.

   public OutputPort(final String description) {
        this.description = description;
    }

So a better example would be

this.outputPort = new OutputPort("Returns valid message traces");
super.registerOutputPort("validMessageTraces", outputPort);

In a "clicky bunti" GUI, this description could, for example, be displayed as a tooltip.

[rju]

author Jan Waller -- Tue, 8 Nov 2011 15:11:43 +0100

I converted the FSReader to use Configuration objects instead of the old method.
Now there is only once constructor, which is required by all Plugins. A constructor with a single argument of Configuration.

Be careful to use the correct Configuration (from kieker.analysis).

Further points:

• Use Concurrent Classes instead of regular ones, probably more than one thread will access your data structures!
• Use unmodifiable Collections, check wheter to use lists or maps or sets or ...
• Why Collections instead of Arrays?

[rju]

author nils-christian -- Wed, 9 Nov 2011 22:00:36 +0100

Question: How should we handle the "old" structure of the AnalysisController? Currently there are still methods like addRecordConsumer and registerPlugin which are not used by the new pipe & filter structure (as the structure is currently only configurable via the meta-model).
Either
a) we would remove these methods, but in this case it wouldn't be possible to configure the analysis easily without using the meta-model.
or
b) we would have to change the methods in order to access the pipe & filter structure. This is probably the way to go (especially because the components which use this methods wouldn't have to be modified entirely), but this would also probably mean that the methods would have to operate on AbstractAnalysisPlugin instead of using the IMonitoringRecordConsumerPlugin and IAnalysis interfaces (in fact I am not sure whether the first interface can be used at all). What do you think?

[rju]

author André van Hoorn -- Wed, 9 Nov 2011 22:20:29 +0100

I'd vote for b) as well.

This would still allow to define pipe-and-filter structures manually (or to add additional plugins). And if the AnalysisController would provide a method like saveConfiguration(String filename) (which would also apply to configurations previously loaded using in instance of the meta-model), manually created configurations could even be serialized. Did you think about such method already? I think we need it any case. We should also think about whether the AnalyisController keeps an in-memory configuration model up-to-date or if it computes the model on a call to saveConfiguration by traversing through the pipe-and-filter architecture (collecting the plugins, its configurations, and the interconnections).

[rju]

author Jan Waller -- Thu, 10 Nov 2011 15:09:02 +0100

I'm ok with b)

Other point:

• I improved the handling of the Configuration in all Readers and the DummyReader.
• Now every plugin requires an implementation of protected abstract Properties getDefaultProperties(); to determine its default values.
• Instead of the properties function getProperty use the specialized functions of Configuration (AbstractConfiguration)
• On points that give a Configuration to the plugin (should only happen in the Constructor call), don't give the complete Configuration but a Configuration.getPropertiesStartingWith(CALLEDCLASS.class.getName()).

The addition of the required (but not yet implemented) getDefaultProperties() made the code uncompilable again

[rju]

author nils-christian -- Thu, 10 Nov 2011 17:16:06 +0100

Replying to [avh|comment:19]:
> I'd vote for b) as well.
>
> This would still allow to define pipe-and-filter structures manually (or to add additional plugins). And if the AnalysisController would provide a method like saveConfiguration(String filename) (which would also apply to configurations previously loaded using in instance of the meta-model), manually created configurations could even be serialized. Did you think about such method already? I think we need it any case. We should also think about whether the AnalyisController keeps an in-memory configuration model up-to-date or if it computes the model on a call to saveConfiguration by traversing through the pipe-and-filter architecture (collecting the plugins, its configurations, and the interconnections).

I took a note of this one. The method is still to be implemented, but the signature will be as follows:
_public final boolean saveToFile(final File file) _

[rju]

author nils-christian -- Thu, 10 Nov 2011 17:18:44 +0100

Replying to [jwa|comment:20]:
> I'm ok with b)
>
> Other point:
> * I improved the handling of the Configuration in all Readers and the DummyReader.
> * Now every plugin requires an implementation of protected abstract Properties getDefaultProperties(); to determine its default values.
> * Instead of the properties function getProperty use the specialized functions of Configuration (AbstractConfiguration)
> * On points that give a Configuration to the plugin (should only happen in the Constructor call), don't give the complete Configuration but a Configuration.getPropertiesStartingWith(CALLEDCLASS.class.getName()).
>
> The addition of the required (but not yet implemented) getDefaultProperties() made the code uncompilable again

But why do we now use the class Properties instead of Configuration or AbstractConfiguration?

[rju]

author Jan Waller -- Thu, 10 Nov 2011 17:22:12 +0100

It doesn't really matter in the case of getDefaultProperties(), the method is protected and only used internally to be added as the default values to a configuration object. The object that includes these defaultValues will never be directly accessible and doesn't need to be a Configuration, just a set of properties. But it really doesn't matter. Both would work perfectly fine.

In all other cases Configuration and the associated getters should be used.

[rju]

author nils-christian -- Thu, 10 Nov 2011 17:33:10 +0100

André:
There is still the question what to do with the "IMonitoringRecordConsumerPlugin". I can't add them to the analysis, as they don't have the necessary methods to connect them with the other plugins. Theoretically it seems to be possible for the classes who implement this interface to let them inherit from AbstractAnalysisPlugin. This would mean that the interface in question could be removed from the project, I think...
Another problem is the method addRecordConsumer. How should this method handle plugins and readers with more than one input/output?

[rju]

author André van Hoorn -- Thu, 10 Nov 2011 17:49:06 +0100

Replying to [nie|comment:24]:
> André:
> There is still the question what to do with the "IMonitoringRecordConsumerPlugin". I can't add them to the analysis, as they don't have the necessary methods to connect them with the other plugins. Theoretically it seems to be possible for the classes who implement this interface to let them inherit from AbstractAnalysisPlugin. This would mean that the interface in question could be removed from the project, I think...

The "IMonitoringRecordConsumerPlugin" interface is obsolete. We don't need it any more, since we handle all plugins in the same way. As you suggested, plugins which used to implement "IMonitoringRecordConsumerPlugin" should now extend AbstractAnalysisPlugin and provide the respective input ports with the right monitoring record types.

> Another problem is the method addRecordConsumer. How should this method handle plugins and readers with more than one input/output?

Also obsolete.

[rju]

author nils-christian -- Thu, 10 Nov 2011 17:51:15 +0100

Replying to [avh|comment:25]:
> Replying to [nie|comment:24]:
> > André:
> > There is still the question what to do with the "IMonitoringRecordConsumerPlugin". I can't add them to the analysis, as they don't have the necessary methods to connect them with the other plugins. Theoretically it seems to be possible for the classes who implement this interface to let them inherit from AbstractAnalysisPlugin. This would mean that the interface in question could be removed from the project, I think...
>
> The "IMonitoringRecordConsumerPlugin" interface is obsolete. We don't need it any more, since we handle all plugins in the same way.
>
> > Another problem is the method addRecordConsumer. How should this method handle plugins and readers with more than one input/output?
>
> Also obsolete.

Very well. Should I provide methods to connect plugins though? Something like:
connect(srcPlugin, dstPlugin, srcPortName, dstPortName)

[rju]

author André van Hoorn -- Thu, 10 Nov 2011 17:53:46 +0100

Replying to [nie|comment:26]:
> Replying to [avh|comment:25]:
> > Replying to [nie|comment:24]:
> > > André:
> > > There is still the question what to do with the "IMonitoringRecordConsumerPlugin". I can't add them to the analysis, as they don't have the necessary methods to connect them with the other plugins. Theoretically it seems to be possible for the classes who implement this interface to let them inherit from AbstractAnalysisPlugin. This would mean that the interface in question could be removed from the project, I think...
> >
> > The "IMonitoringRecordConsumerPlugin" interface is obsolete. We don't need it any more, since we handle all plugins in the same way.
> >
> > > Another problem is the method addRecordConsumer. How should this method handle plugins and readers with more than one input/output?
> >
> > Also obsolete.
>
> Very well. Should I provide methods to connect plugins though? Something like:
> connect(srcPlugin, dstPlugin, srcPortName, dstPortName)

I'd recommend not to do this. Connections should be detected when saving the model. If s.o. wants to connect plugins without using a configuration model, he/she should connect an output with an input port directly.

[rju]

author nils-christian -- Fri, 11 Nov 2011 16:46:05 +0100

Replying to [jwa|comment:20]:
> * On points that give a Configuration to the plugin (should only happen in the Constructor call), don't give the complete Configuration but a Configuration.getPropertiesStartingWith(CALLEDCLASS.class.getName()).

Can be done, but in the case of the FSReader the resulting configure-object is empty. This is because the config strings within the FSReader start with "kieker.analysis.reader.AbstractMonitoringReader" instead of the classname (and these strings are used for the example config file). Is this on purpose or probably a mistake?

[rju]

author nils-christian -- Fri, 11 Nov 2011 20:48:37 +0100

Replying to [avh|comment:25]:
> Replying to [nie|comment:24]:
> > André:
> > There is still the question what to do with the "IMonitoringRecordConsumerPlugin". I can't add them to the analysis, as they don't have the necessary methods to connect them with the other plugins. Theoretically it seems to be possible for the classes who implement this interface to let them inherit from AbstractAnalysisPlugin. This would mean that the interface in question could be removed from the project, I think...
>
> The "IMonitoringRecordConsumerPlugin" interface is obsolete. We don't need it any more, since we handle all plugins in the same way. As you suggested, plugins which used to implement "IMonitoringRecordConsumerPlugin" should now extend AbstractAnalysisPlugin and provide the respective input ports with the right monitoring record types.
>
> > Another problem is the method addRecordConsumer. How should this method handle plugins and readers with more than one input/output?
>
> Also obsolete.

What is about the interface IMonitoringRecordReceiver? In nearly every case I have to replace this with AbstractAnalysisPlugin anyway. But if the interface is removed, the following interfaces would also have to be removed, as they inherit from this interface:
IPipeReader, IMonitoringWriter, IWriterController,...

[rju]

author André van Hoorn -- Fri, 11 Nov 2011 20:57:57 +0100

IMonitoringRecordReceiver should not be removed. You're right, that it will no longer be used in Kieker.Analysis. However, it is still used in Kieker.Monitoring.

[rju]

author André van Hoorn -- Fri, 11 Nov 2011 21:00:48 +0100

Replying to [avh|comment:30]:
> IMonitoringRecordReceiver should not be removed. You're right, that it will no longer be used in Kieker.Analysis. However, it is still used in Kieker.Monitoring.

I remember that it's also used by the FS reader (which means it is being used in Kieker.Analysis).

[rju]

author Jan Waller -- Sat, 12 Nov 2011 00:35:07 +0100

Replying to [nie|comment:28]:
> Can be done, but in the case of the FSReader the resulting configure-object is empty. This is because the config strings within the FSReader start with "kieker.analysis.reader.AbstractMonitoringReader" instead of the classname (and these strings are used for the example config file). Is this on purpose or probably a mistake?

public static final String CONFIG_INPUTDIRS = FSReader.class.getName() + ".inputDirs";
public static final String CONFIG_ONLYRECORDS = FSReader.class.getName() + ".readOnlyRecordsOfType";

is what the code says?

[rju]

author nils-christian -- Sat, 12 Nov 2011 12:17:49 +0100

Replying to [jwa|comment:32]:
> Replying to [nie|comment:28]:
> > Can be done, but in the case of the FSReader the resulting configure-object is empty. This is because the config strings within the FSReader start with "kieker.analysis.reader.AbstractMonitoringReader" instead of the classname (and these strings are used for the example config file). Is this on purpose or probably a mistake?
>
> {{

{ > public static final String CONFIG_INPUTDIRS = FSReader.class.getName() + ".inputDirs"; > public static final String CONFIG_ONLYRECORDS = FSReader.class.getName() + ".readOnlyRecordsOfType"; > }

}}
> is what the code says?

Yes
- by now. Beforehand it said AbstractMonitoringReader.class.getName(). I modified it in order to solve the problem temporarily. I just wanted to know whether this is a valid fix or not.

[rju]

author Jan Waller -- Sat, 12 Nov 2011 12:30:15 +0100

Yes it is. If it was otherwise, it was a bug
Thanks for fixing

[rju]

author nils-christian -- Mon, 14 Nov 2011 19:15:06 +0100

Just want to remind: Not all plugins can be configured via a Configuration-instance, like for example some intern Helper-Delegate-Classes (e.g. RecordDelegationPlugin2) as they get an instance during constructing.
For the moment I will not try to enforce the Configuration-Constructor and will instead make a note not to change the visibility modificator to "public" within the Java-Docs. Keep this in mind when you plan to extract these classes or something.

[rju]

author Jan Waller -- Wed, 23 Nov 2011 11:24:58 +0100

Replying to [nie|comment:35]:
> Just want to remind: Not all plugins can be configured via a Configuration-instance, like for example some intern Helper-Delegate-Classes (e.g. RecordDelegationPlugin2) as they get an instance during constructing.
> For the moment I will not try to enforce the Configuration-Constructor and will instead make a note not to change the visibility modificator to "public" within the Java-Docs. Keep this in mind when you plan to extract these classes or something.

As discussed yesterday, all plugins will only have a configuration constructor.

[rju]

author Jan Waller -- Thu, 24 Nov 2011 13:50:30 +0100

see also KIEKER-1047 Done

[rju]

author nils-christian -- Sun, 27 Nov 2011 18:20:35 +0100

There are some classes left which are problematic with regard to the configuration-constructor and the persisting of the configuration. In most cases this is because the classes would get an instance of SystemModelRepository. Is it somehow possible to initialize them without this instance?
Following classes are affected:

• RealtimeReplayDistributor
• AbstractTraceAnalysisPlugin
• ExecutionRecordTransformationFilter
• AggregatedCallTreePlugin
• ComponentDependencyGraphPluginAllocation
• ComponentDependencyGraphPluginAssembly
• ContainerDependencyGraphPlugin
• ExecutionRecordTransformationFilter
• ExecutionTraceWriterPlugin
• InvalidExecutionTraceWriterPlugin
• MessageTraceRepositoryPlugin
• MessageTraceWriterPlugin
• OperationDependencyGraphPluginAllocation
• OperationDependencyGraphPluginAssembly
• SequenceDiagramPlugin
• TraceCallTreePlugin
• TraceEquivalenceClassFilter
• TraceReconstructionFilter

[rju]

author nils-christian -- Wed, 14 Dec 2011 16:03:20 +0100

A little problem occurred while implementing the new pipes&filters structure with annotations, due to Java reflection:
To remind again: The class AbstractPlugin is responsible for the storage of the registered methods, for the connecting of the plugins and the calling of the corresponding methods (via deliver).
The calling is a little bit problematic. The visibility modifiers / access restriction cannot be annulled. This means that the method to be invoked have to be public, but, which is even more important, must also be visible for the AbstractPlugin. This means that classes like the FilesystemLogReplayer are problematic, because they use intern (respectively package-visible) classes (like in this case the RecordDelegationPlugin), which are not visible for AbstractPlugin and whose methods can therefore not be called.

The question is: What to do now? In my opinion the best approach would be to extract and generalize the delegator classes in question sooner or later. What do you think?

[rju]

author Jan Waller -- Wed, 14 Dec 2011 16:18:23 +0100

So, these classes should use public instead of private, package or protected?

Alternatively, call setAccessible(true) to ignore the restrictions. In case of the AnalysisComponent, we assume no SecurityManager preventing the access. (It is already used in AbstractMonitoringRecord.)

[rju]

author nils-christian -- Thu, 15 Dec 2011 17:26:13 +0100

Replying to [jwa|comment:40]:
> So, these classes should use public instead of private, package or protected?
>
> Alternatively, call setAccessible(true) to ignore the restrictions. In case of the AnalysisComponent, we assume no SecurityManager preventing the access. (It is already used in AbstractMonitoringRecord.)

That solved the problem
- thank you.

[rju]

author André van Hoorn -- Tue, 24 Jan 2012 17:04:32 +0100

Just as a reference, the class diagrams of the current analysis configuration meta model:

• [.pdf|/raw-attachment/ticket/330/AnalysisMetaModel-20120124.pdf]
• [.png|/raw-attachment/ticket/330/AnalysisMetaModel-20120124.png]

Unable to render embedded object: File (/raw-attachment/ticket/330/AnalysisMetaModel-20120124.png) not found.