reasoner should update FDs expected QAs levels from QA Observers

[chcorbato]

This is needed to integrate @JWijkhuizen 's Quality Models

Option 1: extend current interface (diagnostics message) we need to extend the info about QA send through diagnostics so that we can indicate which FD needs to be updated

Option2: define a new message

The current method to process the QA input is subscription-based: https://github.com/tud-cor/mc_mros_reasoner/blob/400ac0676e72d5aeee4de054db4ec35957007e33/mros1_reasoner/scripts/mros1_reasoner.py#L199 calls:

https://github.com/tud-cor/mc_mros_reasoner/blob/400ac0676e72d5aeee4de054db4ec35957007e33/mros1_reasoner/scripts/mros1_reasoner.py#L220-L236

[chcorbato]

@JWijkhuizen @darkobozhinoski what information does the QA Observer needs from the KB to annotate the QA value sent to metacontrol (e.g. name of the configuration?)

[JWijkhuizen]

I think only the name, or nothing at all. The QA Observer can just send the updated QA values of all the configurations which are in the quality model.

[chcorbato]

I think only the name, or nothing at all. The QA Observer can just send the updated QA values of all the configurations which are in the quality model.

We need at least:

• the name of the FD (configuration), if the mros_reasoner only gets a number there is no way it can know to what configuration to associate that number
• the name of the QA, so that we store it for safety or for a different QA

[JWijkhuizen]

Yes that is what the QA observer publishes in the /diagnostics topic, right? The name of the configuration, the name of the QA and the predicted QA value are all outputs of the quality model. I think the QA observer only needs the name of the current configuration, so it can output the prediction of all the other configurations(So it knows which configuration to skip). To avoid that the predicted QA value of the current configuration overrides the measured QA value.

Or am I not understanding your question?

[chcorbato]

@JWijkhuizen do you have a different equation for each configuration? How do you compute the QA value expected for a particular configuration?

[JWijkhuizen]

Yes right now I have a different model for each configuration. I still have to research the possibility to have them all in the same model. What do you think is easier for the implementation?

There are two observers which are monitoring the environment, one for Narrowness and one for Obstacle Density. So these values are updated each timestep (I think with 10Hz by default). These values together with their derivatives are used by the quality model to predict the QA's for the non-used configurations. Each configuration has its own scikit-learn object, which contains the safety model. The input for this object is: the environment metrics and their derivatives, the output is the expected QA (safety) for that configuration.

I think I will make an extra observer, in which the safety models are loaded. This observer subscribes on the /diagnostics topic to receive the environment metrics. And will publish the predicted QA's of all the configurations it knows (minus the currently used configuration I think?) to the /diagnostics topic again, so the mros_reasoner can update them in the knowledge base.

Please tell me if you have a better idea for the implementation of the quality model.

[chcorbato]

Is each model a different Python/Tensorflow/Pytorch object? Please document that and also your previous answer in detail (or point to that par of your document if you already have it), so that anyone can understand how it works for theoretically and implementation wise.

[JWijkhuizen]

I do not have a detailed documentation yet in my thesis. I will work on that in the coming days I think. I updated my previous comment with more information

[chcorbato]

Yes right now I have a different model for each configuration. I still have to research the possibility to have them all in the same model. What do you think is easier for the implementation?

There are two observers which are monitoring the environment, one for Narrowness and one for Obstacle Density. So these values are updated each timestep (I think with 10Hz by default). These values together with their derivatives are used by the quality model to predict the QA's for the non-used configurations. Each configuration has its own scikit-learn object, which contains the safety model. The input for this object is: the environment metrics, and their derivatives and the output is the expected QA (safety) for that configuration.

I think I will make an extra observer, in which the safety models are loaded. This observer subscribes on the /diagnostics topic to receive the environment metrics. And will publish the predicted QA's of all the configurations it knows (minus the currently used configuration I think?) to the /diagnostics topic again, so the mros_reasoner can update them in the knowledge base.

Please tell me if you have a better idea for the implementation of the quality model.

I can not provide a solid answer atm, but my first suggestion would be to:

• Implement one single ROS node that is a Quality Observer node for safety. This would be an alternative Observer node to the one @darkobozhinoski has already implemented, so I hope he can guide you.
• This node will publish QA safety values for each configuration, and the reasoner will subscribe to it.
• Internally, this safety Observer maintains a list of known configurations, and computes the expected safety for each of them using the different scikit-learn objects.
• Your narrowness and density observers (are they different ROS nodes, btw?) can be internal methods (this would probably more efficient than have them as separate nodes, if update speed is an issue)
• You have a sync update that:
  • gets the latest info from the environment (which info is this? what topics does your narrowness and density observers subscribe to?)
  • computes density and narrowness
  • uses the previous values and the scikit-learn objects to compute the safety values for each configuration
  • publishes the diagnostics messages for the mros_reasoner

[JWijkhuizen]

Right now I have 3 Observers implemented using the rosgraph_monitor, the same method as the safety observer that Darko implemented. (And a 4th one for Performance, but this one is not relevant for this issue) Observers:

• Safety
• Narrowness
• Obstacle Density Narrowness used the /front/scan Laserscan topic. Obstacle Density uses the local_costmap topic.

I think that it will be easier to have two observers:

• Safety Observer
• Safety prediction Observer The first Safety observer is monitoring the current safety QA, almost exactly the same as Darko uses now. The current safety level is published The second Safety prediction Observer will monitor the environment to determine the Narrowness and Obstacle Density environment metrics, and use the scikit-learn objects to predict the safety QA for all (other) configurations. So the environment metrics are not published, but only used internally in the observer. The predicted safety value for each configuration is published.

However, if I want to use the rosgraph_monitor TopicObserver class, I think I need to have a different Safety predicted Observer for each configuration(so load this observer multiple times, for each configuration one time). Since the Observer needs to publish a different message for each configuration. Or I need to use a new message type to include the predicted values for all configurations in one message. I think a different message for each configuration is preferred.

[darkobozhinoski]

@JWijkhuizen , @chcorbato , I am wondering if it is needed to observe the values of all configurations at all time. Initially, the observer role was to investigate the characteristics of the current configuration, while the role of the reasoner is to compute if there should be a switch. Taking that in consideration, is it possible to publish only the value of the current configuration, while the environmental metrics are encoded in the ontology? There can be a separate message type and separate environment observers that update that knowledge. In this case, every-time the reasoner runs, the reasoner obtains new values for safety using the scikit-learn objects that are updated in the Knowledge base. My belief is that the job of the observer should be only to report values about the current configuration, while the reasoner should handle the rest. Please let me know if I misunderstood something.

[chcorbato]

the observer role was to investigate the characteristics of the current configuration

what do you mean by "characteristics"?

is it possible to publish only the value of the current configuration

what value?

while the environmental metrics are encoded in the ontology?

• atm the OWL reasoner cannot use the density and narrowness metrics for anything, it can only reason using safety, performance and energy values
• for the reason above, I think the use of density and narrowness metrics to compute safety is better done outside of mros_reasoner, in an observer
• the question is how to architect that observer:
• a) single node
• b) 3 nodes: density observer and narrowness observers + safety observer that uses the narrowness and density values published by the others

There can be a separate message type and separate environment observers that update that knowledge.

I agree, the narrowness and density metrics are semantically different from safety, because safety is domain-independent. So if we separate the metacontrol so that mros_reasoner is completely domain-independent, it can handle safety, and have an interface to receive that information, but not narrowness or density, that are dealt with Observers (note that these are not (System) Quality Attribute Observers, but "Context" Observers)

My belief is that the job of the observer should be only to report values about the current configuration, while the reasoner should handle the rest.

This is hard to agree or disagree, since "the rest" is vague and we are on the process of defining it. Can you be more explicit?

[JWijkhuizen]

I think it makes sense to have the observers only observe, and have a node in the analyse part (from the mape-k point of view) predict the safety levels of the other configurations using the environment measures from the observers. This node in the analyse part can either be a new node, or this can be the mros_reasoner(I think only 1 or 2 extra functions are needed).

But, maybe it is not desired to have something like this implemented in the mros_reasoner, which I can understand. In that case a simple single extra node can do the work. This node will predict the QA values of the other configurations, and the reasoner will update these values in the knowledge base.

In this case, every-time the reasoner runs, the reasoner obtains new values for safety using the scikit-learn objects that are updated in the Knowledge base.

Updating the predicted safety levels in the knowledge base only once before a new reasoning cycle start makes sense to me. This can possibly also save some computational effort.

[darkobozhinoski]

the observer role was to investigate the characteristics of the current configuration

what do you mean by "characteristics"?

What I mean is investigating runtime values for safety, energy etc. In this context, it is the computed value for safety between 0 and 1 which is different than the expected value.

is it possible to publish only the value of the current configuration

what value?

What I mean here is the quality value computed at runtime (in this context safety). This means using the same interface of the observers as the one we have it now.

while the environmental metrics are encoded in the ontology?

• atm the OWL reasoner cannot use the density and narrowness metrics for anything, it can only reason using safety, performance and energy values
• for the reason above, I think the use of density and narrowness metrics to compute safety is better done outside of mros_reasoner, in an observer

I was thinking here in a long term the values of each configuration can be updated in the reasoner before the new value for safety is computed [this means introducing conext model as the one I described in MROS]. In Jaspers system, the context model is modeled through density and narrowness. However, I understand that this not might be applicable now, but I think it is a better solution for long-term, as you might have two types of observers: context observers and quality observers and both report only the current state of the system.

• the question is how to architect that observer:

  • a) single node
  • b) 3 nodes: density observer and narrowness observers + safety observer that uses the narrowness and density values published by the others

There can be a separate message type and separate environment observers that update that knowledge. I agree, the narrowness and density metrics are semantically different from safety, because safety is domain-independent. So if we separate the metacontrol so that mros_reasoner is completely domain-independent, it can handle safety, and have an interface to receive that information, but not narrowness or density, that are dealt with Observers (note that these are not (System) Quality Attribute Observers, but "Context" Observers)

My belief is that the job of the observer should be only to report values about the current configuration, while the reasoner should handle the rest.

This is hard to agree or disagree, since "the rest" is vague and we are on the process of defining it. Can you be more explicit?

I hope this makes things more clear. The general idea is to introduce Context as part of the ontologies which I think is more beneficial on a long-run by separating context and quality observers.

[chcorbato]

I think it makes sense to have the observers only observe, and have a node in the analyse part (from the mape-k point of view) predict the safety levels of the other configurations using the environment measures from the observers. This node in the analyse part can either be a new node,

Sounds reasonable to me. But note that MAPE-K does not suggest that each step should be performed by a separate node. If we want to decouple the steps (e.g. because we can exchange different monitors, and different analyzers) then one way to implement that decoupling is using separate ROS nodes, as you suggest. So, if the narrowness and density metrics can be potentially used by processes different than the safety observer, than you may want to decouple them by splitting them in separate nodes.

or this can be the mros_reasoner(I think only 1 or 2 extra functions are needed).

But, maybe it is not desired to have something like this implemented in the mros_reasoner, which I can understand. In that case a simple single extra node can do the work. This node will predict the QA values of the other configurations, and the reasoner will update these values in the knowledge base.

This is the preferred case, do not add any observer logic or semantics in the mros_reasoner, to keep it domain-independent, as explained above

In this case, every-time the reasoner runs, the reasoner obtains new values for safety using the scikit-learn objects that are updated in the Knowledge base.

Updating the predicted safety levels in the knowledge base only once before a new reasoning cycle start makes sense to me. This can possibly also save some computational effort.

Maybe it should be upon request, as @darkobozhinoski mentioned that we only need to update predicted/expected QA values if a need for reconfiguration has already been detected. @JWijkhuizen what is the reconfiguration trigger in your scenarios?

[JWijkhuizen]

Maybe it should be upon request, as @darkobozhinoski mentioned that we only need to update predicted/expected QA values if a need for reconfiguration has already been detected. @JWijkhuizen what is the reconfiguration trigger in your scenarios?

Upon request is maybe a good solution yes. For now I am using the same reconfiguration trigger as you do, so when the non-functional requirements are violated.

Where will this request be processed? An extra node I think? This node will have the quality models loaded, and can use the observed environment measures to predict the QA levels of the other configurations. The environment observers will monitor the environment and publish these measures all the time, so not only upon request.

[chcorbato]

Maybe it should be upon request, as @darkobozhinoski mentioned that we only need to update predicted/expected QA values if a need for reconfiguration has already been detected. @JWijkhuizen what is the reconfiguration trigger in your scenarios?

Upon request is maybe a good solution yes. For now I am using the same reconfiguration trigger as you do, so when the non-functional requirements are violated.

Where will this request be processed? An extra node I think?

If it is upon request, that node could implement a service that the mros_reasoner calls to received updated info about the QAs of the configurations.

[JWijkhuizen]

I now have created the node for this. This node:

• Has a /load_model service, to load the models which are located in the metacontrol_move_base_configurations repository
• Listens to the /diagnostics topic for the updates about the Environment metrics.
• Has a /qa_update service, so the reasoner can request updates of the QA's of all the configurations. https://github.com/JWijkhuizen/mros_quality_models

The only things that needs to be done are the modifications of the mros_reasoner, and the interface (the service type of the /qa_update service is not yet defined). @chcorbato can you help me with this?

[chcorbato]

The only things that needs to be done are the modifications of the mros_reasoner, and the interface (the service type of the /qa_update service is not yet defined). @chcorbato can you help me with this?

I am working on fixing issues with the reasoner, I can come back to this as soon as that is fixed, hopefully by Friday.

[JWijkhuizen]

What I think that should be in the service call:

• The mros_reasoner should ask for an update of a specific QA
• Maybe also the name of the current config (so the QA of this config is not updated, this is what the quality observers do already)

What is returned: for each config with a loaded QA model (Maybe minus the current config):

• The config name and the new QA value

What the QA updater node already knows:

• The environment measures (it listens to the /diagnostics topic, this is where the environment observers publish their metric updates)
• A list of the configs with the loaded QA models

[gavanderhoorn]

Suggestion: don't use services for potentially long running operations.

There is no way to cancel a service invocation, nor can you be updated about the status of the operation without the service actually returning/completing.

Use actions.

[JWijkhuizen]

I will try to implement this service/action today/tomorrow. @chcorbato can you help me with the best location for this call? Can I place this in the timer_cb function?

[chcorbato]

@JWijkhuizen are you implementing it as an action?

If this is something that happens only from time to time, I think it should be triggered by the QA Observer (no need for the mros_reasoner node to keep checking periodically)

If using actions (recommended) the I think the action server needs to be the mros_reasoner node, and the QA Observer the client. In that case you only need to implement in the mros_reasoner node the action callback (and maybe at some point the actio feedback publisher, but not sure we need it now). So no need to add code in the timer_cb function as far as I can think of. Beware to use a lock when performing operations on the KB, to avoid conflicts with the MAPE loop in the timer_cb

[JWijkhuizen]

It should happen every time before the search for a new configuration, or before every reasoning cycle. How does the QA observer know when this will happen?

If I understand the code correctly, the timer_cb is the reasoning cycle, right? So this happens once every 2 seconds? timer = rospy.Timer(rospy.Duration(2.), timer_cb)

The QA Observer is able to send the predicted QA values at the same frequency as the environment metrics observe, which is 10Hz by default. I dont think this is desired. I think it is better to only update the predicted QA values when necessary, so when the mros_reasoner searches for a new configuration or before every reasoning cycle.

[chcorbato]

It should happen every time before the search for a new configuration, or before every reasoning cycle. How does the QA observer know when this will happen?

Good point, sorry I though the node detecting the change in context should trigger it, but that is indirectly triggered by the need of reconfiguration, correct?

I thing we should not do this when not necessary, so only before the mros_reasoner searches for a new configuration seems a good option to me.

[JWijkhuizen]

Good point, sorry I though the node detecting the change in context should trigger it, but that is indirectly triggered by the need of reconfiguration, correct?

The changes (and the measurements) of the environment are continuous, so every timestep the environment can be different.

I thing we should not do this when not necessary, so only before the mros_reasoner searches for a new configuration seems a good option to me.

@chcorbato Where in the mros_reasoner do you recommend to implement this?

[chcorbato]

Good point, sorry I though the node detecting the change in context should trigger it, but that is indirectly triggered by the need of reconfiguration, correct?

The changes (and the measurements) of the environment are continuous, so every timestep the environment can be different.

But this relates to updating the QA models in the KB, not necessarily trigger reconfiguration. So only update the KB when we use the KB to make a decision, in our case update the QA models for the FDs (aka configurations) when we need to select one.

I thing we should not do this when not necessary, so only before the mros_reasoner searches for a new configuration seems a good option to me.

@chcorbato Where in the mros_reasoner do you recommend to implement this?

We are about to merge a big refactoring of mros_reasoner, so that there is a more clear code structure. @marioney shall we do the merge? DO you have time today and tomorrow to support @JWijkhuizen trying the new version?

[JWijkhuizen]

I dont think I need a lot of help if the code structure is more clear @marioney .

Is there a stable version that I already can use? I really want to start soon with my final experiments. @chcorbato

[chcorbato]

New version can be installed following these instructions @JWijkhuizen @shreyashpalande @karelflere : https://github.com/rosin-project/metacontrol_sim courtesy of @marioney 👏 but I am to blame for any issue with the install instructions