xGSN - Pushing data to LSM

[premjayaraman]

In the two day hackathon at IoT London, we discovered the following issues

1. xGSN does not support ability to push string data type while LSM supports it. The corresponding method has to be implemented
2. The major issue is how xGSN handles sensor data. a. For a sensor with 3 parameters (temp, humidity, pressure), when a observation is stored, the three values as observations of different value instead of observation that are all linked. To test it, if a query for a observation is made, the related observations that were made by same sensor cannot be recovered. Relation here is the connection between the 3 value that a sensor observes at a instance.

Please look into LSMExporter where each observation is sent separately with a new observation ID, instead of the same observation ID.

[jpcik]

Yes one observation is created for each one of the properties. They are 3 different observations, if we follow the SSN ontology. Otherwise how do we know what value belongs to which obsevedProperty?

According to SSN, it is the observation that should point to the observed property, (and not the observation value). Moreover there should be a triple linking the Observation and ObservationValue.

Now to link the 3 observations, all of them should point to the same Sensor URL (e.g. :sensor1), through the observedBy property.: e.g. consider these triples: :obs1 a ssn:Observation :obs1 ssn:observedProperty openiot:air-temperature :obs1 ssn:observedBy :sensor1

At least this is my understanding following the SSN documentation.

[premjayaraman]

Hi Jean-Paul,

Thanks for the clarification.

Let me try to write down what we wanted to achieve.

Let’s assume we have a human sensor with name, email. This human sensor keeps publishing this data as a stream e.g. Prem, pre@prem.com Jean-Paul, jp@jp.com and so on.

Now when the data gets into the triple store each observation Prem and prem@prem.com has a observation id. Similarly Jean-paul and jp@jp.com

When I query “get the observerd value of this sensor” using the observedBy value, I will get 4 values. But how will I know that which name and email are pairs (groups)? In summary I would like some form of grouping.

Not sure if I expalined it well. Tried my best ☺.

Thanks and Regards prem

From: Jean-Paul Calbimonte [mailto:notifications@github.com] Sent: Wednesday, 23 July 2014 1:41 AM To: OpenIotOrg/openiot Cc: Jayaraman, Prem (DP&S, Acton) Subject: Re: [openiot] xGSN - Pushing data to LSM (#71)

Yes one observation is created for each one of the properties. They are 3 different observations, if we follow the SSN ontology. Otherwise how do we know what value belongs to which obsevedProperty?

According to SSN, it is the observation that should point to the observed property, (and not the observation value). Moreover there should be a triple linking the Observation and ObservationValue.

Now to link the 3 observations, all of them should point to the same Sensor URL (e.g. :sensor1), through the observedBy property.: e.g. consider these triples: :obs1 a ssn:Observation :obs1 ssn:observedProperty openiot:air-temperature :obs1 ssn:observedBy :sensor1

At least this is my understanding following the SSN documentation.

— Reply to this email directly or view it on GitHubhttps://github.com/OpenIotOrg/openiot/issues/71#issuecomment-49756760.

[jpcik]

Ok, in that case the grouping is done using the time right? You want those observations captured at the same time. I don't think the purpose of the observation is to group things, but I may be wrong. In that case an Observation would be a group of observed values (of different kind) measured at the same time, but I don't think it works that way.

Again this is my interpretation of SSN, there could be other opinions.

[premjayaraman]

Thanks for the clarification Jean-Paul.

Like I said, I myself am no expert in this.

When I discussed with Hoan and showed him the code as we were trying to retrieve pairs of name and email id, he suggested observation id should be same.

@Hoan, could you please provide your view.

thanks prem

---

From: Jean-Paul Calbimonte [notifications@github.com] Sent: Wednesday, July 23, 2014 7:06 PM To: OpenIotOrg/openiot Cc: Jayaraman, Prem (DP&S, Acton) Subject: Re: [openiot] xGSN - Pushing data to LSM (#71)

Ok, in that case the grouping is done using the time right? You want those observations captured at the same time. I don't think the purpose of the observation is to group things, but I may be wrong. In that case an Observation would be a group of observed values (of different kind) measured at the same time, but I don't think it works that way.

Again this is my interpretation of SSN, there could be other opinions.

— Reply to this email directly or view it on GitHubhttps://github.com/OpenIotOrg/openiot/issues/71#issuecomment-49849569.

[hylkevds]

Looking at the SSN ontology using Protege, it seems that an Observation has to have exactly 1 observed property. Unfortunately that means that we can't use the observation to group different measurements. What is the granularity of the observation Sampling/Result Time? Is that in second or milliseconds?

If milliseconds then using the time to group observations of a sensor could work. We could even add a small check to see if a sensor already has an observation for that millisecond, and increase the time by 1ms to avoid collisions.

If it's seconds then we need a different solution, since it's quite likely that some sensors produce several results in a given second.

[jpcik]

But for one sensor, time should be enough right?

time sensor prop value 1 s1 temp 35.5 1 s1 hum 80 1 s2 temp 36 1 s3 humid 70 2 s2 temp 33 2 s3 humid 80

regardless of the time granularity we can group all temp,humid values per time per sensor.

2014-08-01 14:40 GMT+02:00 Hylke van der Schaaf notifications@github.com:

Looking at the SSN ontology using Protege, it seems that an Observation has to have exactly 1 observed property. Unfortunately that means that we can't use the observation to group different measurements. What is the granularity of the observation Sampling/Result Time? Is that in second or milliseconds?

If milliseconds then using the time to group observations of a sensor could work. We could even add a small check to see if a sensor already has an observation for that millisecond, and increase the time by 1ms to avoid collisions.

If it's seconds then we need a different solution, since it's quite likely that some sensors produce several results in a given second.

— Reply to this email directly or view it on GitHub https://github.com/OpenIotOrg/openiot/issues/71#issuecomment-50879314.

jp

[premjayaraman]

I think going by timestamp is the best way forward.

I looked at GSN code

The date is generated for each field. If we use a common timestamp, will this cause an issue?

/prem

public void dataAvailable(String inputStreamName, StreamElement data) {

    Long t = data.getTimeStamp();
    for (int i = 0; i < fields.size(); i++) {
        String field = fields.get(i);
        Double v = (Double) data.getData(field);
        Date d = new Date(t);
        String fieldName = data.getFieldNames()[i];
        logger.debug(fieldName + " : t=" + d + " v=" + v);

        if (!allow_nulls && v == null)
            return; // skipping null values if allow_nulls flag is not st to true

        if (publish_to_lsm) {
            LSMRepository.getInstance().publishSensorDataToLSM(sensorName, fieldName, v, d);
        }

        //dataProduced(data);

    }

}

Prem Jayaraman Postdoctoral Research Fellow CSIRO Computational Informatics E prem.jayaraman@csiro.au T +61 2 6216 7122 Post: GPO Box 664, Canberra, ACT 2601, Australia

PLEASE NOTE The information contained in this email may be confidential or privileged. Any unauthorised use or disclosure is prohibited. If you have received this email in error, please delete it immediately and notify the sender by return email. Thank you. To the extent permitted by law, CSIRO does not represent, warrant and/or guarantee that the integrity of this communication has been maintained or that the communication is free of errors, virus, interception or interference. Please consider the environment before printing this email.

From: Jean-Paul Calbimonte [mailto:notifications@github.com] Sent: Saturday, 2 August 2014 6:57 PM To: OpenIotOrg/openiot Cc: Jayaraman, Prem (DP&S, Acton) Subject: Re: [openiot] xGSN - Pushing data to LSM (#71)

But for one sensor, time should be enough right?

time sensor prop value 1 s1 temp 35.5 1 s1 hum 80 1 s2 temp 36 1 s3 humid 70 2 s2 temp 33 2 s3 humid 80

regardless of the time granularity we can group all temp,humid values per time per sensor.

2014-08-01 14:40 GMT+02:00 Hylke van der Schaaf notifications@github.com<mailto:notifications@github.com>:

Looking at the SSN ontology using Protege, it seems that an Observation has to have exactly 1 observed property. Unfortunately that means that we can't use the observation to group different measurements. What is the granularity of the observation Sampling/Result Time? Is that in second or milliseconds?

If milliseconds then using the time to group observations of a sensor could work. We could even add a small check to see if a sensor already has an observation for that millisecond, and increase the time by 1ms to avoid collisions.

If it's seconds then we need a different solution, since it's quite likely that some sensors produce several results in a given second.

— Reply to this email directly or view it on GitHub https://github.com/OpenIotOrg/openiot/issues/71#issuecomment-50879314.

jp

— Reply to this email directly or view it on GitHubhttps://github.com/OpenIotOrg/openiot/issues/71#issuecomment-50957959.

[hylkevds]

Yes, if the time granularity is high enough, then grouping by time should work just fine.

If time granularity is not high enough, then the time value night no longer be unique per-sensor-per-measurement.

On 02/08/14 10:57, Jean-Paul Calbimonte wrote:

But for one sensor, time should be enough right?

time sensor prop value 1 s1 temp 35.5 1 s1 hum 80 1 s2 temp 36 1 s3 humid 70 2 s2 temp 33 2 s3 humid 80

regardless of the time granularity we can group all temp,humid values per time per sensor.

2014-08-01 14:40 GMT+02:00 Hylke van der Schaaf notifications@github.com:

Looking at the SSN ontology using Protege, it seems that an Observation has to have exactly 1 observed property. Unfortunately that means that we can't use the observation to group different measurements. What is the granularity of the observation Sampling/Result Time? Is that in second or milliseconds?

If milliseconds then using the time to group observations of a sensor could work. We could even add a small check to see if a sensor already has an observation for that millisecond, and increase the time by 1ms to avoid collisions.

If it's seconds then we need a different solution, since it's quite likely that some sensors produce several results in a given second.

— Reply to this email directly or view it on GitHub https://github.com/OpenIotOrg/openiot/issues/71#issuecomment-50879314.

jp

— Reply to this email directly or view it on GitHub https://github.com/OpenIotOrg/openiot/issues/71#issuecomment-50957959.

Dr. Hylke van der Schaaf Fraunhofer Institute of Optronics, System Technologies and Image Exploitation IOSB Fraunhoferstr. 1, 76131 Karlsruhe, Germany Phone +49 721 6091 613 hylke.vanderschaaf@iosb.fraunhofer.de www.iosb.fraunhofer.de