search

urbanobservatory / standards

Standards and schema documentation for the observatories programme
2 stars 0 forks source link

Common Vocabulary - observedProperty #23

Open SiBell opened 4 years ago

SiBell commented 4 years ago

Intro

Following previous discussions we appear to have come to the agreement that our observations MUST have an observedProperty field. This term has come from the Semantic Sensor Network ontology.

Example observation:

{
  "madeBySensor": "thermistor-abc",
  "hasResult": {
    "value": 21.2,
    "unit": "deg_c"
  },
  "observedProperty": "AirTemperature",
  "hasFeatureOfInterest": "EarthAtmosphere",
  "discipline": ["Meteorology"]
}

Aim

To agree on a common list of ObservableProperty's.

Why

Let's say all 6 urban observatories have sensors capable of measuring air temperature. The first observatory tags all its observations with {"observedProperty": "AirTemperature"}. Their end-users can filter the observations with a URL query string parameter, e.g:

GET https://api.urbanobservatory.com/observations?observedProperty=AirTemperature

Wouldn't it be a shame it the second observatory used a different term for air temperature, e.g. {"observedProperty": "temperature"} or {"observedProperty": "thermodynamic-temperature"}. All of a sudden it becomes very difficult for end-users to pull air temperature observations from multiple observatories. Not only would they need to change the URL query string parameter value, but they may not even be sure if AirTemperature is even equivalent to thermodynamic-temperature.

Tasks

I'll start an initial list below. Just comment below if you think any need modifying or removing, likewise if you have any more that you'd like to add.

Wherever possible please try to reuse or reference properties already defined in other vocabularies.

SiBell commented 4 years ago
observedProperty notes units equivalent to
AirTemperature Could be outside, inside, in a container, etc. "Dry bulb" temperature. °C (unit:DEG_C) CF:air_temperature, subclass of ThermodynamicTemperature?
RelativeHumidity % (unit:PERCENT) quantityKind:RelativeHumidity, CF:relative_humidity
AirPressure hPa (unit:HectoPA) CF:air_pressure
AirPressureAtMeanSeaLevel hPa (unit:HectoPA) CF:air_pressure_at_mean_sea_level
WindSpeed m/s (unit:M-PER-SEC) CF:wind_speed
WindDirection Direction the wind is coming FROM ° (unit:DEG) CF:wind_from_direction
MaxWindSpeed a.k.a wind gust speed m/s (unit:M-PER-SEC) CF:wind_speed_of_gust
MaxWindDirection a.k.a wind gust direction. Direction the wind is coming FROM ° (unit:DEG)
PrecipitationDepth tipping bucket rain gauges typically convert precipitation volume into a depth based on the area of the gauge. mm (unit:MilliM)
PrecipitationRate mm/hr
GlobalRadiation Commonly measured by a Pyranometer. Needs more detail as there's various measures of solar irradiance. W/m2 (unit:W-PER-M2)
Co2Concentration ppm is good, because everyone is familiar with the keeling curve which uses ppm ppm (unit:PPM)
NoConcentration ppb (unit:PPB)
No2Concentration ppb (unit:PPB)
Pm1MassConcentration Using same units as national air quality objectives μg/m3 CF:mass_concentration_of_pm1_ambient_aerosol_particles_in_air
Pm2p5MassConcentration PM2.5 μg/m3 CF:mass_concentration_of_pm2p5_ambient_aerosol_particles_in_air
Pm4MassConcentration PM4 μg/m3
Pm10MassConcentration PM10 μg/m3 CF:mass_concentration_of_pm10_ambient_aerosol_particles_in_air
NoiseLevel dB (unit:DeciB) quantityKind:SoundExposureLevel?
SiBell commented 4 years ago

We need to decide how we want to handle Average, Max and Min values.

Do we want to have separate observedProperty's, e.g. WindSpeed, MeanWindSpeed, MaxWindSpeed, MinWindSpeed?

Or, only ever use WindSpeed and use a usedProcedure to make the distinction. E.g.

{ 
  "madeBySensor": "weather-station-6-anemometer",
  "observedProperty": "WindSpeed",
  "resultTime": "2020-02-12T11:30:00.000Z",
  "phenomenonTime": {
    "hasBeginning": "2020-02-12T11:00:00.000Z",
    "hasEnd": "2020-02-12T11:30:00.000Z"
  },
  "hasResult": {
    "value": 10.9
  },
  "usedProcedure": "Mean"
}

N.B. I've used a Time Interval approach to detail the time frame each observation is valid over.

I think there's also an argument for allowing usedProcedure to be an array, for example you might first apply a bias correction and then perform the mean average in case you'd have "usedProcedure": ["BiasCorrection", "Mean"].

EttoreHector commented 4 years ago

Thank you, Simon.

I would add:

as ObservedProperty-s pertaining to traffic, and

as an ObservedProperty pertaining to hydrology. I could not find their counterparts in other vocabularies.

In regard to Air Quality I would add:

Also, almost all of the air quality measurement here in Manchester are provided in μg/m3, while the concentration provided in the CF vocabulary for the pollutants of interest is expressed in kg/m3. Could we defined new terms for those same pollutants with μg/m3 as alternative unit of measurement?

SiBell commented 4 years ago

Thanks Ettore. Couple of things:

SiBell commented 4 years ago

A few more that have just occurred to me:

observedProperty notes units equivalent to
BatteryVoltage V (qudt:V)
BatteryPercent % (unit:Percent)
SignalStrength Can we use this for RSSI? dB (unit:DeciB) quantityKind:SignalStrength

These raise an interesting point. Best explained with an example: say you have a weather station, virtually every sensor on that weather station will be producing observations that can be tagged with the following properties {"featureOfInterest": "EarthAtmosphere", "discipline": ["Meteorology"]"}. However, the weather station also records its BatteryVoltage and SignalStrength, in which case the featureOfInterest and discipline of these observations probably needs to change, perhaps to {"featureOfInterest": "weather-station-no-5", "discipline": ["Diagnostics"]"}.

EttoreHector commented 4 years ago

Thanks, Simon.

As for the time interval approach, I'm happy with that.

Regarding the units my guess is that we have to redefine terms such as _mass_concentration_of_carbon_monoxide_inair and the like in our own vocabulary with the units we choose.

SiBell commented 4 years ago

Great. Thanks Ettore.

How about WaterDepth instead of WaterLevel then? Bit clearer?

EttoreHector commented 4 years ago

Yes, Simon. WaterDepth is clearer. Thank you for the suggestion.

EttoreHector commented 4 years ago

Looking at the data produced by our new noise sensor devices, I realised that when it comes to noise level the raw data usually pertains to much more specific observed properties.

Based on this I would suggest to add the following ObservedProperty-s

All this quantities are measured in dB and you can find their definition here (but not in a semantic web friendly-fashion).

If these properties are too specific and not of much use for the other observatories, I can create a vocabulary that is specific to Manchester.

SiBell commented 4 years ago

Thanks Ettore. Who knew there were that many different types of sounds! I was going to say that these are too abbreviated, e.g. would LA10 be better written as AWeightedSoundLevel10, but these abbreviations seem to be the industry standard, so I'd say best to stick with them.

Again these look like they might need to use a Time Interval.

SiBell commented 4 years ago

I've just had a chat with one of our air pollution experts and she seemed to think we'd be better off using the same units as used in the national air quality objectives.

For example, µg/m3 is a far more common unit for measuring PM in. I've therefore updated my table above to use this instead of µg/cm3.

We also need to be aware that gases can be expressed in either parts per million/billion or as a mass concentration, e.g. in µg/m3. We'd need to decided which way to handle this, either:

  1. We have two different observedProperty's e.g. Co2Concentration (in ppm) and Co2MassConcentration (in µg/m3).
  2. We just have Co2Concentration and handle the unit conversions ourselves, and make sure the unit is specifically defined in the observations we serve.
SiBell commented 4 years ago

We'll also have a lightning detector, which will generate an observation each time lightning is either directly overhead (an alert) or nearby (a warning). I'm not too sure what's the best way to handle this.

Option 1)

observedProperty notes units
LightningAlert lightning detected overhead A boolean?
LightningWarn distant lightning detected A boolean?

Option 2)

observedProperty notes units
LightningStatus An enum? E.g. 'overhead' or 'nearby'.
lukeshope commented 4 years ago

Many thanks for pulling this together.

As discussed on our last call, I have now put together a stub for documenting these vocabulary terms in machine-readable form, and will at some point write some JavaScript to pull the JSON-LD document and descriptions into the actual standards document.

Some initial comments on the above:

We'll also have a lightning detector, which will generate an observation each time lightning is either directly overhead (an alert) or nearby (a warning). I'm not too sure what's the best way to handle this.

An enum sounds like a more future proofed option than a boolean for the lightning detector. Is there a way of generalising it so it's not tied to the thresholds for distant/nearby associated with that specific piece of equipment?

We also need to be aware that gases can be expressed in either parts per million/billion or as a mass concentration, e.g. in µg/m3. We'd need to decided which way to handle this, either:

I presume we're agreed that the units should be handled in the @type of the object under the hasResult property of the observation, but just wanted to make sure. It makes sense to be able to express both ppm/ppb and ug/m3, given the differences in conversion applied by the WHO vs DEFRA etc.

I don't think we have to mandate the units for every observed property, but it would be helpful to give a recommendation.

I'd be in favour of the observedProperty simply being NO2Concentration, CO2Concentration, PM10Concentration etc. or a more verbose version of the same, i.e. NitrogenDioxideConcentration.

How about WaterDepth instead of WaterLevel then? Bit clearer?

I think we need both. We're likely to be using depths in most cases, but level (relative to datum) for tidal reaches. Bonus points if we specify the datum in a second property, which is probably going to be Newlyn in all our cases.

We need to decide how we want to handle Average, Max and Min values.

Do we want to have separate observedProperty's, e.g. WindSpeed, MeanWindSpeed, MaxWindSpeed, MinWindSpeed?

Or, only ever use WindSpeed and use a usedProcedure to make the distinction. E.g.

Your example with usedProcedure as an array looks like a perfect solution to me. A huge proportion of our observations are likely to be averages in reality, very few of them will be instantaneous samples. Certainly all air quality and lots of weather stations in Newcastle are giving us values that are actually averages over a few seconds or a few minutes.

SiBell commented 4 years ago

Cheers for setting up the stub @lukessmith.

W.r.t. the lightning detector, how about 'alert' and 'warning' as generic enum values?

An alternative to having the units in the @type of hasResult is to have a unit property in the hasResult object. Seems to be the approach that most of the examples in the SSN docs take. This would certainly be easier to manage on the front-end. E.g. for your web app to work out what units to show on a line graph.

Happy to keep the observed property simple, e.g. NO2Concentration. Although I quite like the verbose option (e.g. NitrogenDioxideConcentration) we could end up with some very long class names if we have more complex compounds. NOx would need to be NitrogenOxidesConcentration rather than NOxConcentration.

I agree with the following:

I don't think we have to mandate the units for every observed property, but it would be helpful to give a recommendation.

Your example with usedProcedure as an array looks like a perfect solution to me.

Although WindSpeed is perhaps a bad example. We may need to have WindSpeed, WindGustSpeed, WindDirection and WindGustDirection. Although WindGustSpeed could be handled fairly easily as WindSpeed:

{
  "observedProperty": "WindSpeed",
  "usedProcedure": ["max"]
}

Modelling WindGustDirection as WindDirection would be a bit more gnarly:

{
  "observedProperty": "WindDirection",
  "usedProcedure": ["direction-during-max-windspeed"]
}
SiBell commented 4 years ago

What about an observedProperty called Location? e.g. for observations collected by a GPS sensor.

This is a weird one because we'll probably already have a location property in our observation with a GeoJSON object, and therefore the value in the hasResult object would just be a replication of this locational information.

SiBell commented 4 years ago

More to add from Simon J: https://www.aviationweather.gov/dataserver/fields?datatype=metar

SiBell commented 4 years ago

A reminder that we need to add in DewPointTemperature

EttoreHector commented 4 years ago

This is a message I posted some time ago on this thread. Just wondering if these observables can be added to our list of observed properties.

Old Message

Looking at the data produced by our new noise sensor devices, I realised that when it comes to noise level the raw data usually pertains to much more specific observed properties.

Based on this I would suggest to add the following ObservedProperty-s

LAeq LAFmax LA10 LA50 LA90 LCeq LCFmax LC10 LC50 LC90

All this quantities are measured in dB and you can find their definition here (but not in a semantic web friendly-fashion).

If these properties are too specific and not of much use for the other observatories, I can create a vocabulary that is specific to Manchester.

SiBell commented 4 years ago

Feel free to add them to the JSON vocabulary file. I would have added them if I had the time. Try to follow the format I used for the others. Up to you if you want to do it on a separate branch and create a pull request, or push straight to master if you feel confident. The label and comment fields will prove handy for these ones to explain what they mean.

EttoreHector commented 4 years ago

No worries, Simon.

I will add those terms myself in the common vocabulary as I understand this is ok with you.

On Fri, 10 Apr 2020 at 14:25, Si Bell notifications@github.com wrote:

Feel free to add them to the JSON vocabulary file. I would have added them if I had the time. Try to follow the format I used for the others. Up to you if you want to do it on a separate branch and create a pull request, or push straight to master if you feel confident. The label and comment fields will prove handy for these ones to explain what they mean.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/urbanobservatory/standards/issues/23#issuecomment-612027174, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB6X6YOOVC5JSDOU7LLFJM3RL4M6HANCNFSM4K3LFAWQ .

EttoreHector commented 4 years ago

I noticed that there is an ObservableProperty in our vocabulary called "WindSpeed". I wonder whether it would be sensible to define a more general ObservableProperty called "Speed", since there are many things we can measure the speed of, for example the speed of vehicles going down the road. Or shall we rather define specific observableProperty-s for every entity of which we measure the speed (following on the previous example, "VehicleSpeed")?

SiBell commented 4 years ago

My thought process for this is what if an end-user wanted to plot a map of vehicle speeds across Manchester. How would this user go about filtering our observations to do this.

Certainly having an observable property called VehicleSpeed would make this easier:

GET .../observations?observedProperty=VehicleSpeed

The question is if we don't do this, and have a generic Speed how would we separate the speed of a car from the speed of a river, or an elevator, etc, etc.

Using the Platform doesn't really help us, because many vehicle speed sensors won't be hosted on the vehicle itself, they could just be a camera or radar unit pointed at the road.

The Discipline would be a big help, e.g. Transport for vehicles, Hydrology for vehicles, but it would rely on there not being any other Speed measurements in the Transport discipline that aren't of a vehicle.

The hasFeatureOfInterest helps slightly, e.g. the feature of interest might be five-ways-roundabout or west-street-northbound, but the end user will have to sift through and manually work out which are roads or railway lines, etc.

With that in mind I'd veer towards VehicleSpeed, although it does feel as if we need some sort of inheritance/sub-class system. E.g. VehicleSpeed and WaterSpeed are all sub-classes of Speed, and LorrySpeed could be sub-class of VehicleSpeed. Too complicated perhaps?

This ties in with a big challenge I still have, an example being if the camera/radar that measures vehicle speed can also distinguish what type of vehicle it is (lorry/car/bike), how do we make sure both the speed and vehicle type stay linked. I.e. we don't what one observation object with the speed in it, and another with the vehicle type, and no way to make the link between the two. I'm still having this issue with WindSpeed measurements and linking it to the WindDirection measurement made at the same time.

EttoreHector commented 4 years ago

Yes I agree, Simon. However, we may have to add even more specific ObservedPropery-s, like BusSpeed, BicycleSpeed, CarSpeed etc, if we go down this route. It's all good with me, though.