PIOT-GDA-10-003: Update DeviceDataManager to analyze messages from the CDA and take an appropriate action

[labbenchstudios]

Description

• Add functionality to DeviceDataManager to handle incoming CDA messages: SensorData, SystemPerformanceData, and ActuatorData (response messages).

Review the README

• Please see README.md for further information on, and use of, this content.
• License for embedded documentation and source codes: PIOT-DOC-LIC

Estimated effort may vary greatly

• The estimated level of effort for this exercise shown in the 'Estimate' section below is a very rough approximation. The actual level of effort may vary greatly depending on your development and test environment, experience with the requisite technologies, and many other factors.

Actions

• The actions related to DeviceDataManager in PIOT-GDA-05-005 provided guidance on creating two methods: handleSensorMessage(ResourceNameEnum resourceName, SensorData data) and handleSystemPerformanceMessage(ResourceNameEnum resourceName, SystemPerformanceData data). Each currently logs a message, so let's add some real functionality.
  • NOTE: If you've implemented Option 1 for the GDA's MqttClientConnector callback, you'll first have to modify handleIncomingMessage(ResourceNameEnum resourceName, String msg) to determine the content of msg and then convert it into the appropriate data object using DataUtil. From there, you can invoke one of these handle...() methods. Option 2 will invoke these directly, which is the suggested approach.
  • The handleSensorMessage() implementation is flexible - and up to you to implement MOSTLY as you wish. Here are some guidelines, with an example implementation included at the end of this card:
  • Log a message indicating this method has been called - this should already be done.
  • Invoke the soon-to-be created handleIncomingDataAnalysis(ResourceNameEnum resource, SensorData data) private method.
  • Within handleIncomingDataAnalysis(ResourceNameEnum resource, SensorData data), consider the following guidelines (for students of Connected Devices, these are required):
  • Process any threshold crossing floor and ceiling values stored within the GDA's PiotConfig.props. Much like with the CDA, the GDA should also check if incoming SensorData crosses these threshold values. Since the CDA already handles this for temperature, the GDA can process this for humidity instead.
    • If the humidity levels fall below the configured floor value, or exceeds the configured ceiling value, create an ActuatorData message with the command set to turn the humidifier on or off - respectively (e.g., ON if humidity is too low, and OFF if humidity is too high (or already at nominal)).
    • To determine if a threshold crossing occurred, let's assume that two threshold crossings within 5 minutes is sufficient to trigger the actuation event (ON or OFF). If easier, you can assume two threshold crossings in sequence is also sufficient.
    • If the rule above is triggered, create the relevant ActuatorData message and publish it to the CDA's ResourceNameEnum.CDA_ACTUATOR_CMD_RESOURCE topic if using MQTT, or resource if using CoAP. For your own testing / validation, I recommend including a message with ActuatorData (as the state data entry) - this should be rendered in the Sense HAT LED when running with the emulator enabled.
    • NOTE: You can choose a different type of actuation event, of course - this is merely a suggestion. The objective is to have the GDA trigger an appropriate actuation event based on the data received from the CDA.
  • Check if the persistence client is active, and if so, write the response to the local datastore (this is mostly for logging and event tracking purposes). PIOT-GDA-05-006 is an additional exercise (optional) for Lab Module 05 that covers the basic persistence client requirements. If you prefer, you can simply write the object's JSON to the filesystem.
  • Call the private method handleUpstreamTransmission(), passing it the ResourceNameEnum, newly generated JSON data, and - if preferable - a QoS value. The signature for this method is already in place, and the implementation can - for now - simply remain as a log message only.
  • Return a boolean indicating success or failure in processing the data - this should already be done.
• Add the private methods you declared previously, as follows:
  • For handleIncomingDataAnalysis(ResourceNameEnum resourceName, ActuatorData data), do the following:
  • Log a debug message indicating this method has been called. FINE is probably best.
  • For now, this will be empty. It will eventually publish back to the CDA using either MQTT or CoAP (or attempt both). We'll revisit this in Part 04.
  • For handleIncomingDataAnalysis(ResourceNameEnum resourceName, SystemStateData data), do the following:
  • Log a debug message indicating this method has been called. FINE is probably best.
  • For now, this will be empty. You may implement this however you choose, as it's a command the GDA should interpret and handle internally.
  • For handleIncomingDataAnalysis(ResourceNameEnum resourceName, SensorData data), do the following:
  • Log a debug message indicating this method has been called. FINE is probably best.
  • Implement the logic to process a simple threshold crossing for humidity (again, temperature should already be handled within the CDA's logic). This will require the following:
    • Add the requisite threshold crossing properties to the GDA's PiotConfig.props configuration file, and load them within DeviceDataManager.
    • Add the requisite class-scoped variables to store the last known humidity data for the CDA, along with timestamp, and compare it with the latest received.
    • Add the requisite class-scoped variables to store the last known humidity ActuatorData command and response from the CDA. This will allow the GDA to check if the command has been sent to, received and acknowledged by, the CDA.
    • Log a message indicating when the actuation event is triggered.
    • Optionally, save the result to the local data store via a call to the persistence client.
    • Here's one approach you may want to consider for implementing the above requirements. While this isn't complete, it should help you get started.

Threshold Crossing Rules - Configuration File Updates

NOTE: The example code provided here assumes there's a single CDA and GDA in the test loop.

• Add the following to the GDA's PiotConfig.props (you may also want to store these key names within ConfigConst for convenience):

  
  # if true, GDA will process humidity change events from CDA
  handleHumidityChangeOnDevice = True

min seconds between readings before triggering actuation event

humidityMaxTimePastThreshold = 300

ideal average humidity level (% relative)

nominalHumiditySetting = 40.0

min value before turning on humidifier (% relative)

triggerHumidifierFloor = 30.0

max value before turning off humidifier (% relative)

triggerHumidifierCeiling = 50.0

***Threshold Crossing Logic - `DeviceDataManager` Updates***
- Add the requisite class-scoped threshold crossing properties:
```java
private ActuatorData   latestHumidifierActuatorData = null;
private ActuatorData   latestHumidifierActuatorResponse = null;
private SensorData     latestHumiditySensorData = null;
private OffsetDateTime latestHumiditySensorTimeStamp = null;

private boolean handleHumidityChangeOnDevice = false; // optional
private int     lastKnownHumidifierCommand   = ConfigConst.OFF_COMMAND;

// TODO: Load these from PiotConfig.props
private long    humidityMaxTimePastThreshold = 300; // seconds
private float   nominalHumiditySetting   = 40.0f;
private float   triggerHumidifierFloor   = 30.0f;
private float   triggerHumidifierCeiling = 50.0f;

• Update the constructor to parse rules from configuration:
  • NOTE: Only partial code sample shown. This is NOT the full constructor - your code may look different than that shown below.

    
    // parse config rules for local actuation events

// TODO: add these to ConfigConst this.handleHumidityChangeOnDevice = configUtil.getBoolean( ConfigConst.GATEWAY_DEVICE, "handleHumidityChangeOnDevice");

this.humidityMaxTimePastThreshold = configUtil.getInteger( ConfigConst.GATEWAY_DEVICE, "humidityMaxTimePastThreshold");

this.nominalHumiditySetting = configUtil.getFloat( ConfigConst.GATEWAY_DEVICE, "nominalHumiditySetting");

this.triggerHumidifierFloor = configUtil.getFloat( ConfigConst.GATEWAY_DEVICE, "triggerHumidifierFloor");

this.triggerHumidifierCeiling = configUtil.getFloat( ConfigConst.GATEWAY_DEVICE, "triggerHumidifierCeiling");

// TODO: basic validation for timing - add other validators for remaining values if (this.humidityMaxTimePastThreshold < 10 || this.humidityMaxTimePastThreshold > 7200) { this.humidityMaxTimePastThreshold = 300; }

- Update `handleSensorData(ResourceNameEnum, SensorData data)` to invoke the analysis functionality. 
  - NOTE 1: Your code may look much different than that given below.
  - NOTE 2: Be sure to add an implementation for 'handleUpstreamTransmission(resourceName, jsonDat, qos)`

```java
@Override
public boolean handleSensorMessage(ResourceNameEnum resourceName, SensorData data)
{
    if (data != null) {
        _Logger.fine("Handling sensor message: " + data.getName());

        if (data.hasError()) {
            _Logger.warning("Error flag set for SensorData instance.");
        }

        String jsonData = DataUtil.getInstance().sensorDataToJson(data);

        _Logger.fine("JSON [SensorData] -> " + jsonData);

        // TODO: retrieve this from config file
        int qos = ConfigConst.DEFAULT_QOS;

        if (this.enablePersistenceClient && this.persistenceClient != null) {
            this.persistenceClient.storeData(resourceName.getResourceName(), qos, data);
        }

        this.handleIncomingDataAnalysis(resourceName, data);

        this.handleUpstreamTransmission(resourceName, jsonData, qos);

        return true;
    } else {
        return false;
    }
}

private void handleUpstreamTransmission(ResourceNameEnum resource, String jsonData, int qos)
{
    // NOTE: This will be implemented in Part 04
    _Logger.info("TODO: Send JSON data to cloud service: " + resource);
}   

• Add the requisite threshold crossing logic to handleIncomingDataAnalysis(ResourceNameEnum resourceName, SensorData data). NOTE: You'll notice in the sample (and incomplete) code below that I added two convenience methods: one to convert the ISO 8601 timestamp String into a Java OffsetDateTime instance, and another to send the ActuatorData command to the CDA.
  • IMPORTANT NOTE: This is an INCOMPLETE and VERY BASIC time-series analysis code sample to help you get started and is not intended to provide a solution. For more advanced time-series analytics, it would be beneficial to explore the use of other tools and ML utilities. Your code implementation may look significantly different than that given below.
  • Also: You'll notice the call to this.actuatorDataListener.onActuatorDataUpdate(data) in the private method sendActuatorCommandtoCda(ResourceNameEnum resource, ActuatorData data). If you create a standalone test for this functionality (as depicted below under the Tests section), you'll need to review the updates to the actuator handler in PIOT-GDA-08-003 to ensure null ref's are handled properly. Why? The handler maintains an ActuatorData ref that is invoked within this call; however, it's only initialized when the handler's GET method is invoked. The test below never initializes the ActuatorData ref (by design), so it's important to check for a null ref first.

private void handleIncomingDataAnalysis(ResourceNameEnum resource, SensorData data)
{
    // check either resource or SensorData for type
    if (data.getTypeID() == ConfigConst.HUMIDITY_SENSOR_TYPE) {
        handleHumiditySensorAnalysis(resource, data);
    }
}

private void handleHumiditySensorAnalysis(ResourceNameEnum resource, SensorData data)
{
    //
    // NOTE: INCOMPLETE and VERY BASIC CODE SAMPLE. Not intended to provide a solution.
    //

    _Logger.fine("Analyzing humidity data from CDA: " + data.getLocationID() + ". Value: " + data.getValue());

    boolean isLow  = data.getValue() < this.triggerHumidifierFloor;
    boolean isHigh = data.getValue() > this.triggerHumidifierCeiling;

    if (isLow || isHigh) {
        _Logger.fine("Humidity data from CDA exceeds nominal range.");

        if (this.latestHumiditySensorData == null) {
            // set properties then exit - nothing more to do until the next sample
            this.latestHumiditySensorData = data;
            this.latestHumiditySensorTimeStamp = getDateTimeFromData(data);

            _Logger.fine(
                "Starting humidity nominal exception timer. Waiting for seconds: " +
                this.humidityMaxTimePastThreshold);

            return;
        } else {
            OffsetDateTime curHumiditySensorTimeStamp = getDateTimeFromData(data);

            long diffSeconds =
                ChronoUnit.SECONDS.between(
                    this.latestHumiditySensorTimeStamp, curHumiditySensorTimeStamp);

            _Logger.fine("Checking Humidity value exception time delta: " + diffSeconds);

            if (diffSeconds >= this.humidityMaxTimePastThreshold) {
                ActuatorData ad = new ActuatorData();
                ad.setName(ConfigConst.HUMIDIFIER_ACTUATOR_NAME);
                ad.setLocationID(data.getLocationID());
                ad.setTypeID(ConfigConst.HUMIDIFIER_ACTUATOR_TYPE);
                ad.setValue(this.nominalHumiditySetting);

                if (isLow) {
                    ad.setCommand(ConfigConst.ON_COMMAND);
                } else if (isHigh) {
                    ad.setCommand(ConfigConst.OFF_COMMAND);
                }

                _Logger.info(
                    "Humidity exceptional value reached. Sending actuation event to CDA: " +
                    ad);

                this.lastKnownHumidifierCommand = ad.getCommand();
                sendActuatorCommandtoCda(ResourceNameEnum.CDA_ACTUATOR_CMD_RESOURCE, ad);

                // set ActuatorData and reset SensorData (and timestamp)
                this.latestHumidifierActuatorData = ad;
                this.latestHumiditySensorData = null;
                this.latestHumiditySensorTimeStamp = null;
            }
        }
    } else if (this.lastKnownHumidifierCommand == ConfigConst.ON_COMMAND) {
        // check if we need to turn off the humidifier
        if (this.latestHumidifierActuatorData != null) {
            // check the value - if the humidifier is on, but not yet at nominal, keep it on
            if (this.latestHumidifierActuatorData.getValue() >= this.nominalHumiditySetting) {
                this.latestHumidifierActuatorData.setCommand(ConfigConst.OFF_COMMAND);

                _Logger.info(
                    "Humidity nominal value reached. Sending OFF actuation event to CDA: " +
                    this.latestHumidifierActuatorData);

                sendActuatorCommandtoCda(
                    ResourceNameEnum.CDA_ACTUATOR_CMD_RESOURCE, this.latestHumidifierActuatorData);

                // reset ActuatorData and SensorData (and timestamp)
                this.lastKnownHumidifierCommand = this.latestHumidifierActuatorData.getCommand();
                this.latestHumidifierActuatorData = null;
                this.latestHumiditySensorData = null;
                this.latestHumiditySensorTimeStamp = null;
            } else {
                _Logger.fine("Humidifier is still on. Not yet at nominal levels (OK).");
            }
        } else {
            // shouldn't happen, unless some other logic
            // nullifies the class-scoped ActuatorData instance
            _Logger.warning(
                "ERROR: ActuatorData for humidifier is null (shouldn't be). Can't send command.");
        }
    }
}

private void sendActuatorCommandtoCda(ResourceNameEnum resource, ActuatorData data)
{
    // NOTE: This is how an ActuatorData command will get passed to the CDA
    // when the GDA is providing the CoAP server and hosting the appropriate
    // ActuatorData resource. It will typically be used when the OBSERVE
    // client (the CDA, assuming the GDA is the server and CDA is the client)
    // has sent an OBSERVE GET request to the ActuatorData resource.
    if (this.actuatorDataListener != null) {
        this.actuatorDataListener.onActuatorDataUpdate(data);
    }

    // NOTE: This is how an ActuatorData command will get passed to the CDA
    // when using MQTT to communicate between the GDA and CDA
    if (this.enableMqttClient && this.mqttClient != null) {
        String jsonData = DataUtil.getInstance().actuatorDataToJson(data);

        if (this.mqttClient.publishMessage(resource, jsonData, ConfigConst.DEFAULT_QOS)) {
            _Logger.info(
                "Published ActuatorData command from GDA to CDA: " + data.getCommand());
        } else {
            _Logger.warning(
                "Failed to publish ActuatorData command from GDA to CDA: " + data.getCommand());
        }
    }
}

private OffsetDateTime getDateTimeFromData(BaseIotData data)
{
    OffsetDateTime odt = null;

    try {
        odt = OffsetDateTime.parse(data.getTimeStamp());
    } catch (Exception e) {
        _Logger.warning(
            "Failed to extract ISO 8601 timestamp from IoT data. Using local current time.");

        // TODO: this won't be accurate, but should be reasonably close, as the CDA will
        // most likely have recently sent the data to the GDA
        odt = OffsetDateTime.now();
    }

    return odt;
}

Estimate

• Medium

Tests

Simple Test Setup

• Testing the updated DeviceDataManager logic ONLY
  • NOTE: You'll need to write your own test to verify the logic functions as expected. One way to do so is to use the instructions provided below.
  • Create a new test class within ./src/test/java/programmingtheiot/part03/integration/app named DeviceDataManagerSimpleCdaActuationTest (or any other name you choose). If you'd like, you can copy the template from DeviceDataManagerWithCommsTest.java and remove all the existing test methods.
  • Implement the following test logic, then run the test and observe the output.
  • NOTE: This will test the analysis functionality of DeviceDataManager when receiving SensorData messages that are from the CDA's humidity readings. Actions include the following:
    • In PiotConfig.props, set humidityMaxTimePastThreshold = 10 seconds (for this test)
    • Make sure DeviceDataManager reads this, or set the value directly (for this test)
    • Create a Humidity SensorData message
    • Set typeID = ConfigConst.HUMIDITY_SENSOR_TYPE
    • Set nominal value, send two messages, watch log output (no actuation event generated)
    • Set value to nominalHumiditySetting from PiotConfig.props
    • Set exceptional value, send two+ messages separated by > humidityMaxTimePastThreshold
    • Log output should indicate actuation ON event generated
    • Set nominal value, send two+ messages separated by > humidityMaxTimePastThreshold
    • Log output should indicate actuation OFF event generated
    • If desired, repeat the above steps and ensure the logic executes as expected.
  • The sample code below is a notional example for this test case. Your code may look significantly different:

/**
 * Test method for running the DeviceDataManager.
 * 
 */
@Test
public void testSendActuationEventsToCda()
{
    DeviceDataManager devDataMgr = new DeviceDataManager();

    // NOTE: Be sure your PiotConfig.props is setup properly
    // to connect with the CDA
    devDataMgr.startManager();

    ConfigUtil cfgUtil = ConfigUtil.getInstance();

    // TODO: add these to ConfigConst
    float nominalVal = cfgUtil.getFloat(ConfigConst.GATEWAY_DEVICE,   "nominalHumiditySetting");
    float lowVal     = cfgUtil.getFloat(ConfigConst.GATEWAY_DEVICE,   "triggerHumidifierFloor");
    float highVal    = cfgUtil.getFloat(ConfigConst.GATEWAY_DEVICE,   "triggerHumidifierCeiling");
    int   delay      = cfgUtil.getInteger(ConfigConst.GATEWAY_DEVICE, "humidityMaxTimePastThreshold");

    // Test Sequence No. 1
    generateAndProcessHumiditySensorDataSequence(
        devDataMgr, nominalVal, lowVal, highVal, delay);

    // TODO: Add more test sequences if desired.

    devDataMgr.stopManager();
}

private void generateAndProcessHumiditySensorDataSequence(
    DeviceDataManager ddm, float nominalVal, float lowVal, float highVal, int delay)
{
    SensorData sd = new SensorData();
    sd.setName("My Test Humidity Sensor");
    sd.setLocationID("constraineddevice001");
    sd.setTypeID(ConfigConst.HUMIDITY_SENSOR_TYPE);

    sd.setValue(nominalVal);
    ddm.handleSensorMessage(ResourceNameEnum.CDA_SENSOR_MSG_RESOURCE, sd);
    waitForSeconds(2);

    sd.setValue(nominalVal);
    ddm.handleSensorMessage(ResourceNameEnum.CDA_SENSOR_MSG_RESOURCE, sd);
    waitForSeconds(2);

    sd.setValue(lowVal - 2);
    ddm.handleSensorMessage(ResourceNameEnum.CDA_SENSOR_MSG_RESOURCE, sd);
    waitForSeconds(delay + 1);

    sd.setValue(lowVal - 1);
    ddm.handleSensorMessage(ResourceNameEnum.CDA_SENSOR_MSG_RESOURCE, sd);
    waitForSeconds(delay + 1);

    sd.setValue(lowVal + 1);
    ddm.handleSensorMessage(ResourceNameEnum.CDA_SENSOR_MSG_RESOURCE, sd);
    waitForSeconds(delay + 1);

    sd.setValue(nominalVal);
    ddm.handleSensorMessage(ResourceNameEnum.CDA_SENSOR_MSG_RESOURCE, sd);
    waitForSeconds(delay + 1);
}

private void waitForSeconds(int seconds)
{
    try {
        Thread.sleep(seconds * 1000);
    } catch (InterruptedException e) {
        // ignore
    }
}

Integration Test Setup

• Testing the GDA and CDA together
  • NOTE: This assumes you're using MQTT to communicate between the CDA and GDA.
  • Make sure the MQTT broker is running and available on your local network for testing the CDA and GDA.
  • Make sure the SenseHAT emulator is running.
  • Make sure the CDA application is running.
• Testing
  • Run the GDA application and the CDA application separately. Make sure each applications runtime timer is set to a value that gives you enough time to test the following:
  • Adjust the SenseHAT emulator temperature slider. Move it beyond the floor and / or ceiling thresholds and leave it in place for at least a minute (assuming the CDA is processing temp events at least every minute). You should see an actuator event triggered by the CDA to re-adjust the temperature.
  • Adjust the SenseHAT humidity slider. Move it beyond the ceiling threshold specified by the GDA and leave it in place for at least 5 minutes (assuming humidityMaxTimePastThreshold = 300). You should see an actuator event triggered by the GDA to re-adjust the humidity.

[tangyisheng2]

Hey, I found a typo in DeviceDataManagerSimpleCdaActuationTest

float lowVal     = cfgUtil.getFloat(ConfigConst.GATEWAY_DEVICE,   "triggerHumidiferFloor");

Should be triggerHumidifierFloor

Same typo also in DeviceDataManager

this.triggerHumidiferFloor = configUtil.getFloat(ConfigConst.GATEWAY_DEVICE, "triggerHumidifierFloor");

[labbenchstudios]

Now corrected; fixed throughout code samples.