Question: Can a plugin send MQTT messages itself ?

[s0170071]

Can a plugin send MQTT messages itself ?

Context: I do own a Viessmann Vitocal heater which I can probe through its optical interface. It supplies about 40 parameters. Using the usual sensor interfaces, I can only have four sensor values.

Question: What would be the best way to forward those 40 values to the MQTT controller ?

Maybe: I found the void sendData(struct EventStruct *event) method in controller.cpp which may be used but it seems not to be intended for that purpose. Is there already a way to do this ?

[TD-er]

I think it might be possible, but agreeing on breaking this now, is something I will regret in the near future I guess. It is not how ESPeasy is designed and creating something that will break such design decisions is always a bad idea since it will come back and make things way more complicated to maintain.

Just to get a feeling for the use-case, what sort of data is there and what kind of MQTT messages should be expected? How many parameters are there to update on every sample and how many occasionally? And can you give a structural overview of the MQTT messages/topics? Should there be any set of samples that have to be sent at once? There may be some other way I was thinking about. That may take some changes, but would be more universal in the end.

[Grovkillen]

Side note: I got a Vitocal as well and I'm really interested in your hardware. Is it a Vissmann cable or own built?

[s0170071]

@TD-er There are no parameters that require urgent updates. At the moment I do sync them to my Mosquitto broker once every 10 minutes. There are about 40 parameters. There is no "format". Each parameter has its own subject and the data is just a float converted to a string. E.g. subject / value Vito/outsideTemperature 22.2 Vito/setpoint 21 Vito/partyTemp 21

@Grovkillen: I use this https://github.com/bertmelis/VitoWifi/issues/27 schematic and hook it up to a WemosD1 mini.

[TD-er]

So if we add the option to include the parameter name into the MQTT topic it would already be an improvement.

Also adding some kind of "virtual" device, could help to increase the number of parameters.

[Andrey2509]

I also need to get 38 float values from my electric meter (custom plugin). Please add new SENSOR_TYPE_STRING/SENSOR_TYPE_JSON to get numerous values in json format or ability to send string (json data) to mqtt broker from plugin.

[s0170071]

@TD-er I am not sure what you mean by virtual device. I was thinking about two or three controller access functions:

controller_ready (uint8 id, uint8 type) // returns true/false if connected or just set up controller_send (uint8 id, string value) // send value. topic is defined by the controller settings. controller_send (uint8 id, string topic, string value) // with custom topic

[s0170071]

@Andrey2509 what kind of meter are you using ? SML over IR ?

[Andrey2509]

I get data via UART/RS485 converter using software serial TX/RX.

[TD-er]

I have 2 ideas to get more values possible.

The simplest one is to somewhat rotate through Alle values and just send them to the controller. This makes using them in rules impossible. So it is violating the design ideas.

Another idea is a bit more work. That's to allow extra virtual devices, not really communicating to hardware but just keeping the state of the last values of sensor data. Those could then send those to selected controllers when updated and be used in rules.

[s0170071]

I would then send values to the virtual device which buffers them until the transmission takes place ? Or am I (my plugin) the virtual device ? That could require a little more ram than just sending them away.

[Andrey2509]

For me virtual device is not the best idea. It demands 40/4 = 10 devices with 4 values...

[TD-er]

@Andrey2509 my idea is that the concept of virtual devices would not be limited to 12, which is now the set limit. I guess it would not even have to be stored, but could be generated my some plugins that would support such virtual devices. But as I said, it is currently just a concept I was thinking about.

The current limit of 4 values per device is a hard limit to overcome, since it is being used all over the code and is set in the stored settings.

[Andrey2509]

My solution to send numerous values using json string. Use for your own risk!

in file ESPEasy-Globals.h:

float UserVar[VARS_PER_TASK * TASKS_MAX];        
String UserVarJson[VARS_PER_TASK * TASKS_MAX];  //add this line

#define SENSOR_TYPE_LONG                   20   
#define SENSOR_TYPE_WIND                   21    
#define SENSOR_TYPE_JSON                   22   //add this line 

in file _CPlugin_SensorTypeHelper.ino:

   case SENSOR_TYPE_QUAD:           
      return 4;                     
    case SENSOR_TYPE_JSON:          //add this line 
      return 5;                     //add this line 

in file Misc.ino:

void createRuleEvents(byte TaskIndex)
{
  LoadTaskSettings(TaskIndex);
  byte BaseVarIndex = TaskIndex * VARS_PER_TASK;
  byte DeviceIndex = getDeviceIndex(Settings.TaskDeviceNumber[TaskIndex]);
  byte sensorType = Device[DeviceIndex].VType;
  for (byte varNr = 0; varNr < Device[DeviceIndex].ValueCount; varNr++)
  {
    String eventString = ExtraTaskSettings.TaskDeviceName;
    eventString += F("#");
    eventString += ExtraTaskSettings.TaskDeviceValueNames[varNr];
    eventString += F("=");

    if (sensorType == SENSOR_TYPE_LONG)
      eventString += (unsigned long)UserVar[BaseVarIndex] + ((unsigned long)UserVar[BaseVarIndex + 1] << 16);
    else
    if (sensorType == SENSOR_TYPE_JSON)                       //add this line
      eventString += UserVarJson[BaseVarIndex + varNr];       //add this line
    else                                                     //add this line

      eventString += UserVar[BaseVarIndex + varNr];

    rulesProcessing(eventString);
  }
}

in file StringConverter.ino:

String doFormatUserVar(byte TaskIndex, byte rel_index, bool mustCheck, bool& isvalid) {
  isvalid = true;
  const byte BaseVarIndex = TaskIndex * VARS_PER_TASK;
  const byte DeviceIndex = getDeviceIndex(Settings.TaskDeviceNumber[TaskIndex]);
  if (Device[DeviceIndex].ValueCount <= rel_index) {
    isvalid = false;
    String log = F("No sensor value for TaskIndex: ");
    log += TaskIndex;
    log += F(" varnumber: ");
    log += rel_index;
    addLog(LOG_LEVEL_ERROR, log);
    return "";
  }
  if (Device[DeviceIndex].VType == SENSOR_TYPE_LONG) {
    return String((unsigned long)UserVar[BaseVarIndex] + ((unsigned long)UserVar[BaseVarIndex + 1] << 16));
  }
  if (Device[DeviceIndex].VType == SENSOR_TYPE_JSON) {       //add this line
    return UserVarJson[BaseVarIndex + rel_index];            //add this line
  }                                                          //add this line
  float f(UserVar[BaseVarIndex + rel_index]);
  if (mustCheck && !isValidFloat(f)) {
    isvalid = false;
    String log = F("Invalid float value for TaskIndex: ");
    log += TaskIndex;
    log += F(" varnumber: ");
    log += rel_index;
    addLog(LOG_LEVEL_DEBUG, log);
    f = 0;
  }
  return toString(f, ExtraTaskSettings.TaskDeviceValueDecimals[rel_index]);
}

  void parseEventVariables(String& s, struct EventStruct *event, boolean useURLencode)
{
  SMART_REPL(F("%id%"), String(event->idx))
  if (s.indexOf(F("%val")) != -1) {
    if (event->sensorType == SENSOR_TYPE_LONG) {
      SMART_REPL(F("%val1%"), String((unsigned long)UserVar[event->BaseVarIndex] + ((unsigned long)UserVar[event->BaseVarIndex + 1] << 16)))
    } else {
     if (event->sensorType == SENSOR_TYPE_JSON) {                        //add this line
        SMART_REPL(F("%val1%"), UserVarJson[event->BaseVarIndex])          //add this line
        SMART_REPL(F("%val2%"), UserVarJson[event->BaseVarIndex+1])         //add this line       (need check if value exist)
        SMART_REPL(F("%val3%"), UserVarJson[event->BaseVarIndex+2])         //add this line       (need check if value exist)
        SMART_REPL(F("%val4%"), UserVarJson[event->BaseVarIndex+3])         //add this line       (need check if value exist)

            } else {

      SMART_REPL(F("%val1%"), formatUserVarNoCheck(event, 0))
      SMART_REPL(F("%val2%"), formatUserVarNoCheck(event, 1))
      SMART_REPL(F("%val3%"), formatUserVarNoCheck(event, 2))
      SMART_REPL(F("%val4%"), formatUserVarNoCheck(event, 3))
     }                                                                        //add this line
    }
  }
}

in csustom plugin _Pxxx_PluginTemplate.ino just use:

in case PLUGIN_DEVICE_ADD:

Device[deviceCount].VType = SENSOR_TYPE_JSON;

in case PLUGIN_READ:

            String myjson = String("{");
                myjson += String( "\"Ia1\":" + String(Ia[0]/10) + ",\"Ia2\":"+   String(Ia[1]/10) + ",\"Ia3\":" +   String(Ia[2]/10) + ",");
                myjson += String( "\"Uv1\":" + String(Uv[0]) + ",\"Uv2\":"+   String(Uv[1]) + ",\"Uv3\":" +   String(Uv[2]) + ",");
                myjson += String( "\"Pw0\":" + String(Pw[0]) + ",\"Pw1\":"+   String(Pw[1]) + ",\"Pw2\":" +   String(Pw[2]) +   ",\"Pw3\":" +  String(Pw[3]) + ",");
                myjson += String( "\"T1\":" + String(T1) + ",\"T2\":"+   String(T2) + ",\"T3\":"+   String(T3)+   ",\"T4\":"+   String(T4));
                myjson += String("}");
                UserVarJson[event->BaseVarIndex] = myjson;

Note: On web page "Device" EspeEasy correct value with long json string will appear for a short moment, after that it will show "NaN" . It is not problem, value correctly send via mqtt controller (Openhub).

[TD-er]

That's a very specific implementation of Json (for Openhub) But I guess we could use it as inspiration on how to allow more parameters. However you cannot use the values in rules and String UserVarJson[VARS_PER_TASK * TASKS_MAX]; may take quite some memory.

There is also some JSON formatting function in the stringconverter.ino file, which makes it a bit more readable. And you can try to call reserve() on the myjson String, since you have a rather good estimate on its size. That will help heap fragmentation.

If it is OK to send lots of data (>4 values) like this, I could add some other call to the send to controller code, to add function to send an arbitrary number of values and keys. For example std::vector<std::pair<String, float>> Or just something like this: send....(const std::vector<String>& key, const std::vector<float>& value)

That wold make it more universal with as only drawback that you cannot use the values in rules for such a specific plugin. (or max 4)

[Andrey2509]

@TD-er I can use any format json on server side, so if you can add any method to transfer numerous values it would be great

[s0170071]

I could fetch the controller settings and open a mqtt connection myself. If the controller is not enabled no one should notice Dirty, isn't it ? ;-)

[s0170071]

@Grovkillen I made an alpha version of a plugin for Vitotronic. It adds two webpages /read and /write which can be used for interfacing with the heater. There is no other interaction with ESPEasy at this time but maybe this gives you ideas for a more useful integration.

https://github.com/letscontrolit/ESPEasyPluginPlayground/pull/109

[Grovkillen]

@s0170071 Thanks! I will try to get time to test it out next week. Good job. :)