Plugin support

[nunofgs]

Hi Xose, thanks for all your work. I've been using this firmware since the early versions and its incredible how far you've taken it!

Anyway, do you have any plans to add support for user-developed plugins? The reason is I have a very specific integration in mind that would most likely be rejected as a PR (since it's so specific).

[xoseperez]

You have several "tools" to do so. There is the custom.h file to add custom configuration settings, the USE_EXTRA setting to force calling a certain setup method for your plugin and several callback register methods to bind your code to the main loop, MQTT, WIFI or the local broker (and get relay and sensor notifications).

Of course this might not be enough. But maybe you can start working on this and we can merge other integration tools you feel necessary.

[zafrirron]

Hi @nunofgs (cc @xoseperez)

I did like your idea! thanks. As Xose mentioned integration hooks are available and also public functions are accessible. So this may serve as template and integration model documentation to make it easier when integrating plugins (or 3rd party code). Please see the attached files, were I've created a plugin template/tool/model (all files includes inline help documentation for all possible options). The main idea (as you probably asked for) is to allow adding new functionality to espurna with zero to minimal code changes to espurna (just adding the new files and integrating by setting a build flag only).

The rest of this comment moved Wiki page (https://github.com/xoseperez/espurna/wiki/3rd-Party-Plugins) and maintained/updated there...files removed from this comment to prevent duplications

Files:

• custom.h - this is the generic espurna custom header include file (included by all.h based on USE_CUSTOM_H build flag) allows setting and overriding espurna core definitions. To this file I've added the plugin activation code and a single flag to each plugin to set if to include the plugin in the image file.

• plugin1.h - The plugin header file (should be placed in config folder) includes plugin specific defines. this header is included by custom.h based on the plugin include flag (INCLUDE_PLUGINx) to allow switching on/off multiple plugins.

• plugin.ino - The plugin template, to be placed in code folder. this template enables writing a plugin and use all espurna services and utils (I may missed some....but will improve).

Status:

In development phase, working, partially tested.

Limitations

• Due to current frontend structure, I did not add the HTML/JS/CSS files, since I did not want to change core files, so in case of frontend need for plugin, you need to add the plugin page, but all the web services hooks included in the template (don't forget to gulp).
• I'm working to add a generic hook system that will enable a more tightly coupled plugins that will get callbacks from different espurna internal hook points.

Template structure:

(All inline documented ) Like any other espurna module, includes Helper functions, setup and main loop:

• Helper functions you can copy/add any the template includes a single main loop sample function _pluginFunction(); with all possible services hooks, you can copy it. Other helper functions are utils used by setup or loop function (like set a relay or read sensor).
• Setup function (pluginxSetup()) called by extraSetup() defined in custom.h. extraSetup() is called by espura.ino setup function based on USE_EXTRA flag (this is the generic espurna 3rd party code integration hook).
• Plugin main loop called by espurna loop (via callback function registrations array). plugin loop is registered by pluginsetup function.

Template features:

(services enabled by the plugin template), most code is commented out, to allow uncomment/delete needed/unneeded code. The services are all public espurna functions to be used for the specific plugin needs logic and data.

• Main loop features
  • Get sensors data
  • Set relays
  • Factory reset
  • AP mode
  • System reset
  • Setter/Getter for plugin params
  • Broker publish
  • MQTT publish
  • DOMOTICZ publish
  • INFLUXDB publish
  • THINGSPEAK publish

Installation:

• Copy the two .h files into espurna/config folder
• Copy the .ino file to espurna folder
• Build with -DUSE_CUSTOM_H flag
• See on debug (web/serial/telnet) a loop counter (every PLUGIN_REPORT_EVERY param)
• Terminal command plugin1 0 will stop plugin execution and plugin1 1 start
• API command /plugin1?apikey=xxxxx&value=0 stop (value=1) start
• Read plugin1.ino and plugin1.h inline documentation and write your espurna magic

Please feel free to give any feedback/comment/suggestion

plugin1-v0.0.1.zip

[nunofgs]

Thanks @zafrirron, this looks great. I'll give this a shot and report back.

One suggestion I'd make is to somehow have a plugin loading/unloading system within the UI itself. This would allow users to simply flash (and keep up to date) the latest espurna and simply install the plugins afterward.

[zafrirron]

@nunofgs thanks for your feedback, I'm not sure if I understood you correctly, my plan was to enable using latest espurna version to enable creation of your personal version including your plugin. The plugin should include all your unique definitions (overriding defines and so... ) to become independent of new version. Since the loading is single image loading, I guess you meant activating/deactivating of plugin, this is definitely a good idea! another Item I'm working on is adding a generic hook system so inside any espurna module it will be possible to create a hook and in any plugin possible to register to this hook.

Currently my template enables to run a private ("3rd party") code the can use any espurna feature but does not change espurna core behaviour. I real plugin system also register to hooks and change/add functionality to core. for example if you want to run a code before espurna changes a relay status, you just register to 'RELAY_BEFORE' action hook with your callback function (assuming core espurna code include this hook as Xose mentioned).

[zafrirron]

Hi @nunofgs

Based on your comment I've updated the description and files to new version, including plugin activation/deactivation terminal and api options. see update text for installation and explanation. Loaded also as Wiki page here https://github.com/xoseperez/espurna/wiki/3rd-Party-Plugins

[xoseperez]

I like the idea a lot. I started moving thins around a couple of months ago to ease this kind of developments.

@zafrirron You said you are working on a "generic hook system". What is your idea? I guess you mean something closer to how the Register methods work (like mqttRegister) then the broker module, for instance.

Plugin UI is certainly a complication right now. I have foreseen several options until now but haven't decided for any of them:

• Includes in build time, handled by the gulp builder. This would be equivalent to split the current custom HTML, JS and CSS files and have a way to inline them using placeholders, maybe in different places of a base HTML file.
• A generic description language to add common objects to the web UI (tab, field, select,...). The plugin should then send a description object as a JSON object that the web UI would parse and render. There is something in this line in the BMX280Sensor, I called it "manifest".
• Send HTML code from the firmware, this code will be generated in C. I feel this is really ugly since it couples different application layers.

Have you thought about a plugin folder structure? It can be interesting to be able to isolate all plugin files under the same folder.

[zafrirron]

Ok,

@xoseperez See my new pull request regarding generic plugin hooks (a very very basic one to start with...sorry...), exactly as you described, a very simple code enables registration and execution of hooks. Regarding separate folder, correct I missed that one. all plugins should be under separate folder (I'll fix it in the template), I'm sure PIO framework can handle it, not sure about other development platforms that developers use and how they handle libraries in seperate folders. Regarding UI, I'm looking into a UI templating model, I saw some gulp implementations that include templating system so the frontend could be templated (I know you would like to save on image size...), this will also allow plugins UI. My general view on the architecture is that some of the very hard work you perform on integrating new sensors and new hardware platform, could be moved to plugins (like) model, and the main focus of the core development should be the backbone (clean, stable, lightweight, flexible scaleable) of core system/UI. A new board (or sensor) integration should be only downloading the proper files to a folder and build with proper flag (without new core version). As new boards will come to the market, the base platform and services and ease of integration will make the difference.
(I started this journey, since I wanted to integrate a new hardware and could no understand why you or me need to bother with core change, so custom.h was good starting point that allows that, but I had a different stupid STM type relay module so I needed a hook :-) and quiet serial port, now you can see were all my comments came from).
With that been said...I'm sure you will like the spare time it will give you to "rest" more now...(just referring some of the enhancements requests to plugins... ;-)), it will give more time to the esp32 porting...(hint, this new chip screams for espurna, and its starting to get more market attention...) and focus on how to make espurna to the next generation backbone of embedded iot devices.

[zafrirron]

Regarding separate plugin folder, it may be my frameworks knowledge limitations, but after reading PIO docs, forums and testing, it looks like the plugin header files could be moved to seperate folder. (PIO lib model refer to "/lib/plugin1" folder structure were you should put library code into) trying to put plugin code into same folder however creates a little mess since PIO knows to link lib folder files if header included from .ino files on "src_dir" folder. so for now I moved only the plugin headers to /lib/plugin(x) folder and will look into possible options (maybe one of the frameworks gurus will guide me...)

[xoseperez]

You mention it. I'm pretty sure it will work with PlatformIO but, again, 80% (to say a number) of ESPurna users use Arduino IDE which is far more limited, maybe not when it comes to build the firmware, but I'm afraid it does not support autoloading files (in tabs) in a recursive way. For newbies this would mean they will have to know these files exist and load them files manually and this add complexity...

[uberflyx]

Hi all. This plugin idea is great. I'm thinking of adding some extra sensors as plugins rather than, well, sensors. I have some thoughts: on the latest code, to use a plugin one has to include custom.h. I don't think this is the way to go as it confounds the point of custom.h. I think it will be better to create a plugins.h file which is included by all.h, and then the "user" can uncomment the plugins they wish in the plugins.h file. Otherwise, the architecture seems like it will work nicely.

[zafrirron]

Agree about the entry point, I had to select it for existing integration, custom.h was the only entry point. A more suitable way will be plugin.h (I'm working on improved structure for it), and change the USE_EXTRA to control the plugin.h inclusion (USE_PLUGIN?) this is existing integration point.

@uberflyx if you have some thoughts about folder structure for the plugins that will fly for all development frameworks, please share....(header files looks easy but plugin c code files looks little more trickier).

My personal view after further looking into the framework for espurna I'm sure that arduino sdk is not a proper development environment for a project on this size (I'm sure not surprising you) and espurna should be bounded/limited by arduino. But also the PlatformIO I found limited in its folder structure model. Looking forward I do think the file/folder structure of this project should be changed with PlatformIO as main target (I'm checking the needed adjustments). Arduino IDE users/developers should be able to switch from Arduino SDK.

My vied on folder structure: |-code | ----|-Config |---------|-all-core-related-config-headers |----|-Data (/static ... what is the difference?) |----|-Sensors |--------|-SensorA |------------|-SensorA.h |------------|-SensorA.ino |--------|-SensorB |----|-Boards |--------|-BoardA |------------|-BoardA.h (board configuration here not in hardware.h) |------------|-BoardA.ino |--------|-BoardB |----|-Services |--------|-ServiceA (MQTT) |--------|-ServiceB (HomeAssistance) |--------|-ServiceC (Telnet) |----|-Html |----|-Lib |--------|-all-helper-utils-files |----|-Plugins |--------|-Plugin1 |------------|-Plugin1.h |------------|-Plugin1.ino |--------|-Plugin2 |-main.ino |-other-core-code.ino |-doc

So adding board or sensor should be dropping the proper folder/file to place and setting proper build flags to include the code (handled by core config files). Plugins also should be dropped to Plugins folder and included by flags or core config.

So core espurna configuration should be:

1. board selection (all board config in board folder)
2. Sensors selection (...)
3. Core services activation (WEB, TELNET, MQTT...)
4. Plugin activation
5. Other core specific settings.

A proper build tool should accommodate such project

BTW - I found the bug preventing platformio from processing arduino style project from using subfolders, opened a bug and hope for the best. (its related to src_filter once fixed we can move to a better folder structure just by adding SRC_FILTER flags for the subfolders and the files will be included as they were in the espurna SRC_FOLDER. in the future moving from arduino model (from .ino to .cpp) has different implications....

[Misiu]

I'm using ESPurna as an awesome code base, having plugin system and scheduler would allow me to drop my custom code and focus on supporting ESPurna. I need simple temperature logging to SD card every X minutes. Plugin system would allow me to create simple 2 files instead of hacking whole source code.

GULP is already used so maybe it can be reused when creating Web UI for specific plugins? In my case I need to be able to specify minutes between writes to SD and be able to download file that contains data.

Looking forward for plugin support 👍

[zafrirron]

@Misiu First iteration (MVP?) for plugin option described here.. https://github.com/xoseperez/espurna/wiki/3rd-Party-Plugins

This very first version doesn't include UI yet, but you could try to use it and have your parameters injected thru API (included in the template) or terminal command...

I'm working on next version, that will include web support and some other features, but it will take some time...

[gn0st1c]

@zafrirron: how about using gulp-inject?

you can have placeholders for plugin's html/css/js codes and if the plugin folder has any of the file(s), they will be injected. if there is no file, it will be skipped and remarks will be removed anyways.

https://www.npmjs.com/package/gulp-inject#injecting-files-contents

inside index.html:
  <!-- inject:espurna/plugins/plugin1/plugin_menu.html -->
  <!-- endinject -->
  <!-- inject:espurna/plugins/plugin2/plugin_menu.html -->
  <!-- endinject -->

  <!-- inject:espurna/plugins/plugin1/plugin_panel.html -->
  <!-- endinject -->
  <!-- inject:espurna/plugins/plugin2/plugin_panel.html -->
  <!-- endinject -->

inside custom.css:
  <!-- inject:espurna/plugins/plugin1/plugin.css -->
  <!-- endinject -->
  <!-- inject:espurna/plugins/plugin2/plugin.css -->
  <!-- endinject -->

custom.js:
  <!-- inject:espurna/plugins/plugin1/plugin.js -->
  <!-- endinject -->
  <!-- inject:espurna/plugins/plugin2/plugin.js -->
  <!-- endinject -->

template UI: plugin_UI.zip

things to consider:

• a bad html/css/js may screw up the whole UI
• tabindex

[gn0st1c]

@zafrirron : there were some minor typos inside plugin_0.0.1.zip's ino file. if you'd like to merge to yours; plugin1.ino.zip

[zafrirron]

@nunofgs Thanks! The UI solution sounds like a great option... I will try it later today and add it to the wiki page...

My other thought about UI, was to try to add templating engine to the espurna core itself... In addition to allow the plugins additions it will also add more flexibility to espurna core by enabling inclusion/exclusion of core elements based on annapurna configuration (saving on html size....). while looking into some gulp templating packages, I'm struggling with translating (dynamically) espurna's c header files (defining the core modules) to the template variables...

any thoughts?

[mcspr]

@gn0st1c Wouldn't htmllint part of gulp chain cover that? It did break build for me with invalid tags (not closed, slash position etc.) altough it was not really helpful showing actual error. Similar linter for resulting js could be added. And tabindex could be generated on panel load by js itself.

@zafrirron What did you try? It is not required to use node though, my personal preference would be python+jinja2. For example, platformio uses SCons.Scanner - see platformio-core/builder/tools/piolib.py - and using that data extra_script could render resulting html via pre-action on espurna.ino.cpp.o target: extra_script called -> render index.html -> call gulp -> package .gz.h

Upd Because it is scons, index.html itself could be added as build target and all relevant parts could be there. Rendering, packaging into gzip, converting to binary header.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 30 days if no further activity occurs. Thank you for your contributions.

[lblabr]

is there some progress in here ?