search

cwi-dis / iotsa

This library contains a framework to easily create web servers that can interface to all sorts of sensors and actuators. esp32 and esp8266.
MIT License
28 stars 2 forks source link
arduino-library esp8266-arduino iot iotsa

readme

iotsa - Internet of Things server architecture

build-platformio build-arduino

This library contains a framework to easily create esp8266-based or esp32-based web servers that can interface to all sorts of sensors and actuators. The servers can be REST-compatible, and COAP-compatible, including seamless integration with the Igor home automation server.

Home page is https://github.com/cwi-dis/iotsa. This software is licensed under the MIT license by the CWI DIS group, http://www.dis.cwi.nl.

Note that the build badges above are for the develop branch, for the iotsa library itself and the examples. Build status for iotsa applications against the develop branch can be found below.

Installation and use for developers

Iotsa can be used with both the Arduino IDE and with the PlatformIO build system (which can be used from within Atom or VSCode or from the command line). The Arduino IDE is easiest to get started with, but PlatformIO is more powerful if you want to target multiple device types, use Git integration, etc.

Arduino IDE

Download the zipfile (via https://github.com/cwi-dis/iotsa) and install into Arduino IDE with Sketch -> Include Library -> Add .ZIP Library... . (If you downloaded the zipfile through github you may have to rename the directory because it may be called something like _iotsa__master).

Build the Hello example (File -> Examples -> iotsa -> Hello) and flash it onto an ESP-12 or similar board.

Note: the structure of the iotsa examples may be slightly different than what you are used to for Arduino examples: the Hello.ino file is basically empty, and the code is contained in the mainHello.cpp file, which is in a separate tab in the Arduino IDE.

PlatformIO

The iotsa library is known to the PlatformIO library manager, so simply adding it to your platformio.ini file should do the trick for adding the iotsa framework to your project..

To build an example you can open the iotsa source directory in VSCode or Atom and look at the [env:nodemcuv2-example-skeleton] section. Replace the references to skeleton with the name of the example you want to build.

But: each example in the examples folder also has its own platform.ini file to build it. So you can also open iotsa/examples/Hello in VSCode or Atom and build it, or use the command line:

$ cd iotsa/examples/Hello
$ platformio run --target build
$ platformio run --target upload

Both Arduino IDE and PlatformIO

On reboot, the board will first initialize the SPIFFS flash filesystem (if needed) and then create a WiFi network with a name similar to config-iotsa1234. Connect a device to that network and visit http://192.168.4.1. Configure your device name (at http://192.168.4.1/config) and WiFi name and password (at http://192.168.4.1/wificonfig), and after reboot the iotsa board should connect to your network and be visible as http://yourdevicename.local.

When the device is running normally you can visit http://yourdevicename.local/config and request the device to go into configuration mode, or to do a factory reset. After requesting this you have 5 minutes to power cycle the device to make it go into configuration mode again (see previous paragraph) or do a complete factory reset. When in configuration mode you have five minutes to change the configuration (device name, WiFi name, password, maybe other parameters) before the device reverts to normal operation. The idea behind this sequence (request configuration mode, then power cycle, then change parameters) is that you need both network acccess and physical access before you can do a disruptive operation on the device.

Build time options

A number of features of the iotsa framework can be enabled and disabled selectively at build time. These features are encoded in iotsaBuildOptions.h.

The general naming convention is that a feature WIFI will be triggered by a define IOTSA_WITH_WIFI. If the WIFI feature is enabled by default (which happens to be the case for WIFI, obviously) that can be overridden by defining IOTSA_WITHOUT_WIFI at compile time of the iotsa library.

When using the Arduino IDE you can edit this file to change the options, by commenting out the defines for features you do not need.

When using Platformio you can also use the build_flags in platformio.ini. For example, to enable COAP and disable REST:

    build_flags = -DIOTSA_WITHOUT_REST -DIOTSA_WITH_COAP

The following features are defined:

There are a few more that are not very important, please inspect iotsaBuildOptions.h.

HTTPS support

If a iotsa application is built with IOTSA_WITH_HTTPS if will initially use a builtin key and certificate. This is NOT secure, because the key and certificate are contained in the github source repository, and therefore known to the world.

After building and flashing your software for the first time you should create a new, unique, key and certificate. There are three ways to do this, using scripts in the extras directory:

Note that HTTPS support here refers to iotsa as a server only, HTTPS client support (for the iotsaButton and iotsaRequest modules) is completely independent.

Controlling iotsa devices

Command line tool

There is a command line helper program iotsa. It will be documented here at some point. For now, install iotsa with the following commands:

cd extras/python
pip3 install .

Then you can get a list of the available commands with iotsa help and a list of the available options with iotsa --help.

OTA programming

If you have enabled over-the-air programming http://yourdevicename.local/config will also allow you to request the device to go into programmable mode. Again, you have 5 minutes to power cycle and then 5 minutes to reprogram:

General design philosophy

Unlike most Arduino libraries and frameworks Iotsa does expose some of its C++ interfaces, but for simple applications you do not have to worry about this.

Iotsa has two main types of objects:

Do-nothing application

You create a global variable application of type IotsaApplication to hold the basic implementation of your service framework, plus the ESP8266WebServer object on which the application will serve. You also create one IotsaWifiMod and link it to the application so the end user can configure the WiFi network to join and such:

IotsaApplication application("Iotsa Hello World Server");
IotsaWifiMod wifiMod(application);

In your setup() function you call app.setup() and app.serverSetup() which will initialize the Iotsa framework:

void setup(void){
  application.setup();
  application.serverSetup();
  // Add your own setup code here....
}

In your loop() function you call app.loop(). This will take care of handling requests and every thing else that is needed to make the framework work:

void loop(void){
  application.loop();
  // Add your own loop code here....
 }

You have now created a Iotsa server that does absolutely nothing.

The Skeleton example does exactly this (but there are various #define statements at the top that you either turn into #undef, or keep to give you extra functionality).

Hello world application

One step more functionality is provided by the Hello example. It registers a handler function for the /hello URL (which provides a form through which a user can type in his or her name) and an info function that returns a paragraph (or more) on the home page /:

void
helloHandler() {
  // Parse form arguments using server->args() and server->arg(...).
  String message = "<html>...construct html page...</html>";
  server->send(200, "text/html", message);
}

String helloInfo() {
  String rv = "<p>See <a href=\"/hello\">/hello</a> for info.</p>";
  return rv;
}

These two functions (and the URL) are registered in the application as a IotsaSimpleMod:

IotsaSimpleMod helloMod(application, "/hello", helloHandler, helloInfo);

Another hello world application

From a functionality point of view the HelloCpp program is identical to Hello, but it is structured differently, as a C++ class. This is a bit more difficult to read (when you are used to standard Arduino programming and not C++ programming) but has the advantage that the functionality can easily be reused in other servers. Actually, most standard modules (below) started their life this way. Also, if you want to create a device without a user interface (REST or COAP only) you should follow this pattern.

You declare the class for your functionality:

class IotsaHelloMod : public IotsaMod {
public:
  IotsaHelloMod(IotsaApplication &_app) : IotsaMod(_app) {}
  void setup();
  void serverSetup();
  void loop();
  String info();
private:
  void handler();
};

Then you implement the 5 methods, of which only serverSetup needs explanation: it is called when this module is added to the application, and should register the URL:

void IotsaHelloMod::serverSetup() {
  // Setup the web server hooks for this module.
  server->on("/hello", std::bind(&IotsaHelloMod::handler, this));
}

Finally you create a single object of your new IotsaHelloMod type and register it to the application:

IotsaHelloMod helloMod(application);

Iotsa will now take care of calling your classes IotsaHelloMod ::setup(), IotsaHelloMod ::serverSetup() and IotsaHelloMod ::loop() methods at the right time without you needing to add any code to the normal setup() and loop() functions. Which means that this implementation of Hello can be combined with as many other modules as you want, just by adding that 1-line declaration.

Hello World with an API

If you want your server to have a programming interface (either REST to access it over HTTP/TCP or COAP to access it over UDP) you use the base class IotsaApiMod. You provide methods for handling GET, PUT and POST requests (only for the ones you need) and register your API endpoint in your serverSetup.

The HelloApi example has the full details, but here are the GET and PUT methods:

bool IotsaHelloMod::getHandler(const char *path, JsonObject& reply) {
  reply["greeting"] = greeting;
  return true;
}

bool IotsaHelloMod::putHandler(const char *path, const JsonVariant& request, JsonObject& reply) {
  JsonVariant arg = request["greeting"];
  if (arg.is<char*>()) {
    greeting = arg.as<String>();
    return true;
  }
  return false;
}

And here is the modified serverSetup method:

void IotsaHelloMod::serverSetup() {
  // Setup the web server hooks for this module.
  server->on("/hello", std::bind(&IotsaHelloMod::handler, this));
  api.setup("/api/hello", true, true);
  name = "hello";
}

HTML encoding

Strings that are interpolated into the HTML returned from the info() or handler() function must be ampersand-encoded. There is a static method in IotsaMod to do this for you:

String IotsaMod::htmlEncode(String data);

Next steps

The Hello and HelloCpp examples shows how to do basic interaction with the user (through a browser form, and through information on the how page). HelloApi shows how to add an API to your service. Various types of access control are demonstrated in the HelloUser, HelloPasswd, HelloRights and HelloToken examples.

To use this to create an interface to some bit of sensor hardware simply add the usual code to your setup() and loop() functions, and store the sensor value that you read in your loop() in a global variable. Pick up the value of this variable in your handler or info function and format it in HTML. Light and Temperature are example programs of this type.

To interface to an actuator you present an HTML form in your handler, and store the user-supplied value in a global variable. Your setup() and loop() functions are again as usual, but in loop() you pick up the value from the global variable. Led is an example program of this type.

Slightly more elaborate API examples are Button and Ringer. Button waits for a button to be pressed on the device and then makes a call to a user-specifyable URL. If that URL happens to point to Ringer it will in turn sound a buzzer.

User interface and operation

Core API

iotsa.h

This file declares the IotsaApplication and IotsaMod classes explained earlier. In addition it declares a class IotsaAuthMod which is a subclass of IotsaMod (so it has all the functionality of a normal module, like the handler) but it can be used as the authenticator for another module. This allows the other module to use access control: it will only work after the user has provided a username/password combination, or pressed a certain button (probably key-operated) or any other means of authentication you can think of.

Here are the constructors of the three classes:

IotsaApplication(const char *_title);
IotsaMod(IotsaApplication &_app, IotsaAuthMod *_auth=NULL, bool early=false);
IotsaAuthMod(IotsaApplication &_app, IotsaAuthMod *_auth=NULL, bool early=false);

The optional early argument signifies that the module should be initialized early, this is generally used only by the WiFi module.

iotsaApi.h

Contains a class IotsaApiMod, a subclass of IotsaMod. Subclass this to implementing modules that also provide a REST interface. The class provides a method to register your API endpoint:

void apiSetup(const char* path, bool get=false, bool put=false, bool post=false);

Call this in your serverSetup() method. Call multiple times if your class implements multiple endpoints. You should also provide the following methods to implement your GET, PUT and POST methods (for the ones for which you passed true in your apiSetup call):

bool getHandler(const char *path, JsonObject& reply);
bool putHandler(const char *path, const JsonVariant& request, JsonObject& reply);
bool postHandler(const char *path, const JsonVariant& request, JsonObject& reply);

These methods are called on incoming REST or COAP calls. The request parameter object (or array, or value) is in request, store results in reply (which is already an initialized empty object). Return true for success, false for any failure (which will not return the reply object to the caller).

The actual implementations are in iotsaRestApi.h and iotsaCoapApi.h, but these are transparently renamed or combined.

iotsaConfig.h

Handles the iotsaConfig object, which is a singleton object that contains all configuration information, such as hostname and which mode we are running in. You normally don't include this file, it is included through iotsa.h.

iotsaConfigMod.h

Handles control over the iotsaConfig object and general configuration such as factory reset and (if enabled) over-the-air programming. This is technically a module, but unlike other modules it is not really optional (unless your iotsa board is running with WiFi disabled). It should not be instantiated in your program, this happens automatically. This module also opens and/or initializes the SPIFFS filesystem.

If the iotsa server is operating in normal (production) mode a user can access URL /config to request configuration mode, factory reset or ota-programming mode. The user must then demonstrate physical access within 5 minutes to switch the iotsa server to its new mode, for another 5 minutes. If nothing happens during this period the server reverts to normal mode.

The intention of demonstrating physical access is that any "dangerous" operation requires both network access (to request the operation) and physical access. The user can always turn power on and off to demonstrate physical access, but a iotsa program can allow different methods (such as pressing a certain button a number of times in quick succession).

On a factory reset all configuration information (literally: all) is forgotten and the iotsa device is completely new again.

In ota-programming mode the device functions normally, but can also be reprogrammed using the Arduino IDE and its OTA facilities.

The module also provides a REST api on /api/config (and this api depends on whether in configuration mode or not).

iotsaWifi.h

Handles WiFi configuration. It is possible to build a Iotsa application without this module, but except for special use cases such as creating a Bluetooth LE only service it is not recommended. You must instantiate it in your program.

A iotsa device can join an existing WiFi network (normal WiFi mode) or create a temporary network as an Access Point (private WiFi mode). In private mode the device does not connect to a WiFi network, but in stead creates its own network (as a base station) with a name starting with "config-". The user can now connect a device to this network and visit http://192.168.4.1/wificonfig. Here it is possible to set the normal WiFi network to connect to and the password.

If a device is new (it has no WiFi network name) it will enter private mode automatically. If a device cannot find its configured WiFi network it will enter private mode for 5 minutes and then reboot and retry joining the configured network.

If a device has a WiFi network configured but it cannot join this network after trying for 5 minutes it will also enable private network mode.

If a device is in normal WiFi mode the WiFi parameters can only be changed if the device is in configuration mode.

The module also provides a REST api on /api/wificonfig (and this api depends on whether in configuration mode or not).

The module can be disabled by building with IOTSA_WITHOUT_WIFI).

iotsaSimple.h

Allows implementing a module using two simple C functions. See the hello section above for details.

iotsaConfigFile.h

Two classes to save and load configuration variables to a file (on the SPIFFS file system in the flash memory chip). IotsaConfigFileLoad is used to load configuration values and IotsaConfigFileSave to store them.

The general paradigm is that in your setup() function you create a local (stack based) loader variable with 

IotsaConfigFileLoad cf("/config/filename.cfg");

and then repeatedly call cf.get("name", variable, default) for each configuration variable. Variables can be of type int, float or String.

To save configuration values you call

IotsaConfigFileSave cf("/config/filename.cfg");

and then repeatedly call cf.put("name", variable).

available standard modules

iotsaBattery.h

Allows sleeping, hibernating and shutting down WiFi of your iotsa device to conserve battery power.

iotsaBLEServer.h

Allows a iotsa device to export an API as a Bluetooth LE service. ESP32 only. Can be used with the module (which allows setting some BLE parameters such as advertising interval through the web or REST interface) or without it.

Note: the current Bluetooth LE implementation uses a lot of RAM, about 160KB. Therefore when you use Bluetooth you are probably not able to use OTA, HTTPS and various other features. This is expected to be addressed in a future version of the ESP32 Arduino framework.

Note that this file can be included if BLE is not supported (such as on an esp8266), in which case it will only declare an empty class iotsaBLEApiProvider. This helps a bit with #ifdef.

iotsaButton.h

A generalized module for handling buttons and switches attached to GPIO pins. Uses iotsaRequest.h to enable REST calls when buttons are pressed. Also see iotsaInput.h with which this will eventually be integrated.

iotsaCapabilities.h

An authentication module (IotsaAuthMod) that trusts an external server, the issuer, to determine access rights to this board.

The issuer and this board share a secret key.

More documentation will be forthcoming.

iotsaFiles.h

Allows read access to files stored in /data on the SPIFFS file system (in the flash memory chip). Could be used for a simple web server. Requires IOTSA_WITH_WEB.

iotsaFilesUpload.h

Allows write access to files stored in /data on the SPIFFS filesystem (where iotsaFiles reads from), through POST requests to the /upload URL. Requires IOTSA_WITH_WEB.

iotsaFilesBackup.h

Creates a backup of the complete SPIFFS filesystem (including /data and /config) as a tarfile when you access URL /backup.tar. Can be used to clone iotsa devices. Requires IOTSA_WITH_WEB.

iotsaInput.h

Handles buttons, rotary encoders and (on ESP32) capacitive touchpads for input. Allows for callbacks to be made on changes, and (on ESP32) wakeup from sleep. Will be integrated with iotsaButton.h at some point in the future.

iotsaLed.h

Allows showing static colors and repeating patterns on a NeoPixel LED. By default does not provide a web interface, only an API set(rgb, onDuration, offDuration, count) for use in your program. But see the Led example for providing a web interface.

The iotsaLed module also implements the iotsaStatusInterface protocol, and shows status information during the boot sequence and when the iotsa board is running in a nonstandard mode (configuration mode, OTA mode, etc).

The module does not provide a user-visible endpoint or REST api, but can be used as a base class for this. See examples/Led for an example.

iotsaLogger.h

Replaces the standard Serial object by an object that stores data in a memory buffer, and allows access to that memory buffer through the URL /logger. The buffer is tiny, 4Kb, but this allows some limited debugging over WiFi. Requires IOTSA_WITH_HTTP or IOTSA_WITH_HTTPS.

iotsaNothing.h

A module that does nothing. Use this as the basis of your own modules. Provides a user interface at /nothing and a REST interface on /api/nothing.

iotsaNtp.h

A module that contacts an NTP server to set the local clock. Accessing URL /ntpconfig allows setting the NTP server to use, as well as the timezone.

The modules sets the libc time, so functions like time() and localtime() will work with the correct time.

The module provides an API to get access to the time information:

unsigned long utcTime();    // Seconds since 1-Jan-1970, UTC (unix time)
unsigned long localTime();  // Seconds since 1-Jan-1970, local time zone
int localSeconds();         // Local time-of-day, seconds (0-59)
int localMinutes();         // Local time-of-day, minutes (0-59)
int localHours();           // Local time-of-day, hours (0-23)
int localHours12();         // Local time-of-day, hours (0-11)
bool localIsPM();           // AM/PM indicator

Provides a user interface at /ntp and a REST interface at /api/ntp.

iotsaRtc.h

Support for DS1302 realtime clock chip, which usually comes on a board with battery backup so it can run for years. Will set libc clock from RTC shortly after boot, and periodically set RTC clock from the system clock. So, when used together with iotsaNtp this automatically uses the RTC time when no network is available, and resynchronizes the RTC when a network is available.

iotsaOta.h

Allows Over-the-air reprogramming of a iotsa server. After ota-programming has been enabled the device will show up (for 5 minutes) in the Arduino IDE, menu Tools -> Port, under the Network Ports section. Select it, and press the checkmark on your sketch to upload. Requires IOTSA_WITH_HTTP or IOTSA_WITH_HTTPS.

iotsaRequest.h

Allows issuing of http and https requests. Carrying username/password credentials and tokens is possible. This is not a complete module in itself, but provides the building blocks to allow other modules to easily add the ability to send (configurable) web requests.

iotsaUser.h

An authentication module (IotsaAuthMod) that stores a single username and password. Other modules can then specify they are only accessible after the user authenticates with this username/password combination.

Accessing URL /users allows changing the password, and if the password has never been set the default password is shown whenever the device is booted in configuration mode.

iotsaMultiUser.h

An authentication module (IotsaAuthMod) that stores a multiple usernames, passwords and rights. Other modules can then specify they are only accessible after the user authenticates with a username/password combination that has the correct right in its set of rights.

Accessing URL /users allows adding and changing users. A factory-installed device has no users, and allows all rights always.

sample programs

more projects using iotsa

Here are some projects that use iotsa, and that also be used as further examples (Note that as of this writing not all projects may be publicly accessible yet):

lissabon

build-platformio

This is a whole set of appliances aimed at very low power overhead control of lighting in an off-grid situation (think: battery-backed solar power holiday home). Dimmers, LED strip controllers, remotes.

iotsaGPIO

build-platformio build-arduino

Allows web access (or REST access) to analog and digital input and output pins.

iotsaDisplayServer

build-platformio build-arduino

Drives an OLED display, such as an i2c 4x40 character module. Support for a buzzer (to attract user attention) and buttons (programmable to trigger actions by accessing programmable URLs) is included.

iotsaOLED

build-platformio build-arduino

Drives an LCD display, such as an Adafruit SSD1306. Same external API as iotsaDisplayServer, but also allows Bluetooth LE control.

iotsaMotorServer

build-platformio build-arduino

Drives one or more stepper motors. Schematics and 3D models are included for a device to lift an object (such as a plant in a pot) to a height that can be changed through the web.

iotsaNeoClock

build-platformio build-arduino

A clock comprised of 60 NeoPixel LEDs. Shows the time, but can also show programmable patterns (as alerts) and temporal information (such as expected rainfall for the coming hour). Schematics and building instructions included.

iotsaDoorOpener

build-platformio build-arduino

Operates a solenoid to open a door. On web access, or when an RFID tag (such as a keychain fob or a mifare contactless transport card) is presented. RFID cards are programmable (over the net, or using a special "learn" card). A web request can be sent to a programmable URL when a card is presented.

iotsaSmartMeter

build-platformio build-arduino

Reads electricity and gas usage of a dutch Smart Meter through the standardised P1 port and makes the data available on the net.

iotsaDoorbellButton

build-platformio build-arduino

Very simple server that emits a GET request when a button is pressed. Uses capabilities.

iotsaDoorbellRinger

build-platformio build-arduino

Very simple server that sounds a buzzer when a GET request is received. Uses capabilities.

iotsaDMXLedstrip

build-platformio build-arduino

A driver for NeoPixel or similar LED strips to use for theatre lighting. The individual LEDs are controllable using the Art-Net DMX protocol, which is supported by many theatre lighting consoles or software such as the open source QLC+.

iotsaDMXSensor

build-platformio build-arduino

Reads sensor values from Estimote Bluetooth LE sensors and makes these available as DMX slider values (using DMX over Art-Net over WiFi). Can be used to input sensor values into a standard theatre lighting console or software such as QLC+.

iotsa433

build-platformio build-arduino

Server to receive and transmit 433 MHz home automation signals (the cheap wireless remote controlled sockets, dimmers, garage doors, etc).

iotsaSensor

build-platformio build-arduino

Read an analog sensor at an interval that can be specified. Sensor readings are kept in a rolling buffer, and the most recent 512 readings can be retreived through the web interface, together with the timestamp of reading. This program is intended to take readings every couple of seconds, or minutes at most.

iotsaDataLogger

build-platformio build-arduino

Read an analog sensor at an interval that can be specified. Sensor readings are saved to flash memory, together with the timestamp of reading. A realtime clock (RTC) can be used to keep time, which will sync with NTP whever WiFi is available. The program can hibernate (using micro-amperes) between readings. This program is intended to take a few measurements per day over long periods of time (months).

iotsaRGBWSensor

build-platformio build-arduino

Read current ambient light level (RGB and White, color temperature) using a VEML6040 sensor. Can be used to calibrate RGB, RGBW and RGBWW led strips.

hardware

Folder extras/fritzing contains design (circuit and PCB, and partial breadboard layout) for a PCB based on an ESP-12 that makes a nice iotsa hardware platform. You need the open source Fritzing tool to open these files.

The iotsa board has the ESP-12, a 3.3v regulator (so it can be powered with a standard 5v-16v power supply) and a free section with enough space for a DIP IC (up to DIP-20) and/or a few discrete components. All easily usable GPIO pins are also available. There's also room for an FTDI header (so you can reprogram the board if you have bricked it with over-the-air programming) and reset and program buttons.

The resistors are all pullups and pulldowns, so their values are not very critical. With one exception: if you use an ESP-12S module R3 (the program button pulldown) must be around 3K3, a 10K resistor will not work with the ESP-12S (as was found experimentally, since 10K works just fine with an ESP-12F).

The v3 board is about 6.35x4.35cm in size, with the ESP-12 antenna sticking out 0.5cm.

There is no board for the ESP32, but iotsa is known to work with many standard esp32 boards such as the Lolin boards or the Esp32Thing. There are too many different esp32 boards with different features sets that are useful for different projects...

Case

Extras also has a file extras/iotsaCase.scad that contains the OpenSCAD source code to 3D-print your own box to fit a iotsa board. Examine the source, there are various ways to adjust the design (to add extra holes, or determine thickness of the box, or make the box fit a iotsa board with some of the experimental area rows removed.

The 3D-printable case for a iotsa board is also available at Thingyverse: http://www.thingiverse.com/thing:2303793.

Creating a new release

Notes to self, mainly, on creating a new release: