search

hoeken / PsychicHttp

Simple + Robust HTTP/S server with websockets for ESP32 based on ESP-IDF http server.
MIT License
112 stars 27 forks source link
esp32 esp32-ardunio http-server https-server websockets-server

readme

PsychicHttp - HTTP on your ESP 🧙🔮

PsychicHttp is a webserver library for ESP32 + Arduino framework which uses the ESP-IDF HTTP Server library under the hood. It is written in a similar style to the Arduino WebServer, ESPAsyncWebServer, and ArduinoMongoose libraries to make writing code simple and porting from those other libraries straightforward.

Features

Differences from ESPAsyncWebserver

Usage

Installation

Platformio

PlatformIO is an open source ecosystem for IoT development.

Add "PsychicHttp" to project using Project Configuration File platformio.ini and lib_deps option:

[env:myboard]
platform = espressif...
board = ...
framework = arduino

# using the latest stable version
lib_deps = hoeken/PsychicHttp

# or using GIT Url (the latest development version)
lib_deps = https://github.com/hoeken/PsychicHttp

Installation - Arduino

Open Tools -> Manage Libraries... and search for PsychicHttp.

Principles of Operation

Things to Note

PsychicHttp

Request Life Cycle

Flowchart of Request Lifecycle

Handlers

Flowchart of Request Lifecycle

Responses and how do they work

Porting From ESPAsyncWebserver

If you have existing code using ESPAsyncWebserver, you will feel right at home with PsychicHttp. Even if internally it is much different, the external interface is very similar. Some things are mostly cosmetic, like different class names and callback definitions. A few things might require a bit more in-depth approach. If you're porting your code and run into issues that aren't covered here, please post and issue.

Globals Stuff

setup() Stuff

Requests / Responses

Usage

Create the Server

Here is an example of the typical server setup:

#include <PsychicHttp.h>
PsychicHttpServer server;

void setup()
{
   //optional low level setup server config stuff here.
   //server.config is an ESP-IDF httpd_config struct
   //see: https://docs.espressif.com/projects/esp-idf/en/v4.4.6/esp32/api-reference/protocols/esp_http_server.html#_CPPv412httpd_config
   //increase maximum number of uri endpoint handlers (.on() calls)
   server.config.max_uri_handlers = 20; 

   //connect to wifi

   //start the server listening on port 80 (standard HTTP port)
   server.listen(80);

   //call server methods to attach endpoints and handlers
   server.on(...);
   server.serveStatic(...);
   server.attachHandler(...);
}

Add Handlers

One major difference from ESPAsyncWebserver is that handlers can be attached to a specific url (endpoint) or as a global handler. The reason for this, is that attaching to a specific URL is more efficient and makes for cleaner code.

Endpoint Handlers

An endpoint is basically just the URL path (eg. /path/to/file) without any query string. The server.on(...) function is a convenience function for creating endpoints and attaching a handler to them. There are two main styles: attaching a basic WebRequest handler and attaching an external handler.

//creates a basic PsychicWebHandler that calls the request_callback callback
server.on("/url", HTTP_GET, request_callback);

//same as above, but defaults to HTTP_GET
server.on("/url", request_callback);

//attaches a websocket handler to /ws
PsychicWebSocketHandler websocketHandler;
server.on("/ws", &websocketHandler);

The server.on(...) returns a pointer to the endpoint, which can be used to call various functions like setHandler(), setFilter(), and setAuthentication().

//respond to /url only from requests to the AP
server.on("/url", HTTP_GET, request_callback)->setFilter(ON_AP_FILTER);

//require authentication on /url
server.on("/url", HTTP_GET, request_callback)->setAuthentication("user", "pass");

//attach websocket handler to /ws
PsychicWebSocketHandler websocketHandler;
server.on("/ws")->attachHandler(&websocketHandler);

Basic Requests

The PsychicWebHandler class is for handling standard web requests. It provides a single callback: onRequest(). This callback is called when the handler receives a valid HTTP request.

One major difference from ESPAsyncWebserver is that this callback needs to return an esp_err_t variable to let the server know the result of processing the request. The response->reply() and request->send() functions will return this. It is a good habit to return the result of these functions as sending the response will close the connection.

The function definition for the onRequest callback is:

esp_err_t function_name(PsychicRequest *request);

Here is a simple example that sends back the client's IP on the URL /ip

server.on("/ip", [](PsychicRequest *request)
{
   String output = "Your IP is: " + request->client()->remoteIP().toString();
   return request->reply(output.c_str());
});

Uploads

The PsychicUploadHandler class is for handling uploads, both large POST bodies and multipart encoded forms. It provides two callbacks: onUpload() and onRequest().

onUpload(...) is called when there is new data. This function may be called multiple times so that you can process the data in chunks. The function definition for the onUpload callback is:

esp_err_t function_name(PsychicRequest *request, const String& filename, uint64_t index, uint8_t *data, size_t len, bool final);

onRequest(...) is called after the successful handling of the upload. Its definition and usage is the same as the basic request example as above.

Basic Upload (file is the entire POST body)

It's worth noting that there is no standard way of passing in a filename for this method, so the handler attempts to guess the filename with the following methods:

//handle a very basic upload as post body
 PsychicUploadHandler *uploadHandler = new PsychicUploadHandler();
 uploadHandler->onUpload([](PsychicRequest *request, const String& filename, uint64_t index, uint8_t *data, size_t len, bool last) {
   File file;
   String path = "/www/" + filename;

   Serial.printf("Writing %d/%d bytes to: %s\n", (int)index+(int)len, request->contentLength(), path.c_str());

   if (last)
     Serial.printf("%s is finished. Total bytes: %d\n", path.c_str(), (int)index+(int)len);

   //our first call?
   if (!index)
     file = LittleFS.open(path, FILE_WRITE);
   else
     file = LittleFS.open(path, FILE_APPEND);

   if(!file) {
     Serial.println("Failed to open file");
     return ESP_FAIL;
   }

   if(!file.write(data, len)) {
     Serial.println("Write failed");
     return ESP_FAIL;
   }

   return ESP_OK;
 });

 //gets called after upload has been handled
 uploadHandler->onRequest([](PsychicRequest *request)
 {
   String url = "/" + request->getFilename();
   String output = "<a href=\"" + url + "\">" + url + "</a>";

   return request->reply(output.c_str());
 });

 //wildcard basic file upload - POST to /upload/filename.ext
 server.on("/upload/*", HTTP_POST, uploadHandler);

Multipart Upload

Very similar to the basic upload, with 2 key differences:

 //a little bit more complicated multipart form
 PsychicUploadHandler *multipartHandler = new PsychicUploadHandler();
 multipartHandler->onUpload([](PsychicRequest *request, const String& filename, uint64_t index, uint8_t *data, size_t len, bool last) {
   File file;
   String path = "/www/" + filename;

   //some progress over serial.
   Serial.printf("Writing %d bytes to: %s\n", (int)len, path.c_str());
   if (last)
     Serial.printf("%s is finished. Total bytes: %d\n", path.c_str(), (int)index+(int)len);

   //our first call?
   if (!index)
     file = LittleFS.open(path, FILE_WRITE);
   else
     file = LittleFS.open(path, FILE_APPEND);

   if(!file) {
     Serial.println("Failed to open file");
     return ESP_FAIL;
   }

   if(!file.write(data, len)) {
     Serial.println("Write failed");
     return ESP_FAIL;
   }

   return ESP_OK;
 });

 //gets called after upload has been handled
 multipartHandler->onRequest([](PsychicRequest *request)
 {
   PsychicWebParameter *file = request->getParam("file_upload");

   String url = "/" + file->value();
   String output;

   output += "<a href=\"" + url + "\">" + url + "</a><br/>\n";
   output += "Bytes: " + String(file->size()) + "<br/>\n";
   output += "Param 1: " + request->getParam("param1")->value() + "<br/>\n";
   output += "Param 2: " + request->getParam("param2")->value() + "<br/>\n";

   return request->reply(output.c_str());
 });

 //upload to /multipart url
 server.on("/multipart", HTTP_POST, multipartHandler);

Static File Serving

The PsychicStaticFileHandler is a special handler that does not provide any callbacks. It is used to serve a file or files from a specific directory in a filesystem to a directory on the webserver. The syntax is exactly the same as ESPAsyncWebserver. Anything that is derived from the FS class should work (eg. SPIFFS, LittleFS, SD, etc)

A couple important notes:

The server.serveStatic() function handles creating the handler and assigning it to the server:

//serve static files from LittleFS/www on / only to clients on same wifi network
//this is where our /index.html file lives
server.serveStatic("/", LittleFS, "/www/")->setFilter(ON_STA_FILTER);

//serve static files from LittleFS/www-ap on / only to clients on SoftAP
//this is where our /index.html file lives
server.serveStatic("/", LittleFS, "/www-ap/")->setFilter(ON_AP_FILTER);

//serve static files from LittleFS/img on /img
//it's more efficient to serve everything from a single www directory, but this is also possible.
server.serveStatic("/img", LittleFS, "/img/");

//you can also serve single files
server.serveStatic("/myfile.txt", LittleFS, "/custom.txt");

You could also theoretically use the file response directly:

server.on("/ip", [](PsychicRequest *request)
{
   String filename = "/path/to/file";
   PsychicFileResponse response(request, LittleFS, filename);

   return response.send();
});
PsychicFileResponse(PsychicRequest *request, FS &fs, const String& path)

Websockets

The PsychicWebSocketHandler class is for handling WebSocket connections. It provides 3 callbacks:

onOpen(...) is called when a new WebSocket client connects. onFrame(...) is called when a new WebSocket frame has arrived. onClose(...) is called when a new WebSocket client disconnects.

Here are the callback definitions:

void open_function(PsychicWebSocketClient *client);
esp_err_t frame_function(PsychicWebSocketRequest *request, httpd_ws_frame *frame);
void close_function(PsychicWebSocketClient *client);

WebSockets were the main reason for starting PsychicHttp, so they are well tested. They are also much simplified from the ESPAsyncWebserver style. You do not need to worry about error handling, partial frame assembly, PONG messages, etc. The onFrame() function is called when a complete frame has been received, and can handle frames up to the entire available heap size.

Here is a basic example of using WebSockets:

 //create our handler... note this should be located as a global or somewhere it wont go out of scope and be destroyed.
 PsychicWebSocketHandler websocketHandler();

 websocketHandler.onOpen([](PsychicWebSocketClient *client) {
   Serial.printf("[socket] connection #%u connected from %s\n", client->socket(), client->remoteIP().toString());
   client->sendMessage("Hello!");
 });

 websocketHandler.onFrame([](PsychicWebSocketRequest *request, httpd_ws_frame *frame) {
     Serial.printf("[socket] #%d sent: %s\n", request->client()->socket(), (char *)frame->payload);
     return request->reply(frame);
 });

 websocketHandler.onClose([](PsychicWebSocketClient *client) {
   Serial.printf("[socket] connection #%u closed from %s\n", client->socket(), client->remoteIP().toString());
 });

 //attach the handler to /ws.  You can then connect to ws://ip.address/ws
 server.on("/ws", &websocketHandler);

The onFrame() callback has 2 parameters:

For sending data on the websocket connection, there are 3 methods:

All of the above functions either accept simple char * string of you can construct your own httpd_ws_frame.

Special Note: Do not hold on to the PsychicWebSocketClient for sending messages to clients outside the callbacks. That pointer is destroyed when a client disconnects. Instead, store the int client->socket(). Then when you want to send a message, use this code:

//make sure our client is still connected.
PsychicWebSocketClient *client = websocketHandler.getClient(socket);
if (client != NULL)
  client->send("Your Message")

EventSource / SSE

The PsychicEventSource class is for handling EventSource / SSE connections. It provides 2 callbacks:

onOpen(...) is called when a new EventSource client connects. onClose(...) is called when a new EventSource client disconnects.

Here are the callback definitions:

void open_function(PsychicEventSourceClient *client);
void close_function(PsychicEventSourceClient *client);

Here is a basic example of using PsychicEventSource:

 //create our handler... note this should be located as a global or somewhere it wont go out of scope and be destroyed.
 PsychicEventSource eventSource;

 eventSource.onOpen([](PsychicEventSourceClient *client) {
   Serial.printf("[eventsource] connection #%u connected from %s\n", client->socket(), client->remoteIP().toString());
   client->send("Hello user!", NULL, millis(), 1000);
 });

 eventSource.onClose([](PsychicEventSourceClient *client) {
   Serial.printf("[eventsource] connection #%u closed from %s\n", client->socket(), client->remoteIP().toString());
 });

 //attach the handler to /events
 server.on("/events", &eventSource);

For sending data on the EventSource connection, there are 2 methods:

All of the above functions accept a simple char * message, and optionally: char * event name, id, and reconnect time.

Special Note: Do not hold on to the PsychicEventSourceClient for sending messages to clients outside the callbacks. That pointer is destroyed when a client disconnects. Instead, store the int client->socket(). Then when you want to send a message, use this code:

//make sure our client is still connected.
PsychicEventSourceClient *client = eventSource.getClient(socket);
if (client != NULL)
  client->send("Your Event")

HTTPS / SSL

PsychicHttp supports HTTPS / SSL out of the box, however there are some limitations (see performance below). Enabling it also increases the code size by about 100kb. To use HTTPS, you need to modify your setup like so:

#include <PsychicHttp.h>
#include <PsychicHttpsServer.h>
PsychicHttpsServer server;
server.listen(443, server_cert, server_key);

server_cert and server_key are both const char * parameters which contain the server certificate and private key, respectively.

To generate your own key and self signed certificate, you can use the command below:

openssl req -x509 -newkey rsa:4096 -nodes -keyout server.key -out server.crt -sha256 -days 365

Including the PsychicHttpsServer.h also defines PSY_ENABLE_SSL which you can use in your code to allow enabling / disabling calls in your code based on if the HTTPS server is available:

//our main server object
#ifdef PSY_ENABLE_SSL
  PsychicHttpsServer server;
#else
  PsychicHttpServer server;
#endif

Last, but not least, you can create a separate HTTP server on port 80 that redirects all requests to the HTTPS server:

//this creates a 2nd server listening on port 80 and redirects all requests HTTPS
PsychicHttpServer *redirectServer = new PsychicHttpServer();
redirectServer->config.ctrl_port = 20420; // just a random port different from the default one
redirectServer->listen(80);
redirectServer->onNotFound([](PsychicRequest *request) {
   String url = "https://" + request->host() + request->url();
   return request->redirect(url.c_str());
});

TemplatePrinter

This is not specific to PsychicHttp, and it works with any Print object. You could for example, template data out to File, Serial, etc....

The template engine is a Print interface and can be printed to directly, however, if you are just templating a few short strings, I'd probably just use response.printf() instead. Its benefit will be seen when templating large inputs such as files.

One benefit may be templating a JSON file avoiding the need to use ArduinoJson.

Before closing the underlying Print/Stream that this writes to, it must be flushed as small amounts of data can be buffered. A convenience method to take care of this is shows in example 3.

The header file is not currently added to PsychicHttp.h and users will have to add it manually:

#include <TemplatePrinter.h>

Template parameter definition:

Template processing

A function or lambda is used to receive the parameter replacement.

bool templateHandler(Print &output, const char *param){
  //...
}

[](Print &output, const char *param){
  //...
}

Parameters:

The handler must return a bool.

See output in example 1 regarding the effects of returning true or false.

Template input handler

This is not needed unless using the static convenience function TemplatePrinter::start(). See example 3.

bool inputHandler(TemplatePrinter &printer){
  //...
}

[](TemplatePrinter &printer){
  //...
}

Parameters:

Example 1 - Simple use with PsychicStreamResponse:

This example highlights its most basic usage.


//  Function to handle parameter requests.

bool templateHandler(Print &output, const char *param){

  if(strcmp(param, "FREE_HEAP") == 0){
    output.print((double)ESP.getFreeHeap() / 1024.0, 2);

  }else if(strcmp(param, "MIN_FREE_HEAP") == 0){
    output.print((double)ESP.getMinFreeHeap() / 1024.0, 2);

  }else if(strcmp(param, "MAX_ALLOC_HEAP") == 0){
    output.print((double)ESP.getMaxAllocHeap() / 1024.0, 2);

  }else if(strcmp(param, "HEAP_SIZE") == 0){
    output.print((double)ESP.getHeapSize() / 1024.0, 2);
  }else{
    return false;
  }
  output.print("Kb");
  return true;
}

//  Example serving a request
server.on("/template", [](PsychicRequest *request) {
  PsychicStreamResponse response(request, "text/plain");

  response.beginSend();

  TemplatePrinter printer(response, templateHandler);

  printer.println("My ESP has %FREE_HEAP% left. Its lifetime minimum heap is %MIN_FREE_HEAP%.");
  printer.println("The maximum allocation size is %MAX_ALLOC_HEAP%, and its total size is %HEAP_SIZE%.");
  printer.println("This is an unhandled parameter: %UNHANDLED_PARAM% and this is an invalid param %INVALID PARAM%.");
  printer.println("This line finished with %UNFIN");
  printer.flush();

  return response.endSend();
});

The output for example looks like:

My ESP has 170.92Kb left. Its lifetime minimum heap is 169.83Kb.
The maximum allocation size is 107.99Kb, and its total size is 284.19Kb.
This is an unhandled parameter: %UNHANDLED_PARAM% and this is an invalid param %INVALID PARAM%.
This line finished with %UNFIN

Example 2 - Templating a file

server.on("/home", [](PsychicRequest *request) {
  PsychicStreamResponse response(request, "text/html");
  File file = SD.open("/www/index.html");

  response.beginSend();

  TemplatePrinter printer(response, templateHandler);

  printer.copyFrom(file);
  printer.flush();
  file.close();

  return response.endSend();
});

Example 3 - Using the TemplatePrinter::start method.

This static method allows an RAII approach, allowing you to template a stream, etc... without needing a flush(). The function call is laid out as:

TemplatePrinter::start(host_stream, template_handler, input_handler);

*these examples use the templateHandler function defined in example 1.

Serve a file like example 2

server.on("/home", [](PsychicRequest *request) {
  PsychicStreamResponse response(request, "text/html");
  File file = SD.open("/www/index.html");

  response.beginSend();
  TemplatePrinter::start(response, templateHandler, [&file](TemplatePrinter &printer){
    printer.copyFrom(file);
  });
  file.close();

  return response.endSend();
});

Template a string like example 1

server.on("/template2", [](PsychicRequest *request) {

  PsychicStreamResponse response(request, "text/plain");

  response.beginSend();

  TemplatePrinter::start(response, templateHandler, [](TemplatePrinter &printer){
    printer.println("My ESP has %FREE_HEAP% left. Its lifetime minimum heap is %MIN_FREE_HEAP%.");
    printer.println("The maximum allocation size is %MAX_ALLOC_HEAP%, and its total size is %HEAP_SIZE%.");
    printer.println("This is an unhandled parameter: %UNHANDLED_PARAM% and this is an invalid param %INVALID PARAM%.");
  });

  return response.endSend();
});

Performance

In order to really see the differences between libraries, I created some basic benchmark firmwares for PsychicHttp, ESPAsyncWebserver, and ArduinoMongoose. I then ran the loadtest-http.sh and loadtest-websocket.sh scripts against each firmware to get some real numbers on the performance of each server library. All of the code and results are available in the /benchmark folder. If you want to see the collated data and graphs, there is a LibreOffice spreadsheet.

Performance graph Latency graph

HTTPS / SSL

Yes, PsychicHttp supports SSL out of the box, but there are a few caveats:

Analysis

The results clearly show some of the reasons for writing PsychicHttp: ESPAsyncWebserver crashes under heavy load on each test, across the board in a 60s test. That means in normal usage, you're just rolling the dice with how long it will go until it crashes. Every other number is moot, IMHO.

ArduinoMongoose doesn't crash under heavy load, but it does bog down with extremely high latency (15s) for web requests and appears to not even respond at the highest loadings as the loadtest script crashes instead. The code itself doesnt crash, so bonus points there. After the high load, it does go back to serving normally. One area ArduinoMongoose does shine, is in websockets where its performance is almost 2x the performance of PsychicHttp. Both in requests per second and latency. Clearly an area of improvement for PsychicHttp.

PsychicHttp has good performance across the board. No crashes and continously responds during each test. It is a clear winner in requests per second when serving files from memory, dynamic JSON, and has consistent performance when serving files from LittleFS. The only real downside is the lower performance of the websockets with a single connection handling 38rps, and maxing out at 120rps across multiple connections.

Takeaways

With all due respect to @me-no-dev who has done some amazing work in the open source community, I cannot recommend anyone use the ESPAsyncWebserver for anything other than simple projects that don't need to be reliable. Even then, PsychicHttp has taken the arcane api of the ESP-IDF web server library and made it nice and friendly to use with a very similar API to ESPAsyncWebserver. Also, ESPAsyncWebserver is more or less abandoned, with 150 open issues, 77 pending pull requests, and the last commit in over 2 years.

ArduinoMongoose is a good alternative, although the latency issues when it gets fully loaded can be very annoying. I believe it is also cross platform to other microcontrollers as well, but I haven't tested that. The other issue here is that it is based on an old version of a modified Mongoose library that will be difficult to update as it is a major revision behind and several security updates behind as well. Big thanks to @jeremypoulter though as PsychicHttp is a fork of ArduinoMongoose so it's built on strong bones.

Roadmap

v1.2: ESPAsyncWebserver Parity

Change: Modify the request handling to bring initail url matching and filtering into PsychicHttpServer itself.

Benefits:

Issues

How to implement

Longterm Wants

If anyone wants to take a crack at implementing any of the above features I am more than happy to accept pull requests.