Is there a way I can use static files located on a sd card instead of directly using them via a header?

[TheEpicFace007]

The generated header file of the static file are taking a lot of place. I am wondering if I could use them via a sd card with something like the sd card library to serve my static files.

[lasselukkari]

You have most likely already seen the FileServer example. It uses the SPIFFS file system but converting that use the SD card is straightforward. The downside is that the files will not be compressed and do not have the additional headers generated by the script.

I have an idea how this feature could be done so that the compressed binary content would be served from the sd card while keeping path routing and headers in the code. I'll get back to once i have done some initial prototyping.

Would you like to participate in the development, documentation or atleast testing it with the hardware you have?

[lasselukkari]

Something like this could work: https://github.com/lasselukkari/awot-scripts/compare/static-files-file?expand=1.

All files are catenated to a single file that would be copied to the SD card.

The StaticFiles.h would then look like this:

File *dataFile;

void static_another_html (Request &req, Response &res) {
  res.set("Content-Type", "text/html; charset=utf-8");
  res.set("Content-Encoding", "gzip");
  res.set("Cache-Control", "public, max-age=31536000");
  res.set("Content-Length", "83");
  res.set("Last-Modified", "Mon, 01 Jan 2525 00:00:00 GMT");

  dataFile->seek(0);

  for (unsigned long i = 0; i < 83; i++) {
    res.write(dataFile->read());
  }
}

void static_index (Request &req, Response &res) {
  res.set("Content-Type", "text/html; charset=utf-8");
  res.set("Content-Encoding", "gzip");
  res.set("Cache-Control", "no-cache");
  res.set("Content-Length", "83");
  res.set("Last-Modified", "Mon, 01 Jan 2525 00:00:00 GMT");

  dataFile->seek(83);

  for (unsigned long i = 0; i < 83; i++) {
    res.write(dataFile->read());
  }
}

Router staticFileRouter;

Router * staticFiles(File *file){
  dataFile = file;
  staticFileRouter.get("/another.html", &static_another_html);
  staticFileRouter.get("/", &static_index);
  return &staticFileRouter;
}

And the main sketch something like this:

#include <ESP8266WiFi.h>
#include <SPI.h>
#include <SD.h>
#include <aWOT.h>
#include "StaticFiles.h"

#define WIFI_SSID ""
#define WIFI_PASSWORD ""

WiFiServer server(80);
Application app;
File file;

void setup() {
  Serial.begin(9600);

  WiFi.begin(WIFI_SSID, WIFI_PASSWORD);
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  Serial.println(WiFi.localIP());

  if (!SD.begin(4)) {
    Serial.println("SD card initialization failed!");
    while (1);
  }

  file = SD.open("data.bin");
  if (!file) {
    Serial.println("Could not open data file");
    while (1);
  }

  app.use(staticFiles(&file));

  server.begin();
}

void loop() {
  WiFiClient client = server.available();

  if (client.connected()) {
    app.process(&client);
  }
}

[lasselukkari]

Actually there is no reason to keep even the headers in the code. Lets see how this evolves.

[lasselukkari]

Quick update. I managed to put the http headers to the sd card. Now most of the space is taken by the path strings. Let's see if they can be moved to some sort of binary search tree and stored to the sd card file.

[TheEpicFace007]

Would you like to participate in the development, documentation or atleast testing it with the hardware you have?

I don’t think I can develop for your project. I’m sorry

[lasselukkari]

This feature has been now implemented and there is an example that you can try out with included static.bin file. The static.bin file is generated with the awot-script tooling by setting the new sdCard option to true. The file contains an html image gallery of my trek in Annapurna area a few years ago.

static.bin.zip

I quickly tested it with the ESP8266 and ESP32. The download speed seems to be around 100 kb/s but this hopefully can be further improved with some tweaks. The code is also compatible with the sdfat library that may offer better performance.

[lasselukkari]

I also tested this with Arduino Uno and the old Ethernet shield with SD card reader. I had to modify the example a bit to make it work with 8-bit processors, but I expect it to work on all platforms now. Let me know if you run into any issues.

This proved out to be more interesting problem than I originally thought. I initially thought about creating some sort of trie structure to the file but that would have required a separate read call for each character in the filename (O(n)). The next idea was to create a array of 32 bit hashes and then perform a binary search to to find the location of the actual payload and length of it (O(log n)).

What I ended up using is a algorithm called minimal perfect hash. The final solution now has complexity of O(1) meaning that it takes a constant time to look up the data position no matter the amount of files in the set.

I haven't had this much fun coding for a long time.