[Solved] Using uPlot on ESP32 microcontroller

[happytm]

I have been searching for small graphing library for home sensor network running on ESP32 microcontroller which has limited resources. It seems like uPlot fit the bill.

• ESP32 collect 40 different sensor values from 10 temperature sensors , 10 humidity sensors , 10 pressure sensors and 10 light level sensors.

• ESP32 can save all this data in a file for historic records.

• ESP32 can run web interface (html, css and javascript) to display these data using uPlot.

I want to graph these data as line charts by each room which have 4 sensors each.

Have someone tried this ? if so how the data have to be stored in a file ?

There is a similar project using google charts here:

https://www.youtube.com/watch?v=_WhmGkI5rKg
https://github.com/G6EJD/ESP32-8266-Sensor-Monitor

Have someone tried this ? if so how the data have to be stored in a file ? What is the easiest way to achieving this ?

Thanks.

[leeoniya]

you can store the data in whatever format is convenient, but CSV or JSON will be easiest to process in JavaScript. if you have an example output CSV or JSON that gets generated, i can show you how to prepare it and feed it to uPlot.

you will obviously need to have a minimal webserver running on the ESP32 that's capable of serving the log file as well as the required html, css and js files, e.g. https://lastminuteengineers.com/creating-esp32-web-server-arduino-ide/

[happytm]

Thank you for your quick response. I will be storing data in a file called sensordata.json on flash of ESP32 located at http://IP of ESP32/data folder. Content of this file will be in following key/value format:

{"location":"Livingroom","rssi":"-70","T":"70","H":"52","P":"28","L": "68"} {"location":"Bedroom","rssi":"-48","T":"72","H":"55","P":"29","L": "57"} {"location":"Kitchen","rssi":"-82","T":"75","H":"64","P":"28","L": "62"}

Each device sends data to ESP32 gateway which converts it to json array like above and append it to sensordata.json file.

Ideally I wanted to plot this with simple line chart for each room showing 5 sensor values in time series chart. If needed I can append timestamp as a part of json array with receipt of sensor values.

There will be folder called data with index.html & sensordata.json files and within data folder there will be 2 more folders called js and css.

Please advise how to go about creating these plots.

Thanks.

[leeoniya]

since space is limited, i would recommend maybe creating a more simple and compact sensordata.csv log file instead:

loc,rssi,t,h,p,l
0,-82,75,64,28,62
1,-70,70,52,28,68

then in JS you would have a mapping to better/longer labels:

const locations = {
  0: "Kitchen",
  1: "Living Room",
};

[happytm]

Good idea.

I was storing my data this way because before appending it to file I was sending the json array to MQTT broker as soon as it was received.

I can certainly save it in format you suggested in csv file. Actually when I receive data from remote sensors location value arrive as integer type and then I I had to convert it to human readable names so it will be better to skip conversion step and append to file directly as original integer value without converting to location names.

Do I have to append first line every time I receive sensor values ?

Do I need to append timestamp too with each line ?

Thanks.

[leeoniya]

really all that matters is how much knowledge you want to delegate to your JS code.

the log file can be completely your own custom format, for example, the first line can encode the initial timestamp and the sampling rate in seconds. the second line can be headers, rest can be data. this way you can avoid storing the literal timestamp with every record and compute each timestamp in JS by doing initial_timestamp + 60 * tick. of course if for some reason your samples cannot be collected due to power outage or network downtime, then the computed timestamps will be wrong :)

the entire file will be loaded by the browser, so you do not need to append the headers more than once, just the new data. how you handle log rotation is up to you.

1620795970,60
tick,loc,rssi,t,h,p,l
1,0,-82,75,64,28,62
1,1,-70,70,52,28,68

[shengyan]

I have a project that use an ESP8266 module to control my aquarium lighting and cooling fan. The sensor data is not saved on file system but stay on memory instead, as the flash chip has a write cycle of about 10,000 cycles only.

the data structure looks like this

#define DATA_ARRAY_LENGTH 288
typedef struct {
  short airTemp[DATA_ARRAY_LENGTH];
  short airHumi[DATA_ARRAY_LENGTH];
  short airPres[DATA_ARRAY_LENGTH];
  short wtrTemp[DATA_ARRAY_LENGTH];
  short fanLoad[DATA_ARRAY_LENGTH];
  short index;
} logsStruct;
logsStruct logs;

the index is used to indicated the latest data item position. datetime is generated on js code based on current time and a 5 minute time interval.

[happytm]

@shengyan Can you somehow share your project code please?

Thanks.

[shengyan]

@happytm I have uploaded the code to a new repository: https://github.com/shengyan/AquaMatic

[happytm]

Thank you @shengyan .

[happytm]

really all that matters is how much knowledge you want to delegate to your JS code.

the log file can be completely your own custom format, for example, the first line can encode the initial timestamp and the sampling rate in seconds. the second line can be headers, rest can be data. this way you can avoid storing the literal timestamp with every record and compute each timestamp in JS by doing initial_timestamp + 60 * tick. of course if for some reason your samples cannot be collected due to power outage or network downtime, then the computed timestamps will be wrong :)

the entire file will be loaded by the browser, so you do not need to append the headers more than once, just the new data. how you handle log rotation is up to you.

1620795970,60
tick,loc,rssi,t,h,p,l
1,0,-82,75,64,28,62
1,1,-70,70,52,28,68

@leeoniya I can generate a csv file just like your data.json file on ESP32 server but without square brackets at the begining and at the end of the file. Is such csv data file still usable with some pre processing before plotting?

If possible can you please show me how to do it?

I want to use your demo at https://leeoniya.github.io/uPlot/bench/uPlot.html.

Thank you.

[happytm]

@leeoniya I have LittleFS file system running on my ESP32 micro controller with webserver.

My data file (sensors.csv) is in root directory and looks like following:

epoch,loc,v,s,t,h,p,l
1623479163,1,2.94,-36,47,77,239,35
1623479313,2,2.84,-43,46,86,239,32
1623479463,3,2.88,-37,51,64,239,89
1623479614,1,2.94,-37,47,77,239,35
1623479764,2,2.84,-43,46,86,239,32
1623479914,3,2.88,-38,50,64,239,89
1623480064,1,2.94,-36,47,77,239,35
1623480215,2,2.85,-44,46,86,239,33

The location 1 = Livingroom, 2 = Kitchen and 3 = bedroom.

Can you please show me how to plot this?

Thanks.

[leeoniya]

i'll make an example for you tomorrow 👍

[happytm]

Thanks.

[leeoniya]

https://jsfiddle.net/upkb42mr/3/

const csv = `
epoch,loc,v,s,t,h,p,l
1623479163,1,2.94,-36,47,77,239,35
1623479313,2,2.84,-43,46,86,239,32
1623479463,3,2.88,-37,51,64,239,89
1623479614,1,2.94,-37,47,77,239,35
1623479764,2,2.84,-43,46,86,239,32
1623479914,3,2.88,-38,50,64,239,89
1623480064,1,2.94,-36,47,77,239,35
1623480215,2,2.85,-44,46,86,239,33
`;

function initLocData() {
  return [
    [], // epoch
    [], // v
    [], // s
    [], // t
    [], // h
    [], // p
    [], // l
  ];
}

const locNames = {
  "1": "Living Room",
  "2": "Kitchen",
  "3": "Bedroom",
};

// data per chart
const locData = {
  "1": initLocData(),
  "2": initLocData(),
  "3": initLocData(),
};

const lines = csv.trim().split("\n");
const header = lines.shift();

// split & push chart data
lines.forEach((line, idx) => {
  let vals = line.split(",");
  let data = locData[vals[1]];

  data[0].push(+vals[0]);
  data[1].push(+vals[2]);
  data[2].push(+vals[3]);
  data[3].push(+vals[4]);
  data[4].push(+vals[5]);
  data[5].push(+vals[6]);
  data[6].push(+vals[7]);
});

function makeChart(title, data) {
  let opts = {
    title,
    width: 600,
    height: 300,
    series: [
      {},
      {
        label: "v",
        stroke: "red"
      },
      {
        label: "s",
        stroke: "green"
      },
      {
        label: "t",
        stroke: "blue"
      },
      {
        label: "h",
        stroke: "magenta"
      },
      {
        label: "p",
        stroke: "orange"
      },
      {
        label: "l",
        stroke: "pink"
      },
    ]
  };

  let plot = new uPlot(opts, data, document.body);

  return plot;
}

for (let locKey in locNames) {
  makeChart(locNames[locKey], locData[locKey]);
}

[happytm]

Thank you for taking your valuable time to help me. It works perfectly in my ESP32 project.

[leeoniya]

glad you got it working 👍