Web service implementation with ESP8266WebServer and ArduinoJson

[ananchev]

I have been working on web service to host the sensor readings using ESP8266WebServer and ArduinoJson libraries: code is here. Basically on call to a server endpoint, the metriful data readings are getting extracted from their respective structs, slightly transformed and written into JsonObject objects. These are then served as json content via ESP8266WebServer.

All works quite okay at first glance, but as sensor is working for a while (e.g. ~2hrs on 3s reading interval), the values for Temperature, Humidity and Air Quality (except the interpretation string) are randomly returned as null. Is there any specific in the way they are returned from metriful and resp. to be captured into the ArduinoJson objects?

[metriful]

A possible reason is your synchronization with the "ready" signal.

The basic method for the loop() in this program is to wait for the falling edge on the ready signal, then do all the tasks, then repeat.

The output data from the board is only valid for 2.95 seconds (for a 3 second cycle) after the ready signal goes low (see datasheet page 13). If you read during the ready high period, you get bad data.

Here are 3 ways to fix this: 1) Write the program to guarantee that you complete the loop in < 2.95 seconds 2) Identify cases where you exceed the time constraint, then discard the data for that loop and just wait for the next ready assertion. E.g. you could read "ready" before and after reading the data, and only use the data if both reads are low. 3) Use a longer cycle time (100 or 300 seconds) which gives you more time to do tasks.

[ananchev]

I understand what you say, but I believe this is not the case here: in fact the logic I implemented is very similar to the webserver example: in the loop method the client calls are handled in a while loop executed when the ready signal is on false:

  void loop(void) {
  // Wait for the next new data release, indicated by a falling edge on READY
  while (!ready_assertion_event) {
    server.handleClient();
    yield();
  }
  lastRead = millis();  
  ready_assertion_event = false;
  // Air data
  ReceiveI2C(I2C_ADDRESS, AIR_DATA_READ, (uint8_t *) &airData, AIR_DATA_BYTES);

When it changes to true, only the ReceiveI2C is called to populate the globally defined structs (e.g. above airData) with the new measurement. There is nothing else happening that would cause the loop to take more than 2.95 sec and resp. invalidate the data. If when in the while loop, a webserver GET call is received, there is another method that pulls data out of the structs, populates into JsonObject and serves back. The behavior with the nulls described initially is there in all cycle times when running for a while, after a restart it is okay and few hrs later again.

Actually the tests I did show it is related to the use of sprintf formatting function to produce combined string and write into JsonObject. When I change this

  //Humidity
  char H_value[50]={0};
  sprintf(H_value, "%u.%u", airData.H_pc_int, airData.H_pc_fr_1dp);
  JsonObject humidity = doc["Humidity"].createNestedObject();
  humidity["value"] = H_value;
  humidity["unit"] ="%";

to this

//Humidity
  JsonObject humidity = doc["Humidity"].createNestedObject();
  humidity["H_pc_int"] = airData.H_pc_int;
  humidity["H_pc_fr_1dp"] = airData.H_pc_fr_1dp;
  humidity["unit"] ="%";

all works fine - no nulls anymore. I can definitely work this way and handle the json in the app where it is consumed, but curious about the reason causing it.

[metriful]

It looks like a memory problem relating to ArduinoJson (which we are not able to help with).

You can check whether it is a sprintf problem by including the sprintf line, printing the resulting H_value, and removing the parts with the JsonObjects.

This line:

humidity["value"] = H_value;

is presumably copying the string into the json object. Can it run out of memory if you keep doing this? Maybe call memoryUsage() on the DynamicJsonDocument to check, or just clear() it after every send?

Our examples avoid dynamic memory allocation because of these difficulties!

In any case, it seems the problem is not with our hardware/software.

[ananchev]

Thanks a lot for your help here. Will keep looking into it and post an update if find the exact cause. Nevertheless fully agree this now is clearly not at all related to the Metriful product or software!