Client.Flush() Not Working As Expected

[nheffronneuhold]

I am trying to write a program that will pull and parse aviation weather data (XML), and am having issues with the parsing. I started with the WifiWebClientRepeating example and only made small changes to get it to connect via SSL, but when reading the data client.flush() does not seem to be doing anything. The XML comes back with several of the same nodes, and I'm trying to gather only the data from the first one.

Sample XML Data: https://aviationweather.gov/adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&stationString=KORD&hoursBeforeNow=5

I'll put the full program at the end, but here is the code I'm using to parse it:

 while (client.available()) {
    if (client.find("<flight_category>")) {
      String category = client.readStringUntil('<');
      Serial.println(category);
    }
    delay(500);
    client.flush();
  }

It parses the text I want, but then instead of flushing all the rest of the data, it goes through the while loop 4 or 5 more times (depending on how many nodes are returned) and parses all of those too. I have tried changing the delay (which I assume is necessary to let all the data come in) and it has no affect. I have also tried to following:

  while (client.available()) {
    if (client.find("<flight_category>")) {
      String category = client.readStringUntil('<');
      Serial.println(category);
    }
    delay(500);
    while (client.available()) {
      client.read();
    }
  }

and it causes it to parse exactly one less node, regardless of the delay value. Any help would be highly appreciated. Full program:

#include <SPI.h>
#include <WiFi101.h>

char ssid[] = "ENTER_SSIS_HERE";   
char pass[] = "ENTER_PASSWORD_HERE";  
int keyIndex = 0;            // your network key Index number (needed only for WEP)

int status = WL_IDLE_STATUS;

// Initialize the WiFi client library
WiFiSSLClient client;

// server address:
char server[] = "aviationweather.gov";
//IPAddress server(64,131,82,241);

unsigned long lastConnectionTime = 0;            // last time you connected to the server, in milliseconds
const unsigned long postingInterval = 10L * 1000L; // delay between updates, in milliseconds

void setup() {
  WiFi.setPins(8, 7, 4, 2);

  //Initialize serial and wait for port to open:
  Serial.begin(9600);
  while (!Serial) {
    ; // wait for serial port to connect. Needed for native USB port only
  }

  // check for the presence of the shield:
  if (WiFi.status() == WL_NO_SHIELD) {
    Serial.println("WiFi shield not present");
    // don't continue:
    while (true);
  }

  connectToWifi();
}

void loop() {

  while (client.available()) {
    if (client.find("<flight_category>")) {
      String category = client.readStringUntil('<');
      Serial.println(category);
    }
    delay(500);
    client.flush();
  }

  if (millis() - lastConnectionTime > postingInterval) {
    httpRequest();
  }
}

void connectToWifi() {

  // attempt to connect to WiFi network:
  while (WiFi.status() != WL_CONNECTED) {
    Serial.print("Attempting to connect to SSID: ");
    Serial.println(ssid);
    // Connect to WPA/WPA2 network. Change this line if using open or WEP network:
    status = WiFi.begin(ssid, pass);

    // wait 10 seconds for connection:
    delay(10000);
  }
  // you're connected now, so print out the status:
  printWiFiStatus();
}

// this method makes a HTTP connection to the server:
void httpRequest() {
  // close any connection before send a new request.
  // This will free the socket on the WiFi shield
  client.stop();

  // if there's a successful connection:
  if (client.connect(server, 443)) {
    Serial.println("connecting...");
    // send the HTTP PUT request:
    client.println("GET /adds/dataserver_current/httpparam?dataSource=metars&requestType=retrieve&format=xml&stationString=KORD&hoursBeforeNow=5 HTTP/1.1");
    client.println("Host: aviationweather.gov");
    client.println("User-Agent: ArduinoWiFi/1.1");
    client.println("Connection: close");
    client.println();

    // note the time that the connection was made:
    lastConnectionTime = millis();
  }
  else {
    // if you couldn't make a connection:
    Serial.println("connection failed");
    connectToWifi();
  }
}

void printWiFiStatus() {
  // print the SSID of the network you're attached to:
  Serial.print("SSID: ");
  Serial.println(WiFi.SSID());

  // print your WiFi shield's IP address:
  IPAddress ip = WiFi.localIP();
  Serial.print("IP Address: ");
  Serial.println(ip);

  // print the received signal strength:
  long rssi = WiFi.RSSI();
  Serial.print("signal strength (RSSI):");
  Serial.print(rssi);
  Serial.println(" dBm");
}

[sandeepmistry]

I've discussed this with @cmaglie, client.flush() is only intended to flush the transmit buffer and not the receive.

[nheffronneuhold]

I'm not sure if this is something you have control over, but the documentation on the Arduino site makes it sound as if it clears the receive buffer (by saying "Discard any bytes that have been written to the client but not yet read."). I think it would be helpful to future users if it specified that it clears only the transmit buffer.

https://www.arduino.cc/en/Reference/WiFi101ClientFlush

On a separate note, do you have any idea why while (client.available()) { client.read(); } does not seem to be clearing the receive buffer either?

[nheffronneuhold]

I think I may have solved it by adding a 10 millisecond delay within the loop like this while (client.available()) { client.read(); delay(10); }

I don't know why that works while putting a delay before the loop (which I thought would allow all the data to flow into the buffer) doesn't, but I'm hoping someone with a similar issue in the future can find this.

[sandeepmistry]

I'm not sure if this is something you have control over, but the documentation on the Arduino site makes it sound as if it clears the receive buffer (by saying "Discard any bytes that have been written to the client but not yet read."). I think it would be helpful to future users if it specified that it clears only the transmit buffer.

@akash73 @simonepda can we please update the documentation to reflect this?

I don't know why that works while putting a delay before the loop (which I thought would allow all the data to flow into the buffer) doesn't, but I'm hoping someone with a similar issue in the future can find this.

@nheffronneuhold client.available() can return 0 after a client.read() if a data chunk is not received or buffered elsewhere.

Stream's private timedRead() method might give you another idea on how to handle this: https://github.com/arduino/Arduino/blob/master/hardware/arduino/avr/cores/arduino/Stream.cpp#L31-L40