Upload vs. Download Speeds

[technobly]

@ovidiucp, first of all I love this TinyWebServer so far it's exactly what I was looking for to be able to upload files to my Arduino remotely. I will do my best to help contribute something useful for this repo.

That said, the upload speed is much slower than the download speed. 5 minutes on a 1MB image.jpg file upload, vs. 17 seconds to download to the browser. This seems to be true with both the FileUpload and BlinkLed examples. Can you offer any explanation why this is, and if there is any way you could think of to make it faster uploading? Even if the hardware needed to change, please suggest whatever you think might get me going in the right direction. Thanks!

[ovidiucp]

Can you please add some logging with timing information to get some more insights into what's going on in the upload phase? My guess is writing on the SD card is very slow.

[technobly]

Here's the debug log for the file I'm uploading. I tried with a Class 6 card and it's not much faster, however if I test the card with my computer it's very capable of being written to fast. I figured you'd have a better idea of how the upload vs. download to browser is handled, and be able to tell me something like "upload is slower because of xyz protocol and lack of RAM available ... " or something like that.

TWS:New request: PUT /upload/large.jpg HTTP/1.0 User-Agent: curl/7.31.0 Host: 192.168.1.15 Accept: */* Content-Length: 1036939 TWS:Returning 200 Creating LARGE.JPG Wrote 1036941 bytes in 250382 millis (received 1036941 bytes)

Watching the TX / RX lights on the ethernet shield during this upload, they flash on and off together at maybe a 4Hz rate for a duration of 12.6 seconds, then they turn off for a duration of 5 seconds and repeat until the file is done.

During download of this image via 192.168.1.15/static/large.jpg the TX / RX lights are on solid for 17 seconds.

During upload I'm using CURL.EXE but maybe it is slowing things down? I don't see any options that would help speed things up, and the files are so small that apparently the progress meter won't display.

I also tried using the Advanced Rest Client extension for Chrome and did a PUT request to 192.168.1.15/upload/ with a file payload, but it gets no response. Even a GET request to 192.168.1.15/ does nothing.

Maybe you could explain the put_handler briefly?

boolean put_handler(TinyWebServer& web_server) {
  web_server.send_error_code(200);
  web_server.end_headers();

  const char* length_str = web_server.get_header_value("Content-Length");
  long length = atol(length_str);
  uint32_t start_time = 0;
  boolean watchdog_start = false;

  EthernetClient client = web_server.get_client();

  if (put_handler_fn) {
    (*put_handler_fn)(web_server, START, NULL, length);
  }

  uint32_t i;
  for (i = 0; i < length && client.connected();) {
    int16_t size = read_chars(web_server, client, (uint8_t*)buffer, 64);
    if (!size) {
      if (watchdog_start) {
        if (millis() - start_time > 30000) {
          // Exit if there has been zero data from connected client
          // for more than 30 seconds.
#if DEBUG
          Serial << F("TWS:There has been no data for >30 Sec.\n");
#endif
          break;
        }
      } else {
        // We have hit an empty buffer, start the watchdog.
        start_time = millis();
        watchdog_start = true;
      }
      continue;
    }
    i += size;
    // Ensure we re-start the watchdog if we get ANY data input.
    watchdog_start = false;

    if (put_handler_fn) {
      (*put_handler_fn)(web_server, WRITE, buffer, size);
    }
  }
  if (put_handler_fn) {
    (*put_handler_fn)(web_server, END, NULL, 0);
  }

  return true;
}

I tried changing this line from 64 to 128 with no effect: int16_t size = read_chars(web_server, client, (uint8_t*)buffer, 64);

One more really good question... how long do default files in BlinkLed take you to upload? Mine is about 45 seconds or so.

[technobly]

Hmm, maybe the low level client.read(void) is the issue as it only reads one char at a time.

boolean TinyWebServer::read_next_char(Client& client, uint8_t* ch) {
  if (!client.available()) {
    return false;
  } else {
    *ch = client.read();
    return true;
  }
}

Maybe we could utilize the second read method in the ethernet library and read something like 64 bytes at a time?

int EthernetClient::read() {
  uint8_t b;
  if ( recv(_sock, &b, 1) > 0 )
  {
    // recv worked
    return b;
  }
  else
  {
    // No data available
    return -1;
  }
}

int EthernetClient::read(uint8_t *buf, size_t size) {
  return recv(_sock, buf, size);
}

[ovidiucp]

It makes sense. Unfortunately I don't have an Arduino+Ethernet setup handy at the moment. Can you try out your idea and see if it improves things?

[technobly]

I will definitely be working on it. If you have any code advice I'm open for suggestions. I'm just getting started with the Ethernet stuff myself, but I'm experienced with embedded C coding.

[technobly]

Ok this seems to work, but the upload speed is still the same:

// Fills in `buffer' by reading up to `num_bytes'.
// Returns the number of characters read.
int read_chars(TinyWebServer& web_server, Client& client,
               uint8_t* buffer, int size) {
/*
  uint8_t ch;
  int pos;
  for (pos = 0; pos < size && web_server.read_next_char(client, &ch); pos++) {
    //Serial.print(ch);
    buffer[pos] = ch;
  }
  return pos;
*/
  int len;
  if (!client.available()) {
    return -1;
  } else {
    len = client.read(buffer, size);
    return len;
  }
}

and

  uint32_t i;
  for (i = 0; i < length && client.connected();) {
    int16_t size = read_chars(web_server, client, (uint8_t*)buffer, 64);
    Serial << F("Size: ") << size << F(" Total Size: ") << i << "\n";
    if (size == 0 || size == -1) {
      if (watchdog_start) {
        if (millis() - start_time > 30000) {
          // Exit if there has been zero data from connected client
          // for more than 30 seconds.
   ...

EDIT: modified above code to help show the difference between no data available (size = -1). Turns out all those Size = 0 are actually no data available cases. Technically the client.read(buffer, size) will also return -1 when no data is available, so this is kind of redundant but client.available() is slightly faster. So now I'm trying to figure out why that is happening....

So this is reading in 64 bytes at a time... but if you notice the Serial debug line... I can see bursts of Size = 64 and then a whole bunch of Size = 0... and then more 64's and 0's repeating up to the length of the message. All bytes are accounted for and and web page loads after the upload... but I'm still debugging what's causing the 0's. I'm pretty sure the W5100 will buffer 2kB of data... so the delay has to be on getting that data from the W5100. I even commented out the Write portion of the PUT routine and the speed remained the same.

Here is a pastebin file of the debug info for the lava.jpg file transfer http://pastebin.com/BcF1JyGX

Notice at the end of the pastebin file it reports 0 written, that's because I have this section of code commented out:

    i += size;
    // Ensure we re-start the watchdog if we get ANY data input.
    watchdog_start = false;

    //if (put_handler_fn) {
      //(*put_handler_fn)(web_server, WRITE, buffer, size);
    //}
  }
  if (put_handler_fn) {
    (*put_handler_fn)(web_server, END, NULL, 0);
  }

  return true;
}

Can you explain what that (*put_handler_fn)(web_server, WRITE, buffer, size); does and how it does it? It's quite odd to me. I wonder what the benefit to defining it this way is? I think the *put_handler_fn is a pointer to the place in memory where this function exists, but this function prototype only seems to be defined in the TinyWebServer.h file and I'm not quite sure how this is causing the write to SD card.

For reference (TinyWebServer.h)

namespace TinyWebPutHandler {
  enum PutAction {
    START,
    WRITE,
    END
  };

  typedef void (*HandlerFn)(TinyWebServer& web_server,
                PutAction action,
                char* buffer, int size);

  // An HTTP handler that knows how to handle file uploads using the
  // PUT method. Set the `put_handler_fn' variable below to your own
  // function to handle the characters of the uploaded function.
  boolean put_handler(TinyWebServer& web_server);
  extern HandlerFn put_handler_fn;
};

[ovidiucp]

The put_handler is used to specify an application specific function that handles the HTTP PUT action. In the case of the FileUpload example, that function is set to the file_uploader_handler function (see examples/FileUpload/FileUpload.ino).

I suppose the problem could be a hardware issue. The Ethernet and SD card share SPI, a serial bus used to send data at higher speeds. The two devices might interfere with other somehow, by placing locks that expire after a longer than necessary timeout.

There are two things I'd try to do to see who's responsible for introducing delays. To test if the Ethernet code is at fault, I'd comment out the line:

(*put_handler_fn)(web_server, WRITE, buffer, size);

This essentially makes the server receive the requests, but does not attempt to write any data on the SD card. Send the request again using curl and see how long it takes to execute. In normal conditions, this should be speedy, around few seconds. If this is not fast, then the Ethernet code in the library has problems (see the Ethernet/utility/w5100.cpp for more info on how they do it).

If the Ethernet code is fine, the problem might be with the SD card code. I'd write some simple code that uses the SD library and writes a 1MB by repeatedly writing a 64 byte buffer. Don't forget to close the file at the end to make sure the data is written on the disk. This code should be fast too, around few seconds at most. Something like this (not tested):

#include <SPI.h>
#include <SD.h>

// pin 4 is the SPI select pin for the SDcard
const int SD_CS = 4;
const int SD_CS = 4;

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

void setup() {
  if (!card.init(SPI_FULL_SPEED, SD_CS)) {
    Serial << F("card failed\n");
  }

  if (!volume.init(&card)) {
    Serial << F("vol.init failed!\n");
  }
  if (!root.openRoot(&volume)) {
    Serial << F("openRoot failed");
  }

  file.open(&root, "test.dat", O_CREAT | O_WRITE | O_TRUNC);
  char buffer[64];
  static uint32_t start_time = millis();

  for (int i = 0; i < 16384; i++) {
    file.write(buffer, sizeof(buffer));
  }
  file.close();
  Serial << F("Done writing the file" << millis() - start_time << F(" millis"));
}

void loop() {
}

If this is fast too, then the interaction between the two systems (the Ethernet and the SD libraries) has problems.

[technobly]

Thanks for the feedback!

As for your first suggestion of commenting out (*put_handler_fn)(web_server, WRITE, buffer, size); I did exactly that and the time was still slow. I have not tried writing to the SD card only yet and will try that tonight.

I see now that this (*put_handler_fn)(web_server, WRITE, buffer, size);

is linked to this in the sketch:

void file_uploader_handler(TinyWebServer& web_server,
               TinyWebPutHandler::PutAction action,
               char* buffer, int size) {
...
case TinyWebPutHandler::WRITE:
    if (file.isOpen()) {
      file.write(buffer, size);
      total_size += size;
    }
    break;
...
  }
}

Initially it was hard to see how this works since you define a function in your sketch, then kick off the web.process(); which basically makes the Class code take over and call your defined function. Instead of having code in the void loop() { } that kind of takes care of the state machine. For abstracting a working web server though it does a very nice job. This is basically like tying a RESTful API to your defined functions.

If I wanted to run code in my sketch based on timing or user digital input or analog input... could that still be done in the main loop without affecting the server processing too much?

[technobly]

I was just looking at my logs again and noticed the index.html file was transfered MUCH faster. Normally it was about 1400 milliseconds, but now it's 8 milliseconds.

TWS:New request: PUT /upload/index.htm HTTP/1.0

User-Agent: curl/7.31.0
Host: 192.168.1.15
Accept: */*
Content-Length: 299

TWS:Returning 200
Creating INDEX.HTM
Size: 64 Length: 299
Size: 64 Length: 299
Size: 64 Length: 299
Size: 64 Length: 299
Size: 43 Length: 299
Wrote 0 bytes in 8 millis (received 0 bytes)

This doesn't not include writing to the SD card, but also no Size: 0 either. This is about 37kB/s I'll experiment with this file more tonight too, using 32, 128 and 256 bytes for the buffer.

[technobly]

I tweaked your Test.dat File Writer sketch and found that it would reliably write 1MB files in roughly 9.5 to 11 seconds with a 64 byte buffer. I tried 128 byte, 32 byte and 1 byte at a time... and 64 seemed to be the fastest for writing to my SD card. 1 byte at a time was only 45 seconds though.

So I don't think the write speed is slow at all... by itself. I'll keep digging!

#include <SPI.h>
#include <SD.h>
#include <Flash.h>

// pin 4 is the SPI select pin for the SDcard
const int SD_CS = 4;

// pin 10 is the SPI select pin for the Ethernet
const int ETHER_CS = 10;

Sd2Card card;
SdVolume volume;
SdFile root;
SdFile file;

void setup() {
  Serial.begin(115200);
  Serial << F("Free RAM: ") << FreeRam() << "\n";

  pinMode(SS_PIN, OUTPUT);  // set the SS pin as an output
                                // (necessary to keep the board as
                                // master and not SPI slave)
  digitalWrite(SS_PIN, HIGH);   // and ensure SS is high

  // Ensure we are in a consistent state after power-up or a reset
  // button These pins are standard for the Arduino w5100 Rev 3
  // ethernet board They may need to be re-jigged for different boards
  pinMode(ETHER_CS, OUTPUT);    // Set the CS pin as an output
  digitalWrite(ETHER_CS, HIGH); // Turn off the W5100 chip! (wait for
                                // configuration)
  pinMode(SD_CS, OUTPUT);   // Set the SDcard CS pin as an output
  digitalWrite(SD_CS, HIGH);    // Turn off the SD card! (wait for
                                // configuration)

  boolean systemFail = false;                              
  if (!card.init(SPI_FULL_SPEED, SD_CS)) {
    Serial << F("Card init failed!\n");
    systemFail = true; 
  }
  if (!volume.init(&card)) {
    Serial << F("Vol. init failed!\n");
    systemFail = true; 
  }
  if (!root.openRoot(&volume)) {
    Serial << F("Open root failed!\n");
    systemFail = true; 
  }
  while(systemFail); //kill loop

  Serial << F("Writing \"test.dat\" with 1MB of null terminators...\n");
  file.open(&root, "test.dat", O_CREAT | O_WRITE | O_TRUNC);

  char buffer[64] = "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0";

  static uint32_t start_time = millis();
  for (uint32_t i = 0; i < 16384; i++) {
    file.write(buffer, sizeof(buffer));
  }
  file.sync();
  file.close();
  Serial << F("Done writing in ") << (millis() - start_time) << F("ms");
}

void loop() {
}

[technobly]

I modified my above buffered read code to help show the difference between no data available (size = -1). Turns out all those Size = 0 are actually no data available cases. Technically the client.read(buffer, size) will also return -1 when no data is available, so this is kind of redundant but client.available() is slightly faster. So now I'm trying to figure out why that is happening....

I also figured out why my Advanced Rest Client chrome extension wasn't working... I had 192.168.1.15/upload as my ip instead of http://192.168.1.15/upload .. DOAH! So now that's a second option for transferring data, but it is basically just as slow, if not a tad slower. The thing I was trying to figure out is if CURL was intentionally stopping the data transfer.. I don't think so now.

I started looking at packets with Wireshark.org's software... and it was complaining occasionally that my TCP sockets or buffers were full when I was using the 64 byte buffered read, but with 256 byte buffered read it wasn't complaining about that anymore. However, both cases were still slow. Seems like it might be the Ethernet module Wiz5100.

I'm going to try your code with the CC3000 Wifi module and see if it's any different, hopefully better. Currently Adafruit.com supports it with their breakout board and soon Spark Devices will too with their amazing Spark Core. I have both platforms and I'm dying to get them working. Arduino Ethernet shield has been buying me time, but it's too clunky.

[ovidiucp]

The CC3000 shield looks really sweet! Years ago I tried to use the only WiFi shield at the time, and I was surprised to see it only allowed a single client socket, which was really useful.

[technobly]

Kind of getting stuck here... CC3000 shield doesn't have Server support code yet, and testing it's 64 byte buffered read speed by downloading your lava.jpg file takes it 160 seconds! (vs. the already slow 12 seconds of the Ethernet Shield).

I'll update when I can make some real progress ;-)