search

lasselukkari / aWOT

Arduino web server library.
MIT License
283 stars 41 forks source link

Upload files from browser and store them into LittleFS/SPIFFS/SD #136

Closed AppsByDavideV closed 2 years ago

AppsByDavideV commented 2 years ago

Hello, I found this library very useful for working with ethernet and WiFi at the same time. I have been able to serve files from SD (using ESP32+ SDcard reader) to the browser using one of your examples, but due to my lack of knowledge about how library works and data flow, I don't understand how to send files from browser to be stored in the SD (or SPIFFS etc..).

I think it can be done in a similar way as OTA update, but I can't get it :( Is there any example?

Thank you in advance. /Davide

lasselukkari commented 2 years ago

Hi!

What are you trying to do? Do you just want to upload a single file for some special purpose or are you trying create a generic file server?

You are correct that it can be done in a similar way the OTA update works but the details will depend on the actual use case.

It should look like something like this. I did not actually test this code at all.

void upload(Request &req, Response &res) {
  // this is only needed with curl
  if (strcmp(req.get("Expect"), "100-continue") == 0) {
    res.status(100);
  }

  unsigned long start = millis();
  while (!req.available() && millis() - start <= 5000) {}

  if (!req.available()) {
    return res.sendStatus(408);
  }
  // end of what is needed only with curl

  File file = SPIFFS.open("filename.ext", "w");

  while (req.left()) {
   // also deal with clients that stop sending data here

    file.write(req.read());
  }

  file.close();
  res.sendStatus(204);
}
AppsByDavideV commented 2 years ago

Hi, thanks for the fast answer :) I just want to upload a file, for example an html file, a text file or an image with the purpose to update/expand the webserver that has its files saved on the SD card.

Bye /Davide

lasselukkari commented 2 years ago

OK. Then things will get a bit more complicated as you will need to provide the filename with the upload. Normally this is done with multipart/form-data content type. This is definitely doable, but will be a lot more complicated task. If it is only you doing the file updates you could also follow this example https://github.com/lasselukkari/aWOT/blob/master/examples/CardFiles/CardFiles.ino and then update all the files with one file upload like it is done the previous reply from me. Building a generic file server with the possibility for example to also delete files will require lots of work. There is no real limitation in the library that would prevent you from doing this but you will need to know a lot about how browsers handle file uploads and also create a UI for the whole thing. If you really want to do this this will get you to the right track: https://stackoverflow.com/questions/8659808/how-does-http-file-upload-work

AppsByDavideV commented 2 years ago

I need to upload only one file at a time. I have been able to upload and store a 17Kb file to the SD using your previous example, but yes I need the filename provided at start of file and also remove unnecessary HTTP information, otherwise it will be corrupted.

This is the hard task, unless the library provides a way to get the filename and remove unnecessary data before saving it to the sd.

This is an example of what has been uploaded (opened in notepad++):

`------WebKitFormBoundaryp1rTk9tpE5B1Vt77 Content-Disposition: form-data; name="update"; filename="21476_leaflet_simple_it_1-0.pdf" Content-Type: application/pdf

%PDF-1.7`

The original PDF file starts with %PDF-1.7...

I will try working on it

lasselukkari commented 2 years ago

Yes you need to parse that yourself if you decide to go that way. Did you read the stackoverflow link I sent you?

lasselukkari commented 2 years ago

Well, another option would be to add the filename to a custom http header.

But this is really not about this library anymore. This about http, browsers and javascript in general. Unfortunately I will not have time to write you a custom example for this. You are on your own.

Edit: Actually maybe it's not that hard. Let me see.

lasselukkari commented 2 years ago

Change your file upload form to this:

<html>
  <body>
    <form id="form">
      <input id="file" type="file" />
      <input type="submit" value="Send" />
    </form>
  </body>
  <script>
    const form = document.getElementById("form");
    form.onsubmit = function (e) {
      e.preventDefault();
      const file = document.getElementById("file").files[0];
      const fileName = file.name;
      fetch("/update", {
        method: "POST",
        body: file,
        headers: {
          "x-file-name": fileName,
        },
      }).then((response) => {
        if (!response.ok) {
          return alert("File upload failed");
        }
        alert("File upload succeeded");
      });
    };
  </script>
</html>

The request now contains a custom header with a name: x-file-name.

You can now read the filename in the upload handler like this: https://github.com/lasselukkari/aWOT#reading-and-writing-headers

This really is not the most standard way of doing this, but it's at least simple.

AppsByDavideV commented 2 years ago

Thank you so much for your time, I will try asap! I didn't read the link yet, will do soon.

/Davide

lasselukkari commented 2 years ago

Actually now that I'm thinking this again the previous solution was maybe not that smart way of doing this. Semantically more correct would maybe be a PUT request to /files/filename.ext without using the headers. You can then use route parameter to read the filename: 

void upload(Request &req, Response &res) {
  char filename[64];

  req.route("filename", filename, 64);

  File file = SPIFFS.open(filename, "w");

  while (req.left()) {
   // also deal with clients that stop sending data here

    file.write(req.read());
  }

  file.close();
  res.sendStatus(204);
}

void setup() {
  app.put("/files/:filename", &upload);;
}

And the form would then look like

<html>
  <body>
    <form id="form">
      <input id="file" type="file" />
      <input type="submit" value="Send" />
    </form>
  </body>
  <script>
    const form = document.getElementById("form");
    form.onsubmit = function (e) {
      e.preventDefault();
      const file = document.getElementById("file").files[0];
      const fileName = file.name;
      fetch("/files/" + fileName, {
        method: "PUT",
        body: file,
      }).then((response) => {
        if (!response.ok) {
          return alert("File upload failed");
        }
        alert("File upload succeeded");
      });
    };
  </script>
</html>

Again this is completely untested code. I did not even compile any of it.

AppsByDavideV commented 2 years ago

Hi Lasse, I will try it today and report to you the result. Have a nice day.

AppsByDavideV commented 2 years ago

Hi, it is partially working: 1) filename may need a slash before the name (/) otherwise it will be not saved. 2) long files, (say >100kbytes? , not exactly sure) are saved with length =0 bytes and of curse are corrupted.

Anyway it starts to work ;)

AppsByDavideV commented 2 years ago

for point 2, i think files are sent in chunck and after the first one they are not handled.

lasselukkari commented 2 years ago

Yes. That is most likely the case, but actually they are not handled at all and that is why the length is also 0.

What browser are you using. All that i have send the file with normal content length without chunking. Try setting the Transfer-Encoding header to empty string "" and maybe additionally manually set the Content-Length. You can also parse the chunked body manually.

I have not yet added the support for chunked bodys to the library as I have not ever needed it myself but now there is a need so it will get added soon. In the meantime you need to somehow disable the automatic chunking. You can also try to use some other http client library if you are unable to disable it from the fetch.

AppsByDavideV commented 2 years ago

Ok, thanks, let me play with it.

It will be great to add file management to this library in a next release ;)

lasselukkari commented 2 years ago

Can you tell me what browser and OS you are using?

AppsByDavideV commented 2 years ago

Yes: win10 with chrome

lasselukkari commented 2 years ago

Here is a branch with initial support for handling the transfer-encoding header and the chunked request body: https://github.com/lasselukkari/aWOT/tree/transfer-encoding. I still need to update the documentation and verify a few edge cases before this gets releases but you can try it out already.

lasselukkari commented 2 years ago

I just figured out the the branch has problems with larger chunks and chunk extensions. I should have read the spec more carefully. I'll let you know when it's fixed.

AppsByDavideV commented 2 years ago

Hi, no problem. Take your time. I found that the ESP32 core distribution for arduino has a parser for http post multipart upload in the 'server' folder... just in case it could be of some help or can be used directly by loading it.

lasselukkari commented 2 years ago

The problem there is that this library is designed to work with even the smallest Arduino boards. The hard part is not get it done, it's getting it done when you have only 2kb on memory available compared to 520 on the ESP32. I will most likely never add the support for the multipart body directly to the library. How ever I maybe could add an example how to do it. The chunked body is different. Previously it was not possible to handle them without using the client instance directly and this is really error prone.

The branch should be updated to work with larger chunk in about 10 minutes.

AppsByDavideV commented 2 years ago

The problem there is that this library is designed to work with even the smallest Arduino boards. The hard part is not get it done, it's getting it done when you have only 2kb on memory available compared to 520 on the ESP32. I will most likely never add the support for the multipart body directly to the library. How ever I maybe could add an example how to do it. The chunked body is different. Previously it was not possible to handle them without using the client instance directly and this is really error prone.

The branch should be updated to work with larger chunk in about 10 minutes.

Sorry, working with the ESP, I forgot the mission of the library. Great work!

candide33 commented 2 years ago

hi i tested transfer of a file from a browser Firefox to esp with awot webserver, but i am stuck when i want to transfert more then 1500 bytes. by example, if file size is around 4586 bytes, in server i can see req.left = 4806 and req.available = 1565 in this case, i am able to read the 1565 first bytes, but no more...

how to read the total size of file in server side ?

thanks for your help

lasselukkari commented 2 years ago

available() only returns you the the amount of bytes in the buffer available immediately for read. it does not mean the the next call would not return more data. Can you show me your code you are using. Your loop for reading should look something like:

while (req.left()) {
   // also deal with clients that stop sending data here

  file.write(req.read());
}
candide33 commented 2 years ago

Thanks for your help and after more tests, this is what i can see:

Thanks again for this good library.

lasselukkari commented 2 years ago

Unfortunately I have no idea what you mean by header and footer.

candide33 commented 2 years ago

at last, what i wanted to do is working, thanks for your help.

in fact i was working at a wrapper raWOT, to use aWOT library with B4R and now it is working, and a small project is available to show what we can do with aWOT and B4R

more information on raWOT in B4R forum. https://www.b4x.com/android/forum/threads/rawot.135993/

thanks again for this good library and for your support

----- Mail original ----- De: "Lasse Lukkari" @.> À: "lasselukkari/aWOT" @.> Cc: "candide33" @.>, "Comment" @.> Envoyé: Mercredi 10 Novembre 2021 22:27:47 Objet: Re: [lasselukkari/aWOT] Upload files from browser and store them into LittleFS/SPIFFS/SD (Issue #136)

Unfortunately I have no idea what you mean by header and footer.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub , or unsubscribe . Triage notifications on the go with GitHub Mobile for iOS or Android .

gcharles81 commented 2 years ago

Hello Thanks for this great library is there any working example for uploading files from a browser and store them in the SD or SPIFFS Thanks

LOGOLEX commented 2 years ago

Hey, i just tried out to get your examlple to work. But after i upload a File over Ethernet adapter, there is one Byte more at the beginning of the File, so it is corrupted. Upload over Wifi works properly. The uploads add a HEX 0xFF as first Byte I'm running this code on a ESP32 WROOM dev Board with attached ENC28J60

Where is my fault?

Actually now that I'm thinking this again the previous solution was maybe not that smart way of doing this. Semantically more correct would maybe be a PUT request to /files/filename.ext without using the headers. You can then use route parameter to read the filename:

void upload(Request &req, Response &res) {
  char filename[64];

  req.route("filename", filename, 64);

  File file = SPIFFS.open(filename, "w");

  while (req.left()) {
   // also deal with clients that stop sending data here

    file.write(req.read());
  }

  file.close();
  res.sendStatus(204);
}

void setup() {
  app.put("/files/:filename", &upload);;
}

And the form would then look like

<html>
  <body>
    <form id="form">
      <input id="file" type="file" />
      <input type="submit" value="Send" />
    </form>
  </body>
  <script>
    const form = document.getElementById("form");
    form.onsubmit = function (e) {
      e.preventDefault();
      const file = document.getElementById("file").files[0];
      const fileName = file.name;
      fetch("/files/" + fileName, {
        method: "PUT",
        body: file,
      }).then((response) => {
        if (!response.ok) {
          return alert("File upload failed");
        }
        alert("File upload succeeded");
      });
    };
  </script>
</html>

Again this is completely untested code. I did not even compile any of it.

lasselukkari commented 2 years ago

Hi! What does req.available() return when you try to read the first time?

You could try changing your code to something like this:

while (req.left()) {
   if(req.available()){
      file.write(req.read());
  }
}

There are a lot better options than the ENC28J60 for the ESP32 if you need an Ethernet connection.

LOGOLEX commented 2 years ago

Hello lasselukkari,

my test Upload contains 9 bytes. req.available() returns "0" at the first call, on the second call 8. then every call one less.

However, just calling req.available() one time without "if" statement solved my problem. I changed the code to your example and now it works. (eventually you could explain why?)

If there are better options to get Ethernet on ESP32, what do you recommend? With the ENC28j60 i get a 10 MBit/s connection. In this case this is enough. But i would be pleased to have a better option for future projects.

Thank you!

Hi! What does req.available() return when you try to read the first time?

You could try changing your code to something like this:

while (req.left()) {
   if(req.available()){
      file.write(req.read());
  }
}

There are a lot better options than the ENC28J60 for the ESP32 if you need an Ethernet connection.

lasselukkari commented 2 years ago

However, just calling req.available() one time without "if" statement solved my problem. I changed the code to your example and now it works. (eventually you could explain why?)

It's a "feature" of the Ethernet library you are using. Nothing to do with this library.