search

cotestatnt / esp-fs-webserver

ESP32/ESP8266 webserver, WiFi manager and web editor Arduino library
MIT License
121 stars 29 forks source link

NO ISSUE, just question FILE DOWNLOAD ? #39

Open MyRaspberry opened 10 months ago

MyRaspberry commented 10 months ago

play with a ESP32-S3 N16R8 and setup FFAT 10MB partition. and write dir and file (append 1 line / min ) /data/readings.csv

does THIS tool help me to DOWNLOAD that file via Browser?

the example FSBrowser i use, not even print that file (content) my status : at GIT

THX for sharing your work

cotestatnt commented 10 months ago

Hi @MyRaspberry

Yes, it can. Some time ago I've written a sketch to visualize a csv, edit and download I just have to remember where I saved it....

However, it has little to do with the library in question, it's just HTML and Javascript coding.

MyRaspberry commented 10 months ago

i just worry about the file size can grow over the memory of the chip, what also limits the html it can produce dynamically, so how JS could serve big files?

cotestatnt commented 10 months ago

It depends on the size of your csv file. Aniway, this library could serve web pages even from a SD memory card if a lot of space is needed.

Naturally, the larger the size of the csv file, the longer it will take to load the page that displays its content.

This is the result with my ESP32-S3 (only 1.3Mbyte for LittleFS filesystem) using my smartphone as Access Point. Test with a file of approximately 110kb (1000 records) image

and this instead with a file of approximately 1000kb (8000 records) image

If you only need to download, the problem does not arise because it is not necessary to upload the contents of the file to the web page as in this case. It will be sufficient to populate a list with the files to download (if there are more than one) and a button to start the download.

MyRaspberry commented 10 months ago

thanks,

anyhow i started coding the file thing a ? better ? way, making daily files like "data/readings_2024-01-08.csv" will see tomorrow if it works.

and then experiment with your lib

thanks for your time

cotestatnt commented 10 months ago

making daily files like "data/readings_2024-01-08.csv"

This is a great idea. The download page at this point will only have to create a list based on the contents of the filesystem and allow you to download the files.

If you need, I have some working sketches that do similar things (not with csv, but basically it's the same behaviour)

MyRaspberry commented 10 months ago

-1- that first part seems to work: find {"type":"file","name":"data/readings2024-01-09.csv", "size": 21267 B } ( ! size after 8 hours already )_

//update 2024.01.10:
[
{"type":"file","name":"data/readings.csv", "size": 2067 B },
{"type":"file","name":"data/readings_2024-01-08.csv", "size": 19537 B},
{"type":"file","name":"data/readings_2024-01-09.csv", "size": 61945 B },  <<< 62KB /d
{"type":"file","name":"data/readings_2024-01-10.csv", "size": 15032 B }
]

,rec,datetimes,A0,A1,A2, ,1785,2024-01-09 00:00:49,0.00,20.05,1.03, ,1789,2024-01-09 00:01:49,0.00,18.95,0.71, ,1793,2024-01-09 00:02:49,0.00,19.10,0.61, ...

tomorrow i know how big it will be.

-2- i can also now print the content using the server.streamFile String contentType = getContentType(path); File file = FILESYSTEM.open(path, "r"); server.streamFile(file, contentType); file.close();

inside my added function to the FSBrowser code : void handleFilePrint() { // on /print?file=/data/readings.csv

-3- BUT THAT IS NOT A DOWNLOAD ? HOW ? i also failed to find something in your esp-fs-webserver-master\src\esp-fs-webserver.cpp

It will be sufficient to populate a list with the files to download (if there are more than one) and a button to start the download.

sounds like you know how, but i have no idea what you refer to.

-4- i am lost, while years ago i play

and after using Circuit Python just come back to Arduino IDE

now i see that is ?DISCONTINUED?

and having a ESP32-S3 N16R8 module i look into using FFAT where i miss the way to handle any files there regarding UPLOAD & DOWNLOAD.. so finding FSBrowser i hoped that might be the way, but that is very incomplete.

also i miss to see anything ? regarding other tools is there something from the Espressif tools?

or ? i have one of that boards with 2 USB C ports, is there a code ?for Arduino IDE? to use one port of that board as a USB STICK with file access to FFAT? HA, that would be great

MyRaspberry commented 10 months ago

thanks anyhow, i got the basics working.. see GIT

cotestatnt commented 10 months ago

Hi @MyRaspberry

I'm happy you already solved it. In any case, if you want to take a look, I added an example taking inspiration from your needs

https://github.com/cotestatnt/esp-fs-webserver/tree/master/examples/csvLogger

MyRaspberry commented 10 months ago

yes, thanks, i love it, looks great and is very professional,

i see you use littleFS and the /data/ dir in the arduino project, means what? did you use a upload tool? because as i see that you create that log file in the ESP

i assume that i can not use it for my little FFAT exploring project? but i could try it on that esp32-S3 N16R8? BUT NOT IDE 2.2.1 ? or need ESP32? anyhow about your web page / css / js / i could learn a lot. thanks again!

can you open that? my page snap

cotestatnt commented 10 months ago

Of course you can use it! The library is compatible with any type of Espressif microcontroller including the ESP8266.

If you prefer to use FFat, simply include the "FFat.h" file and replace LittleFS with FFat wherever you see it in the sketch.

As for uploading the data folder, it is managed directly by the library using the included /setup page, Update & FS tab.

image

Once you have uploaded your sources, you can edit them directly using the included /editpage Soon I will add a new version of /edit that allows you to download files, but in the meantime you can simply copy and paste to transfer any changes to your PC

Regarding your working HTML code, you can simply put it on a specific page: if you don't want write from beginning, open dev tools of your browser, move to the "source" tab and copy the content. If the name page is index.htm or index.html it will be used as default page

Edit:

In order to use FFat some small chages are needed in the sources because FFat treats the / character in folder paths a little differently. In the file assets/js/index.js edit the function as this listFiles() in order to remove the last '/' from url when a folder is passed to the function

// Fetch the list of files and fill the filelist
function listFiles() {
  var url = '/list?dir=' + dataFolder;
  if (url.charAt(url.length - 1) === '/')
    url = url.slice(0, -1); // Remove the last character

  fetch(url)    // Do the request
  .then(response => response.json())    // Parse the response
  .then(obj => {                        // DO something with response
    fileList.innerHTML = '';
    obj.forEach(function(entry, i) {
      addEntry(entry.name);
    });
    // Load first file
    loadCsv(dataFolder + obj[0].name);    
  });
}
cotestatnt commented 10 months ago

The print screen is from the twin library that uses the ESPAsyncWebServer library for the webserver instead of the one included in the core, but the result does not change (I develop them more or less in parallel)

From some tests I've done it seems to be faster and more responsive.

MyRaspberry commented 10 months ago

as expected nothing is easy.... first ( not yet reading above and with my possibly wrong thinking ) i try a extra working environment

try set laptop wifi to ESP_AP password 123456789 from laptop get:

Oops.

The site at http://www.msftconnecttest.com/redirect has experienced a network protocol violation that cannot be repaired.

The page you are trying to view cannot be shown because an error in the data transmission was detected.

Please contact the website owners to inform them of this problem.

try again to powerup via charger try login via mobile sign in to ESP_AP the webpage at https://192.168.4.1/setup could not be loaded because net::ERR_CONNECTION_REFUSED

cotestatnt commented 10 months ago

I'm sorry @MyRaspberry, but a pull request I've merged some times ago has introduced this bug due to a wrong reply when ESP is acting as captive portal ( "https:://" instead http://).

Once an ESP32 has succesfull connected to a SSID it store the credentials, so I've never realized before about this bug.

With last commit and releases I've fixed it. I've added also some more features like manual WiFi configuration (fixed IP, gatway, subnet)

MyRaspberry commented 10 months ago

sorry, had to fail with a other idea first.....

// 12.1.2024 update to v 1.3.0 and take also that new example

Terminal show continuous booting

05:45:39.592 -> rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
05:45:39.639 -> configsip: 0, SPIWP:0xee
05:45:39.639 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
05:45:39.639 -> mode:DIO, clock div:1
05:45:39.639 -> load:0x3fff0030,len:1184
05:45:39.639 -> load:0x40078000,len:13260
05:45:39.639 -> load:0x40080400,len:3028
05:45:39.639 -> entry 0x400805e4
05:45:39.780 -> ERROR on mounting filesystem. It will be formmatted!
05:45:39.780 -> ets Jun  8 2016 00:22:57
05:45:39.780 -> 
05:45:39.780 -> rst:0xc (SW_CPU_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
05:45:39.780 -> configsip: 0, SPIWP:0xee
05:45:39.827 -> clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
05:45:39.827 -> mode:DIO, clock div:1
05:45:39.827 -> load:0x3fff0030,len:1184
05:45:39.827 -> load:0x40078000,len:13260
05:45:39.827 -> load:0x40080400,len:3028
05:45:39.827 -> entry 0x400805e4
05:45:48.649 -> ets Jun  8 2016 00:22:57
05:45:48.649 ->

again, above i mentioned that i use partition DEFAULT ( whatever that means ) Screenshot 2024-01-12 055415

should i change that?

cotestatnt commented 10 months ago

You are trying to use a FFat based sketch selecting a SPIFFS based partition scheme (the default one), Of course it will not work!

Also, up until now you've always said you have an ESP32-S3, but the board you selected uses a "normal" ESP32 so I'm guessing this is a different board? In this case, select a variant that also has the FFat partition among the choices, or select a generic ESP32 dev-module so you have all the options available.

A curiosity... but why are you so sticky to the FFat partition? If you left LittleFS, the sketch would work as it was without having to specify the partition scheme every time

MyRaspberry commented 10 months ago

oh sorry, -1- my idea was to test a new ESP32-S3 N16R8 with Arduino, so installed Arduino IDE 2.2.1 to learn that too.

-2- my first impression was that i can not use that N16R8 so i dig deeper and that leaded to using FFAT but there i got the feeling that there is a big problem about up and download? so i started to look and try FSBrowser lib.

as i am working and data-collecting to N16R8
in that bigger evaluation project already status

-3- as described above, i did a separate test setup for your lib. -a- laptop -b- Arduino IDE 1.8.19 -c- a NODEMCU ESP32-32s v1.1 as i got the impression you need that Arduino /data/ dir thing where i read that not work in the new IDE?

ok, i changed to LITTLE FS and the old ESP32 is ok,

ok, thank you so much

cotestatnt commented 10 months ago

as i got the impression you need that Arduino /data/ dir thing where i read that not work in the new IDE?

The FS upload plugin for IDE 1.8.x (which don't work with IDE 2.x) is no more necessary because you can upload your files through the esp-fs-webserver library's included web interface.

second try used 192.168.1.213 | 192.168.1.1 | 192.168.1.1

I don't think the subnet mask you used are a valid value. Usually I set as 255.255.255.0

If you edit manual IP or other parameters for no-DHCP configuration, you have to save configuration before start the new connection, otherwise on next restart it will not be used.

MyRaspberry commented 10 months ago

255.255.255.0 my bad

MyRaspberry commented 10 months ago

after some looks i got to say:

-a-

this project needs a FAVICON.ICO like with change in : /data/index.htm < link rel="icon" type="image/x-icon" href="assets/favicon.ico" > and here my idea, pls you use mine if you like it, even i am not a real artist. pls copy from : here

-b-

but just found YOUR /edit page !! very good

i miss info about the 'LittleFS' DRIVE in PSFLASH

-c-

this project csvLogger : funtion is impressive, but the code is 3 level over my head to understand if you look at my GIT you see i just managed to learn .h .cpp class and have html download button browse select one file, but the way seems to be same?

server ON /edit [POST] >> handleFileUpload() >> server.upload()

all just hidden in your complex structure. but i not see where you scan the PC selected /data/ dir and upload ALL files? is it a ONE by ONE via a JS loop?

-d-

just setup other PC

lost all old content, pls verify

looks like the new IDE also messes up about the data dir from example when save to own project

-e-

anyway

thanks and greeting to Rome from Chiang Mai

cotestatnt commented 10 months ago

I'll try to respond to your comments

MyRaspberry commented 10 months ago

now i try 

#include <FS.h>

#include <LittleFS.h>
#define FILESYSTEM LittleFS
ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x44c
load:0x403c9700,len:0xbe4
load:0x403cc700,len:0x2a68
entry 0x403c98d4

[I][esp-fs-webserver.cpp:120] startAP():    AP mode.
 Server IP address: 192.168.4.1

ESP Web Server started on IP Address: 192.168.4.1
Open /setup page to configure optional parameters
Open /edit page to view and edit files

also try:


#include "FFat.h"
#define FILESYSTEM FFat

FLASH size 16MB
Partiton 16MB 3 APP 10 FATFS

[E][esp-fs-webserver.cpp:24] begin():   Failed to open /setup directory. Create new folderESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0xc (RTC_SW_CPU_RST),boot:0x8 (SPI_FAST_FLASH_BOOT)
Saved PC:0x4201b1d4
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x44c
load:0x403c9700,len:0xbe4
load:0x403cc700,len:0x2a68
entry 0x403c98d4

[I][esp-fs-webserver.cpp:120] startAP():    AP mode.
 Server IP address: 192.168.4.1

ESP Web Server started on IP Address: 192.168.4.1
Open /setup page to configure optional parameters
Open /edit page to view and edit files

-1- pls help with the tools board settings for ESP32-S3 -2- why now login problems?

cotestatnt commented 10 months ago

I'm sorry something must have gone wrong with the latest commits... for now I would suggest starting the connection manually with the classic WiFi.begin() and avoiding calling the library startWifi() method

I'll fix this bug and then I'll make a new release.

MyRaspberry commented 10 months ago

ok, n.p. can wait, just wanted to confirm if the rest worked and run pure FFat tools over it

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x44c
load:0x403c9700,len:0xbe4
load:0x403cc700,len:0x2a68
entry 0x403c98d4

================================
Chip Model: ESP32-S3
Chip Revision: 0
with 2 core
Flash Chip Size : 16777216 
Flash Chip Speed : 80000000 
Total Heap : 397212 
Free Heap  : 372320 
PSRAM Total :  8386279, Free : 8386019, Used : 260

Features included:

 2.4GHz WiFi
 Bluetooth LE

FFAT Total :   10240000, Free :   10231808, Used :       8192
Listing directory: /
  DIR : setup
Listing directory: /setup
  FILE: config.json SIZE: 17

no offense, but that guy is wow

MyRaspberry commented 10 months ago

update: try v 1.3.2

cotestatnt commented 10 months ago

Hi @MyRaspberry, thanks for your review! I just published a new major release (due to the big and deep changes).

If you want test also this release, please keep in mind that some small changes are needed on on pre-existing sketches. In any case, the advice is always to start with the examples included.

MyRaspberry commented 10 months ago

try v 2.0.0

cotestatnt commented 10 months ago

Hi @MyRaspberry If you like, try the latest release. I've fixed some of the observations you made such as wrong printing files order. As regards the "Google" IP in access mode, it seems that with this trick you can make the captive portal work even on the most finicky devices (for example some Samsungs smartphones)

Furthermore, in the csvLogger.ino, I have included an example of how it is possible to insert a custom favicon encoded in base64.

The /edit web page must be explicitly enabled. However I have adjusted all the examples to include it (and possibly comment the related instruction)

MyRaspberry commented 10 months ago

v 2.0.1 : test 9 and 10