search

grblHAL / Plugin_WebUI

ESP3D-WEBUI backend for networking capable drivers/boards with SD card support
Other
10 stars 0 forks source link

WebUI with SD Card #4

Closed troth530 closed 2 years ago

troth530 commented 2 years ago

First of all I don't know if this is a SD card issue or an issue with WebUI, but I posted it here regardless. I am using a 4GB SD card on a Teensy 4.1 and it is formatted to Fat32.
1) With WEBUI_INFLASH=0 and no files in the www directory of the SD card, $f shows files only. Of course, WebUI does not work "404: The requested file cannot be found." 2) With WEBUI_INFLASH=1 and no subdirectories (no files in the www directory) $f or file listing in WebUI shows files as expected. WebUI is fully functional, but upon creating a subdirectory on the SD card it will crash the teensy requiring a reboot. 3) With WEBUI_INFLASH=1 and macros and preferences in the www directory. $f or file listing works through serial, but will not show the www subdirectory. WebUI does read the macros and preferences, but refreshing the SD card or upon creating a subdirectory on the SD card will crash the teensy requiring a reboot.

Streaming through sd card works great.

Thank you!

troth530 commented 2 years ago

I confirmed this is a WebUI issue.
I copied a file with a valid file extension to the www directory and it is read through serial.

[NEWOPT:ENUMS,RT+,HOME,TC,SED,ETH,FTP,SD] [FIRMWARE:grblHAL] [NVS STORAGE:*FLASH] [DRIVER:iMXRT1062] [DRIVER VERSION:220517] [DRIVER OPTIONS:USB.2] [BOARD:T41U5XBB] [AUX IO:4,3,0,0] [IP:0.0.0.0] [PLUGIN:WebUI v0.02] [PLUGIN:SDCARD v1.05] ok $fm ok $f [FILE:/alt65k.nc|SIZE:23779652] [FILE:/CMakeLists.txt|SIZE:4998] [FILE:/alt65k.txt|SIZE:23779652] [FILE:/alt65k.gc|SIZE:23779652] [FILE:/rotary.gc|SIZE:283] [FILE:/Stepper Motor Calculator.txt|SIZE:135|UNUSABLE] [FILE:/greyscale_raster_test_65000.nc|SIZE:556266] [FILE:/Kasia_Updated_30k.gc|SIZE:5287535] [FILE:/www/alt65k.nc|SIZE:23779652]

terjeio commented 2 years ago

I am not able to crash the teensy when testing. Note that the $f command filters the files and the www subdirectory will not show up unless there is a file that is not filtered present in it.

troth530 commented 2 years ago

Hi, yes I'm aware of the filters. Teensy will not crash from command line (using $F) or from IOsender, but will crash in WebUI. WebUI without directories reads fine. $F or IOSender both read valid file types in www directory (or any directory). But the presence of any directory in WebuUI will not allow it read SD card through WebUI and it will crash in WebUI.
Thank you!

terjeio commented 2 years ago

I can create and delete directories from the WebUI as I please without issues. Which version of the Teensyduino are you using? Versions prior to v1.54 has issues that could be a reason for your problem.

troth530 commented 2 years ago

Hi, I was using Arduino IDE with Teensyduino 1.56. I just reconfigured for PIO and it compiles with Networking and SD but when I set WEBUI_ENABLE to 1, when it does not compile and get this error.

I performed the lwip patching (I added altcp.h to _\iMXRT1062\grblHALTeensy4.pio\libdeps\teensy41\lwip\src\lwip and overwrote \iMXRT1062\grblHAL_Teensy4.pio\libdeps\teensy41\lwip\src\lwiopts.h)

Executing task: C:\Users\troth530.platformio\penv\Scripts\platformio.exe run <

Processing teensy41 (board: teensy41; platform: teensy@4.12; framework: arduino) ---------------------------------------------------------------------------------------------------------------------Verbose mode can be enabled via -v, --verbose option CONFIGURATION: https://docs.platformio.org/page/boards/teensy/teensy41.html PLATFORM: Teensy (4.12.0) > Teensy 4.1 HARDWARE: IMXRT1062 600MHz, 512KB RAM, 7.75MB Flash DEBUG: Current (jlink) External (jlink) PACKAGES:

Environment Status Duration

teensy41 FAILED 00:00:07.881 ======================================= 1 failed, 0 succeeded in 00:00:07.881 =======================================The terminal process "C:\Users\troth530.platformio\penv\Scripts\platformio.exe 'run'" terminated with exit code: 1.

Terminal will be reused by tasks, press any key to close it.

terjeio commented 2 years ago

You have to modify vfs.h (and vfs.c?) to fix this. Or is it fixed in v1.54 of the framework since that compiles ok? My guess is that Time.h is found by Windows instead of time.h due to Windows not having a case sensitive filing system. So this migth be a platformio or Arduino bug.

troth530 commented 2 years ago

Arduino: uses teensyduiono 1.56, will compile, but I made a few changes to get it to compile. The appropriate libraries to include is somewhat unclear. Can you share you ino file. I had 3 versions of SD (one from teensyduino, one from arduino, one from a previous build of grblHal Teensy) more than 1 seems problematic. I have the WebUI issue with SD card crashing with subdirectories present on SD card during refresh or upon creation of a subdirectory on a sd card with no subdirectories which you could not replicate, so I started from the start with PIO.

PIO: I started with a fresh clone and followed the instructions exactly per your wiki, but in fact forcing the platform to 4.12 (from platformio.ini ${common_teensy.platform}@4.12) forces Teensyduino to 1.53. I don't know exactly how to modify vfs.h and vfs.c or even which one because grblhal has these files in the Ethernet directory and vfs.h and vfs.cpp are also in \grblHAL_Teensy4.pio\libdeps\teensy41\lwip\libraries\lwip-ftpd-sd\src.

As far as time.h or Time.h goes, here are the locations C:\Users\troth530.platformio\packages\framework-arduinoteensy\libraries\Time\Time.h C:\Users\troth530.platformio\packages\toolchain-gccarmnoneeabi\arm-none-eabi\include\time.h

Overall given the issues with 1.53, we are stuck with 1.53 for Platform IO and with Arduino, it's difficult to tell where things went wrong due to all the changes needed to get it to compile and it is not working out of the box in either case unless you have another recommendation. Please advise and thank you.

terjeio commented 2 years ago

Can you share you ino file.

Here it is.

I had 3 versions of SD (one from teensyduino, one from arduino, one from a previous build of grblHal Teensy) more than 1 seems problematic.

The SD library grblHAL uses is a patched version of this driver for FatFS. The current SD card plugin will only compile/work with FatFS.

but in fact forcing the platform to 4.12 (from platformio.ini ${common_teensy.platform}@4.12) forces Teensyduino to 1.53.

1.53 has a serious bug(s) and should not be used for grblHAL - IMO platformio should update their version.

I don't know exactly how to modify vfs.h and vfs.c or even which one because grblhal has these files in the Ethernet directory and vfs.h and vfs.cpp are also in \grblHAL_Teensy4.pio\libdeps\teensy41\lwip\libraries\lwip-ftpd-sd\src.

Platformio has their own lwIP variant? This is the one that is to be used with grblHAL or code has to be changed. Or perhaps you can get around the problems by deleting or renaming Time.h as Windows will pick the wrong one if found earlier than time.h. You may also delete the lwip-ftpd-srch folder as I am pretty sure this is not part of a standard/clean lwip stack. But again, 1.53 is broken...

troth530 commented 2 years ago

@terjeio Thank you. Yes I've used the patched sd_sdhc.c and made the changes in sd_config.h. I am doing more testing in Arduino. In addition to issues reading directories on SD card through WebUI, I am finding issues with SD card data when uploading through WebUI- missing and repeated characters in the gCode. It runs gCode that has been copied from PC, but uploading is not correct.

Looks like PIO is a dead end until teensyduino >1.53 is available, but yes renaming Time.h forced it to read the correct time.h, but that introduced a new set of error such as "HTTP v0.9 support has been removed!" and of course I have added altcp.h to the lwip directory and overwrote lwipopts.h from the patched files in your repo.

grblHAL_Teensy4_Upload.zip

Thank you!

troth530 commented 2 years ago

I should also mention several errors occur due to duplicated definitions that I was able to bypass by removing fs.c and httpd.c. from the \Arduino\libraries\teensy41_ethernet-master\src\apps\httpd directory

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\grblHAL_Teensy4\networking\fs.c.o: In function `fs_open':

C:\Users\troth530\Documents\Arduino\libraries\grblHAL_Teensy4\src\networking/fs.c:152: multiple definition of `fs_open'

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\fs.c.o:C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/fs.c:63: first defined here

c:/program files (x86)/arduino/hardware/tools/arm/bin/../lib/gcc/arm-none-eabi/5.4.1/../../../../arm-none-eabi/bin/ld.exe: Disabling relaxation: it will not work with multiple definitions

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\grblHAL_Teensy4\networking\fs.c.o: In function `vf_write':

C:\Users\troth530\Documents\Arduino\libraries\grblHAL_Teensy4\src\networking/fs.c:57: multiple definition of `fs_close'

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\fs.c.o:C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/fs.c:63: first defined here

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\grblHAL_Teensy4\networking\fs.c.o: In function `vf_write':

C:\Users\troth530\Documents\Arduino\libraries\grblHAL_Teensy4\src\networking/fs.c:57: multiple definition of `fs_read'

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\fs.c.o:C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/fs.c:63: first defined here

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\grblHAL_Teensy4\networking\fs.c.o: In function `vf_write':

C:\Users\troth530\Documents\Arduino\libraries\grblHAL_Teensy4\src\networking/fs.c:57: multiple definition of `fs_bytes_left'

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\fs.c.o:C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/fs.c:63: first defined here

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\grblHAL_Teensy4\networking\httpd.c.o: In function `httpd_init':

C:\Users\troth530\Documents\Arduino\libraries\grblHAL_Teensy4\src\networking/httpd.c:2066: multiple definition of `httpd_init'

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\httpd.c.o:C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/httpd.c:2558: first defined here

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\fs.c.o: In function `fs_open':

C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/fs.c:71: undefined reference to `fs_open_custom'

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\fs.c.o: In function `fs_close':

C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/fs.c:105: undefined reference to `fs_close_custom'

C:\Users\troth530\AppData\Local\Temp\arduino_build_25963\libraries\teensy41_ethernet-master\apps\httpd\fs.c.o: In function `fs_read':

C:\Users\troth530\Documents\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd/fs.c:136: undefined reference to `fs_read_custom'

collect2.exe: error: ld returned 1 exit status

terjeio commented 2 years ago

First of all, do not modify _grblHAL_Teensy4Upload.ino, the project will compile with the supplied one.

I am finding issues with SD card data when uploading through WebUI- missing and repeated characters in the gCode.

This is exactly the issue I had before I added the patch, are you sure you have modified the files used by the compiler and not some other copy? A nasty thing about the Arduino IDE is that it copies all the source code to a temp folder and uses that for compilation. Sometimes that fails and in may leave the source in an inconsistent state, the only way to fix this is to close the IDE and remove the temp folder(s) and try again.

but that introduced a new set of error such as "HTTP v0.9 support has been removed!"

You will get this error if the wrong lwipopts.h has been patched. Line 519 should be #define LWIP_HTTPD_SUPPORT_V09 0 in the patched version. This will supress the error.

troth530 commented 2 years ago

Hi @terjeio.
PIO Removing Time.h from C:\Users\troth530.platformio\packages\framework-arduinoteensy\libraries\Time\ along with the correct lwipopts.h enabled me to compile in fact SD card can read subdirectories. But this is Teensyduino 1.53 which is broken.

Arduino I'm using Teensyduino 1.56 and after clearing out the ino file and applying all the patches I still have the same behavior... I cannot read directories in WebUI and uploaded data to SD card is not correct with repeated or missing characters.
1) Arduino does not seem to use MSC library and 2) I still get these multiple definitions errors (see txt file) which require me to remove fs.c and httpd.c from C:\Users\troth530\Arduino\libraries\teensy41_ethernet-master\src\apps\httpd.
I think you should look into these 2 possible causes somehow related to Teensy 1.56 as I seem to be coming back to this issue in Arduino only.

MultDefErrors.txt

Thank you! .

terjeio commented 2 years ago

  1. I have updated to 1.56 and adding/deleting directories still works for me. Zip and attach your _utility/sdsdhc.c and I'll run a test with that.

  2. The readme for the lwip patch was not updated as it should on a commit in january. It is now ok and it tells you to delete the apps/httpd folder. Sorry about missing that.

troth530 commented 2 years ago

I zipped this file from C:\Users\troth530\Documents\Arduino\libraries\uSDFS-master\src\utility, but it matches the patched file on your repo.
sd_sdhc.zip

Are you using Platform IO or Arduino?

  1. The readme for the lwip patch was not updated as it should on a commit in january. It is now ok and it tells you to delete the apps/httpd folder. Sorry about missing that.

It was easy enough to figure out, but I was just trying to track with your results and I was not sure if I broke something by removing those files. Thank you!

terjeio commented 2 years ago

I am using Arduino. Can you introduce an error in sd_sdhc.c and check if it get picked up by the compiler?

troth530 commented 2 years ago

@terjeio yes introduced errors in sd_sdhc.c gets picked up by the compiler.
What version of uSDFS are you using and are you using USE_MSC? I've been using uSDFS 1.1.3 in Arduino with USE_MSC set to zero. This exhibits the issues I've been reporting with WebUI SD card data errors and If I set USE_MSC to 1, I get a bunch of errors in USB_Host_t36.h from Teensyduino 1.56.

MSC_Errors.txt

Platform IO has been using uSDFS 1.1.2, USE_MSC=1 but of course with Teensyduino 1.53, and does not exhibit the issue.

After installing 1.56, I cannot seem to install Teensyduino 1.54 on this PC. I will do a fresh build of Arduino on a new PC.

terjeio commented 2 years ago

yes introduced errors in sd_sdhc.c gets picked up by the compiler.

Ok, then the patched file is used.

What version of uSDFS are you using and are you using USE_MSC?

The latest from the repo, no I am not using USE_MSC. Here is my library folder: uSDFS-master.zip

Platform IO has been using uSDFS 1.1.2, USE_MSC=1 but of course with Teensyduino 1.53, and does not exhibit the issue.

That is odd since IIRC the same problem was present in that version as well.

Would you like me to upload a binary for you to test?

troth530 commented 2 years ago

@terjeio, I tried on another Win10 PC with Ardiuno 1.8.16 and teensyduino 1.56 and same result- WebUI would not read directories on the SD card. I happened to have an older Win7 PC with Arduino 1.8.15, teensyduino 1.54 and it worked a little. WebUI was able to read directories, but gCode uploads are still having missing or repeated characters. As before reading from SD card is flawless. Uploads to SD card through IO sender seemed to work perfectly. I think WebUI is using HTTP and IO sender is using FTP. I moved working libraries over from Win7 machine to Win10 machine (Win 10 still had Arduino 1.8.16 and Teensy 1.56) and it still will not read directories in WebUI. I will compare verbose compilation reports and try to find the reason this is happening.

Yes, I would like to test also with your hex file, just to confirm.

troth530 commented 2 years ago

@terjeio I found that if you set Optimize code to "Fast" or "Faster" it would read directories in WebUI, still repeats or misses characters in WebUI SD card uploads. Fastest will not compile due to space and the issue is previously I've been using "Smallest Code" as I presumed it was necessary in cases when "fastest" does not have enough memory space. So fast or faster works regardless of OS, Arduino version or Teensyduino version. So half of the mystery is solved, but duplicated or missing characters is still happening.

terjeio commented 2 years ago

I think WebUI is using HTTP and IO sender is using FTP.

Correct.

Optimize is set to "Faster" for my compilation, odd that the code does not work if set differently. Binary here:

http://www.io-engineering.com/downloads/grblHAL_Teensy4_220701_WebUI.zip

Have you tried more than one brand of SD card?

troth530 commented 2 years ago

@terjeio, I've tried many SD cards. All have missing or repeated characters. I have the best results with 4GB, 512 sector size and of course FAT32. I found that some PC's don't have the issue and those PCs are from my day job and use settings managed by my company and use VPN. I see the same results on Win 7 and Win 10 PC's. I got the same result with your hex file.

terjeio commented 2 years ago

There is a bug somewhere that corrupts file data. It is not related to the M7 cache issue I fixed earlier and writing single sectors to the file seems to circumvent the bug. A bit strange is that not all files are beeing corrupted...

The uploaded file is written here, can you relace the highlighted code with this snipped and check on your end?

    case Upload_Write:
        {
            size_t count;

            char* d = data;
            size_t s = size, t;

            while(s) {

                t = s > 512 ? 512 : s;

            if(upload->to_fatfs) {
                f_write(upload->file.fatfs_handle, d, t, &count);
            } else
                count = fwrite(data, sizeof(char), size, upload->file.handle);
            if(count != t)
                upload->state = Upload_Failed;
            upload->uploaded += count;
            s -= t;
            d += t;
            }
        }
        break;

Note that the code above is a hack - I'll have to figure out where the real bug is, kind of hard since the Teensy is not debuggable...

FYI I have tested with ESP32 and STM32F7xx drivers - these do not show this problem.

troth530 commented 2 years ago

@terjeio this did not completely fix the issue. Instead of the error being a single extra carriage return at the beginning and an error approximately every 32 lines I now have only 2 errors. 1) 2 extra carriage returns at the beginning of the file and missing carriage return +G1 on original line 32. image Thank you testFiles.zip

terjeio commented 2 years ago

This is even more odd, with the above hack in place your testfile is uploaded correctly for me.

The corruption appears to happen at sector boundaries (multiples of 512 bytes). Reducing TCP_MSS in lwipopts.h to 512 seems to fix it as well as this also results in 512 or less bytes beeing written per f_write call. This indicates that writing more than one sector per f_write call is triggerering the issue. ftp does not write more than one sector per f_write call so appears to work. Moving the data structures to non-cached memory (by static allocation) does not help so it is not related to the M7 cache issue I had earlier. Splitting multiple sector writes in to separate calls per sector the _sdsdhc.c lowlevel driver code does not fix the issue indicating that the problem appears in the fatfs code. This is odd since I use the exact same fatfs code for other MCUs without any issues...

It is a pity that the Teensy is not debuggable - I need to add code to write data to a debug stream to find out where the corruption happens. This will likely take a bit of time to complete so be patient.

troth530 commented 2 years ago

Reducing TCP_MSS in lwipopts.h to 512 seems to fix it

This fixes it. These things are incredibly complicated and we appreciate you work.
I also can appreciate that you'd prefer to have the driver accommodate all device specific items and these workarounds that are not your fault make your code harder to maintain. Thank you!

terjeio commented 2 years ago

From the iMXRT1060 RM: "Since the address must be word (4 bytes) aligned, ..."

FatFS passes (user) data buffers that contains a full sectors directly to the lowlevel driver. If these are not word aligned the sectors will be written with data from outside of the buffer. As a workaround I have modified _sdsdhc.c to dynamically allocate a buffer (wich is always word aligned) and copy the data to be written to that if the provided buffer is not word aligned. IMO FatFS should be modified to handle such situations?

Here is the updated sd_sdhc.zip for you to test with:

sd_sdhc.zip

troth530 commented 2 years ago

@terjeio, that updated sd_sdhc.c fixes the bug after reverting the other files (lwipopts.h and http_upload.c) to original files from your repo. I will test with 4k and 512bit sector sizes and across platforms and browsers including those that did not show the bug previously.

Overall the first fix was to set Optimize:Faster in Arduino and this updated sd_sdhc.c file.

Please consider also updating the wiki to indicate that libraries are not added but rather unzipped to C:\Users\\Documents\Arduino\libraries. Adding the library will add include statements to the .ino file and it will create some issues.

Issue closed! Thank you!