PR#3646 broke the 'file upload' function in ESPlorer

[serg3295]

Expected behavior

Upload file to ESP

Actual behavior

After applying PR #3646, the 'file upload' function of ESPlorer does not work.

ESPlorer Log:

``` Reconnect: Success. Now we waiting for ESP reboot... Try to reconnect with baud 115200... sending:print(uart.setup(0, 115200, 8, 0, 1, 1 )) Reconnect: Success. Uploader: chooser selected file:1 BinaryFileLoader: Try to load file a.lua ... BinaryFileLoader: Load file a.lua: Success, size:88 1. packets = 0 2. packets = 1 3. packet = 0 BinaryFileLoader: Prepare next packet for send, len=88 BinaryFileLoader: Total packets prepared=1 sendPackets=1 Uploader: Starting... Uploader: Add EventListener: Success. Uploader: Start sending: _up=function(n,l,ll) local cs = 0 local i = 0 local open = file.open or io.open print(">".." ") uart.on("data", l, function(b) i = i + 1 local fh = open("a.lua",'a+') fh:write(b) fh:close() cs=0 for j=1, l do cs = cs + (b:byte(j)*20)%19 end sending: uart.write(0,"~~~CRC-".."START~~~"..cs.."~~~CRC-".."END~~~") if i == n then uart.on("data") end if i == n-1 and ll>0 then _up(1,ll,ll) end end,0) end file.remove("a.lua") sending:_up(1,88,88) Operation done. Duration = 3328 ms Waiting answer from ESP - Timeout reached. Command aborted. ```

The VSCode NodeMCU-tools (marketplace) extension also stopped working. I guess that ESP now sends additional CRLFs over uart.write(0,....) In the extension, the CR LF combination is used as a boundary delimiter and the additional CRLFs change the existing protocol.

Debug output:

// Before PR#3646 ``` C: uart.write(0,tostring(node.chipid()).."\r\n") Buffer(28) [117, 97, 114, 116, 46, 119, 114, 105, 116, 101, 40, 48, 44, 116, 111, 115, 116, 114, 105, 110, 103, 40, 110, 111, 100, 101, 46, 99, buffer: ArrayBuffer(65536), byteLength: 28, byteOffset: 0, length: 28, Symbol(Symbol.toStringTag): 'Uint8Array'] CR: uart.write(0,tostring(node.chipid()).."\r\n") Buffer(32) [104, 105, 112, 105, 100, 40, 41, 41, 46, 46, 34, 92, 114, 92, 110, 34, 41, 13, 10, 48, 120, 54, 101, 55, 99, 57, 101, 98, 100, 48, 55, 54, buffer: ArrayBuffer(65536), byteLength: 32, byteOffset: 28, length: 32, Symbol(Symbol.toStringTag): 'Uint8Array'] CR: 0x6e7c9ebd076b ``` // With PR#3646. Unwanted crlf is marked with asterisks ``` C: uart.write(0,tostring(node.chipid()).."\r\n") CR: uart.write(0,tostring(node.chipid()).."\r\n") CR: Buffer(51) [117, 97, 114, 116, 46, 119, 114, 105, 116, 101, 40, 48, 44, 116, 111, 115, 116, 114, 105, 110, 103, 40, 110, 111, 100, 101, 46, 99, 104, 105, 112, 105, 100, 40, 41, 41, 46, 46, 34, 92, 114, 92, 110, 34, 41, 13, 10, **13, 10,** 62, 32, buffer: ArrayBuffer(65536), byteLength: 51, byteOffset: 0, length: 51, Symbol(Symbol.toStringTag): 'Uint8Array'] R: > 0x6e7c9ebd076b Buffer(19) [48, 120, 54, 101, 55, 99, 57, 101, 98, 100, 48, 55, 54, 98, 13, 13, 10, 62, 32, buffer: ArrayBuffer(65536), byteLength: 19, byteOffset: 51, length: 19, Symbol(Symbol.toStringTag): 'Uint8Array'] ```

Test code

Click the 'Upload...' button to upload a file to ESP

---

Another test case: sending uart.on(0,"data",3,function()print("kxyJ")end, 0) hangs the console

---

strange baudrate

NodeMCU ESP32 build unspecified powered by Lua 5.3.5 [5.3-int32-singlefp] on IDF v5.1.3
cannot open init.lua: No such file or directory
> print(uart.getconfig(0))

> 115942    8   0   1
> uart.setup(0, 115200, 8, uart.PARITY_NONE, uart.STOPBITS_1, 1)

> > print(uart.getconfig(0))

> 115200    8   0   1
> 

NodeMCU startup banner

NodeMCU startup banner:

``` --- esp-idf-monitor 1.4.0 on /dev/ttyUSB0 115200 --- --- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H --- ets Jun 8 2016 00:22:57 rst:0x1 (POWERON_RESET),boot:0x12 (SPI_FAST_FLASH_BOOT) configsip: 0, SPIWP:0xee clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00 mode:DIO, clock div:2 load:0x3fff0030,len:7112 load:0x40078000,len:15616 load:0x40080400,len:4 0x40080400: _init at ??:? ho 8 tail 4 room 4 load:0x40080404,len:3876 entry 0x4008064c I (30) boot: ESP-IDF v5.1.3 2nd stage bootloader I (30) boot: compile time Apr 27 2024 19:20:17 I (31) boot: Multicore bootloader I (35) boot: chip revision: v1.0 I (38) boot.esp32: SPI Speed : 40MHz I (43) boot.esp32: SPI Mode : DIO I (48) boot.esp32: SPI Flash Size : 4MB I (52) boot: Enabling RNG early entropy source... I (58) boot: Partition Table: I (61) boot: ## Label Usage Type ST Offset Length I (68) boot: 0 nvs WiFi data 01 02 00009000 00006000 I (76) boot: 1 phy_init RF data 01 01 0000f000 00001000 I (83) boot: 2 factory factory app 00 00 00010000 00180000 I (91) boot: 3 lfs unknown c2 01 00190000 00010000 I (98) boot: 4 storage Unknown data 01 82 001a0000 00070000 I (106) boot: End of partition table I (110) esp_image: segment 0: paddr=00010020 vaddr=3f400020 size=2f11ch (192796) map I (188) esp_image: segment 1: paddr=0003f144 vaddr=3ff80063 size=00008h ( 8) load I (189) esp_image: segment 2: paddr=0003f154 vaddr=3ffb0000 size=00ec4h ( 3780) load I (196) esp_image: segment 3: paddr=00040020 vaddr=400d0020 size=cb950h (833872) map I (504) esp_image: segment 4: paddr=0010b978 vaddr=3ffb0ec4 size=02d54h ( 11604) load I (508) esp_image: segment 5: paddr=0010e6d4 vaddr=40080000 size=1769ch ( 95900) load I (549) esp_image: segment 6: paddr=00125d78 vaddr=400c0000 size=00064h ( 100) load I (561) boot: Loaded app from partition at offset 0x10000 I (561) boot: Disabling RNG early entropy source... I (573) cpu_start: Multicore app I (573) cpu_start: Pro cpu up. I (573) cpu_start: Starting app cpu, entry point is 0x4008153c 0x4008153c: call_start_cpu1 at /home/serg/Projects/lua/nodeMCU-firmware/sdk/esp32-esp-idf/components/esp_system/port/cpu_start.c:159 I (0) cpu_start: App cpu up. I (591) cpu_start: Pro cpu start user code I (591) cpu_start: cpu freq: 160000000 Hz I (591) cpu_start: Application information: I (596) cpu_start: Project name: nodemcu I (601) cpu_start: App version: 1.4.0-master_20151229-894-g92f3 I (608) cpu_start: Compile time: Apr 30 2024 10:12:23 I (614) cpu_start: ELF file SHA256: f2be832b64d82e0e... I (620) cpu_start: ESP-IDF: v5.1.3 I (625) cpu_start: Min chip rev: v0.0 I (629) cpu_start: Max chip rev: v3.99 I (634) cpu_start: Chip rev: v1.0 I (639) heap_init: Initializing. RAM available for dynamic allocation: I (646) heap_init: At 3FFAE6E0 len 00001920 (6 KiB): DRAM I (652) heap_init: At 3FFB8768 len 00027898 (158 KiB): DRAM I (659) heap_init: At 3FFE0440 len 00003AE0 (14 KiB): D/IRAM I (665) heap_init: At 3FFE4350 len 0001BCB0 (111 KiB): D/IRAM I (671) heap_init: At 4009769C len 00008964 (34 KiB): IRAM I (679) spi_flash: detected chip: generic I (682) spi_flash: flash io: dio W (686) rmt(legacy): legacy driver is deprecated, please migrate to `driver/rmt_tx.h` and/or `driver/rmt_rx.h` W (697) pcnt(legacy): legacy driver is deprecated, please migrate to `driver/pulse_cnt.h` W (706) ADC: legacy driver is deprecated, please migrate to `esp_adc/adc_oneshot.h` I (714) app_start: Starting scheduler on CPU0 I (719) app_start: Starting scheduler on CPU1 I (719) main_task: Started on CPU0 I (729) main_task: Calling app_main() E (799) mmap: esp_mmu_paddr_to_vaddr(752): paddr isn't mapped I (819) wifi:wifi driver task: 3ffc7510, prio:23, stack:6656, core=0 I (819) wifi:wifi firmware version: 0016c4d I (819) wifi:wifi certification version: v7.0 I (819) wifi:config NVS flash: enabled I (819) wifi:config nano formating: disabled I (839) wifi:Init data frame dynamic rx buffer num: 32 I (839) wifi:Init static rx mgmt buffer num: 5 I (839) wifi:Init management short buffer num: 32 I (849) wifi:Init dynamic tx buffer num: 32 I (849) wifi:Init static rx buffer size: 1600 I (849) wifi:Init static rx buffer num: 10 I (849) wifi:Init dynamic rx buffer num: 32 I (869) wifi_init: rx ba win: 6 I (869) wifi_init: tcpip mbox: 32 I (869) wifi_init: udp mbox: 6 I (869) wifi_init: tcp mbox: 6 I (879) wifi_init: tcp tx win: 5744 I (879) wifi_init: tcp rx win: 5744 I (879) wifi_init: tcp mss: 1440 I (889) wifi_init: WiFi IRAM OP enabled I (889) wifi_init: WiFi RX IRAM OP enabled NodeMCU ESP32 build unspecified powered by Lua 5.3.5 [5.3-int32-singlefp] on IDF v5.1.3 cannot open init.lua: No such file or directory ```

Hardware

ESP32 DevKit v4 with USB port.

[bv73]

Latest NodeMCU build for ESP32 and I have the same issue with upload files through ESPlorer and ChiliPeppr ESP32 Web IDE. As far as I know it's because Lua 5.3 version and it's dont have file.open file.writeline etc commands anymore. Instead you must use io.open io.write commands in internal Lua scripts.

https://nodemcu.readthedocs.io/en/dev-esp32/modules/file/

Also I have the same problem with "strange baudrate".

If Lua scripts for certain buttons were external and not built into the application code, then this problem could be solved simply by replacing Lua scripts. Theoretically, you can take the source code and remake it yourself, but I’m more of an amateur programmer than a pro.

[serg3295]

If you build the firmware on the commit (before "Initial support for ESP32-C6 and ESP32-H2,..."), then ESPlorer will work, as it takes into account the changes associated with the io module. My fork of https://github.com/AndiDittrich/NodeMCU-Tool and VSCode NodeMCU-tools (marketplace) extension will also work.

Commit https://github.com/nodemcu/nodemcu-firmware/commit/4cdebe71915a20531fbefe315556a34017a2845b does not allow you to install a uart.on() callback handler. I can rewrite the protocol for both NodeMCU-Tool and VSCode extension nodemcu-tools, but I don't understand how to do it in the best way possible yet.

[marcelstoer]

I don't see a good way out of this other than to fix this on our end. @jmattsson could we maybe get the ability back to install a uart.on() callback handler?

[serg3295]

That would be great!

One more question. nodemcu-tool and nodemcu-tools applications use regex to distinguish between esp8266 and ESP32

// node.chipid() => esp32 chipid (hex with '0x' prefix)
      this._espArch = this._espID.match(/^0x\w+/) ? 'esp32' : 'esp8266'

However, chipid is not applicable for esp32-s2, esp32-h6 etc.

To ensure compatibility with the applications, I suggest making small changes to node.chipid()

// Lua: node.chipid()
static int node_chipid( lua_State *L )
{
#if defined(CONFIG_IDF_TARGET_ESP32)
  // This matches the way esptool.py generates a chipid for the ESP32 as of
  // esptool commit e9e9179f6fc3f2ecfc568987d3224b5e53a05f06
  // Oddly, this drops the lowest byte what's effectively the MAC address, so
  // it would seem plausible to encounter up to 256 chips with the same chipid
  uint64_t word16 = REG_READ(EFUSE_BLK0_RDATA1_REG);
  uint64_t word17 = REG_READ(EFUSE_BLK0_RDATA2_REG);
  const uint64_t MAX_UINT24 = 0xffffff;
  uint64_t cid = ((word17 & MAX_UINT24) << 24) | ((word16 >> 8) & MAX_UINT24);
  char chipid[17] = { 0 };
  sprintf(chipid, "0x%llx", cid);
#else
  const char *chipid = "0x_unknown";
#endif
  lua_pushstring(L, chipid);
  return 1;
}

OR

  esp_chip_info_t chip_info;
  const char *chip_model;

  esp_chip_info(&chip_info);
  switch (chip_info.model) {
    case CHIP_ESP32:
        chip_model = "0xESP32";
        break;
    case CHIP_ESP32S2:
        chip_model = "0xESP32-S2";
        break;
    case CHIP_ESP32S3:
        chip_model = "0xESP32-S3";
        break;        
    case CHIP_ESP32C3:
        chip_model = "0xESP32-C3";
        break;        
    case CHIP_ESP32C2:
        chip_model = "0xESP32-C2";
        break;        
    case CHIP_ESP32C6:
        chip_model = "0xESP32-C6";
        break;        
    case CHIP_ESP32H2:
        chip_model = "0xESP32-H2";
        break;        
    default:
        chip_model = "0x_Unknown Model";
        break;        
}
  lua_pushstring(L, chip_model)

Which way do you think is the most correct?

[jmattsson]

Breaking the uart.on() functionality was certainly not intentional, nor was breaking ESPlorer (again). I'll try to have a look this weekend and sort it out.

[jmattsson]

Okay, I've submitted PR #3652 - please take a look. It should address:

• CR/LF translation
• Inability to use uart.on('data') on the console uart
• No way to determine chip model

For the latter, rather than try to overload node.chipid(), I've introduced a much simpler node.chipmodel() function which can be used to check what series ESP32 NodeMCU is running on.

Also, the 0x in the return value from node.chipid() denotes the numeric value is presented in hexadecimal rather than regular decimal.

[bv73]

I apologize for misunderstanding of the situation. The last time I worked with a real ESP32 project was about a year ago and since then I practically haven’t looked here because... I thought that the NodeMCU project was abandoned. Only a few days ago I saw that there is support for new crystals. By the way, I didn’t find any mention of idf.py set-target esp32c3 in the documentation, so everything had to be looked for on the back streets of the Internet. The first build of the latest fresh dev-esp32 assembled a binary and after flashing the firmware and launching the board produced a strange error at the very beginning of initialization that SPIFFS could not mount the file system, formatting did not work and such firmware was useless. At first I noticed that in the new and old firmware the parameters of the fourth partition were different and I thought that the error was due to this: boot: 4 nodemcuspiffs unknown c2 00 and boot: 4 storage Unknown data 01 82 I replaced components/platform/partitions.csv with the old one, but nothing changed. Then, after much torment, I completely deleted the nodemcu-firmware-esp32 directory and cloned everything again. And lo and behold, formatting works after flashing the firmware. But, as mentioned above, when trying to upload files through the Save to ESP button, the explorer produced a bunch of errors, or my favorite Upload button froze. What at first seemed like an incompatibility between Lua versions, such as a mismatch between the old file.open and the new io.open, turned out to be much more complicated. In addition to the Lua version upgrade, there was an upgrade in the Espressif IDE version, which, as I understand it, led to many associated bugs and incompatibilities. Just today I found out that the author of ESPlorer made changes in the source code to support ESP32 but did not release it and I was using an ancient version of ESPlorer. I had to learn how to collect Java sources, fortunately it turned out to be not as difficult as I thought. So, now with the latest updated version of ESPlorer, the Save to ESP button works, but unfortunately Upload freezes as before. I can't upload arbitrary files to the file system. For example mypage.html As I understand it, after the latest changes @jmattsson this did not affect dev-esp32 and after the commands as indicated in the documentation: git pull origin dev-esp32 git submodule init #only if repo was cloned w/o submodules init git submodule update --recursive I didn’t see any changes in the firmware build and I can’t check it. Or maybe you would be so kind as to tell me how to check the latest changes. Clone the source from the @jmattsson repository? But, I'm glad that things have moved forward and maybe soon, thanks to your joint efforts, I will be able to try my latest project on a tiny ESP32C3 board.

[serg3295]

So, now with the latest updated version of ESPlorer, the Save to ESP button works, but unfortunately Upload freezes as before. I can't upload arbitrary files to the file system.

This code fix "Upload..." button :-) I've tested that on the PR #3652.

        log("sendPackets=" + Integer.toString(sendPackets.size()));
        String cmd = "_up=function(n,l,ll)\n"
                + "     local cs = 0\n"
                + "     local i = 0\n"
                + "     local open = file.open or io.open\n"
                + "     print(\">\"..\" \")\n"
                + "     uart.start(0)\n"
                + "     uart.on(\"data\", l, function(b) \n"
                + "          i = i + 1\n"
                + "          local fh = open(\"" + UploadFileName + "\",'a+')\n"
                + "          fh:write(b)\n"
                + "          fh:close()\n"
                + "          cs=0\n"
                + "          for j=1, l do\n"
                + "               cs = cs + (b:byte(j)*20)%19\n"
                + "          end\n"
                + "          uart.write(0,\"~~~CRC-\"..\"START~~~\"..cs..\"~~~CRC-\"..\"END~~~\")\n"
                + "          if i == n then\n"
                + "               uart.on(\"data\")\n"
                + "               uart.stop(0)\n"
                + "          end\n"
                + "          if i == n-1 and ll>0 then\n"
                + "               _up(1,ll,ll)\n"
                + "          end\n"
                + "          end,0)\n"
                + "end\n"
                + "file.remove(\"" + UploadFileName + "\")\n";

[bv73]

@serg3295 Thanks for rapid responce! If it works then I dont need more. I've added uart.start(0) and uart.stop(0) into ESPlorer code and recompiled it but without committing @jmattsson to the dev-esp32 I wont see any changes.

[bv73]

I answer to myself. If you want to apply the changes from the pull request onto your current branch git pull origin pull/3652/head

Yes. Button Upload is working now.

[bv73]

Unfortunatly button Upload have a bug. When I upload any file without 0x0D bytes then ESPlorer says OK. When I upload, for example, file favicon.png then ESPlorer says Fail but file saves on disk. But after debugging my Lua code I perceived interested thing - all bytes 0x0D was changed to 0x0A. After uploading testing binary file I have the same bug. HexDump of the file prove that. Perhaps it’s not even the button’s fault. everything works on the old firmware from two years ago

[jmattsson]

@bv73 This could potentially be because you still have the IDF-default CR/LF configuration in your sdkconfig file. I only changed the sdkconfig.defaults, which would not override any existing full sdkconfig file. You can check with make menuconfig and then looking under Component config > Newlib and making sure both input and output is set to just LF.

[bv73]

@jmattsson I changed Component config > Newlib settings both to LF and recompiled firmware. The result is the same. After trying upoad favicon.png:

--HexDump start 00000000 89 50 4E 47 0A 0A 1A 0A 00 00 00 0A 49 48 44 52 �PNG........IHDR

for comparison with old ESP32 firmware:

00000000 89 50 4E 47 0D 0A 1A 0A 00 00 00 0D 49 48 44 52 �PNG........IHDR

[jmattsson]

Huh. That should not be the case. According to the documentation all line ending translation should be disabled when the setting is at LF. I am puzzled. If you have the time, could you please try the other newlib line ending settings too for comparison?

[bv73]

@jmattsson I tried all combinations of LF CR CRLF for UART input and UART output but unfortunately the result is still the same.

[serg3295]

I tried all combinations of LF CR CRLF for UART input and UART output but unfortunately the result is still the same.

I've tried the same.

[jmattsson]

@bv73 Okay, this was totally my fault for managing to hardcode line ending settings 😬 I've updated the PR to actually use the Kconfig settings for line endings. I believe this fixes it, but please let me know if it doesn't.

[bv73]

@jmattsson After your updated code I was able to upload favicon.png without any problem. Before that I've set UART output to CRLF and UART input to LF.

Thanks a lot to you and of course to @serg3295 for your efforts!

Some offtopic: Previously, there was a convenient setting in the menuconfig: Serial flasher config -> Default serial port and I've used to /dev/ttyUSB0. But now I dont know where to write this line in new menuconfig

[jmattsson]

@bv73 Excellent, great to hear it! Thank you for testing! To answer the off-topic part - see Ivan's answer in https://github.com/espressif/esp-idf/issues/5051

[jmattsson]

This should now be fixed by PR #3652 having been merged.