Add esptool --port-filter SN filter. (ESPTOOL-950)

cederom commented 2 weeks ago

Is your feature request related to a problem?

If there are several boards attached to a single build machine of the same VID:PID and Name it is impossible to distinguish them when no SN can be provided to port filter.
The perfect unique identifier of a board is its USB SN string.

Describe the solution you'd like

Please add Serial Number (SN) field to --port-filter functionality.

Describe alternatives you've considered

Complex scripting must be done so far to know what exact device is at each /dev/cuaU* device.
Some develkits (i.e. ESP32-DevKitC with CP2102 chip) can have Name and SN fields customized that helps a lot with identification, but still there is no SN filter yet in esptool.

Additional context

https://github.com/espressif/esptool/issues/1026

Dzarda7 commented 2 weeks ago

Hello @cederom, thanks for the suggestion, this sounds reasonable. I will take a look at it and try to implement the feature as soon as possible.

Not sure if you are aware of it, but on linux machines you can make alias for each USB port which will stay the same. It is not the solution how to distinguish each esp chip, but as long as you connect chip into the same port, this works.

Nevertheless I will take a look at it.

cederom commented 2 weeks ago

Thank you @Dzarda7 :-)

I am working on FreeBSD :-) Sure various OS have options to manage USB devices on attach (i.e. devd/devfs). But having option straight in the esptool is the most universal solution and requires no system modification (i.e. this can be dynamically created VM). You can use underlying pyserial functionalities :-)

cederom commented 1 week ago

BIG THANK YOU @Dzarda7 !! :-)

cederom commented 1 week ago

Do you know when it will land in a release and the time of release? We will update the system package :-)

Dzarda7 commented 1 week ago

Will let you know tomorrow.

Dzarda7 commented 1 week ago

We are not sure when the next release will happen, but we can make dev release if this is important for you, just let me know.

Dzarda7 commented 1 week ago

Esptool v4.9.dev1 will be released tomorrow, but won't be part of the esp-idf yet, if you are interested, you can update the version in esp-idf by yourself as stated in the documentation.

cederom commented 1 week ago

Thank you @Dzarda7 this is perfectly fine we only need esptool package on FreeBSD :-) BIG THANK YOU!! :-)

Dzarda7 commented 1 week ago

Esptool v4.9.dev1 was released.

cederom commented 1 week ago

@Dzarda7: Esptool v4.9.dev1 was released.

Thanks.. but why not 4.8.2 / 4.9.0 on GitHub? We use GH releases for packages. Or is it only test preview? Will test and report back in a moment :-)

Dzarda7 commented 1 week ago

We do not have enough features for the release and we do not do dev releases on github. If you are looking for the built binaries, you can find them for example here.

cederom commented 1 week ago

Okay, good idea to test before a release, that port filter does not work for me, and I need to build most of the pip packages as binary packages are usally not provided for FreeBSD :-)

(venv3.9embedded) uname -a
FreeBSD octagon 13.3-RELEASE-p7 FreeBSD 13.3-RELEASE-p7 GENERIC amd64

I have updated esptool to version 4.9.dev1.

(venv3.9embedded) esptool.py version
esptool.py v4.9.dev1
4.9.dev1

I have connected ESP32-DevKitC only to USB having it at /dev/cuaU0.

(venv3.9embedded) esptool.py chip_id
esptool.py v4.9.dev1
Serial port /dev/cuaU0
Connecting...............
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting.........
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 8c:4b:14:81:0b:ec
Uploading stub...
Running stub...
Stub running...
Warning: ESP32 has no Chip ID. Reading MAC instead.
MAC: 8c:4b:14:81:0b:ec
Hard resetting via RTS pin...

Checked the serial string (custom / modified ):

(venv3.9embedded) usbconfig -d 0.6 dump_device_desc
ugen0.6: <Silicon Labs ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)

bLength = 0x0012
bDescriptorType = 0x0001
bcdUSB = 0x0110
bDeviceClass = 0x0000  <Probed by interface class>
bDeviceSubClass = 0x0000
bDeviceProtocol = 0x0000
bMaxPacketSize0 = 0x0040
idVendor = 0x10c4
idProduct = 0xea60
bcdDevice = 0x0100
iManufacturer = 0x0001  <Silicon Labs>
iProduct = 0x0002  <ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d>
iSerialNumber = 0x0003  <ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d>
bNumConfigurations = 0x0001

Then tested the serial port filter with only one device:

(venv3.9embedded) esptool.py --port-filter serial="ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d" chip_id
esptool.py v4.9.dev1
Serial port /dev/cuaU0
Connecting...........
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 8c:4b:14:81:0b:ec
Uploading stub...
Running stub...
Stub running...
Warning: ESP32 has no Chip ID. Reading MAC instead.
MAC: 8c:4b:14:81:0b:ec
Hard resetting via RTS pin...

Then detached ESP32-DevkitC, attached ESP32C3-Devkit first (at /dev/cuaU0), then attached ESP32-DevKitC (at /dev/cuaU1). Providing filter for ESP32 at /dev/cuaU1 esptool connected to ESP32C3 at /dev/cuaU0.
```
(venv3.9embedded) ls /dev/cuaU*
/dev/cuaU0      /dev/cuaU0.init /dev/cuaU0.lock /dev/cuaU1      /dev/cuaU1.init /dev/cuaU1.lock
```

(venv3.9embedded) esptool.py --port-filter serial="ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d" chip_id esptool.py v4.9.dev1 Serial port /dev/cuaU0 Connecting... Detecting chip type... ESP32-C3 Chip is ESP32-C3 (QFN32) (revision v0.3) Features: WiFi, BLE, Embedded Flash 4MB (XMC) Crystal is 40MHz MAC: 60:55:f9:cc:e2:b8 Uploading stub... Running stub... Stub running... Warning: ESP32-C3 has no Chip ID. Reading MAC instead. MAC: 60:55:f9:cc:e2:b8 Hard resetting with RTC WDT...


* Here is the usb device descriptor for ESP32C3 and we can see it has kinda funny serial:

(venv3.9embedded) usbconfig -d 0.6 dump_device_desc ugen0.6: <Espressif USB JTAG/serial debug unit> at usbus0, cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (500mA)

bLength = 0x0012 bDescriptorType = 0x0001 bcdUSB = 0x0200 bDeviceClass = 0x00ef bDeviceSubClass = 0x0002 bDeviceProtocol = 0x0001 bMaxPacketSize0 = 0x0040 idVendor = 0x303a idProduct = 0x1001 bcdDevice = 0x0101 iManufacturer = 0x0001 iProduct = 0x0002 <USB JTAG/serial debug unit> iSerialNumber = 0x0003 <60:55:F9:CC:E2:B8> bNumConfigurations = 0x0001


* I have disconnected both devkits, connected ESP32 first (/dev/cuaU0), then ESP32C3 (/dev/cuaU1), and the esptool still connects to /dev/cuaU0

(venv3.9embedded) esptool.py --port-filter serial='60:55:F9:CC:E2:B8' chip_id esptool.py v4.9.dev1 Serial port /dev/cuaU0 Connecting......... Detecting chip type... Unsupported detection protocol, switching and trying again... Connecting..... Detecting chip type... ESP32 Chip is ESP32-D0WD-V3 (revision v3.0) Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None Crystal is 40MHz MAC: 8c:4b:14:81:0b:ec Uploading stub... Running stub... Stub running... Warning: ESP32 has no Chip ID. Reading MAC instead. MAC: 8c:4b:14:81:0b:ec Hard resetting via RTS pin...


* Okay lets try the name port filter:

(venv3.9embedded) esptool.py --port-filter name='USB JTAG/serial debug unit' chip_id esptool.py v4.9.dev1 Serial port /dev/cuaU0 Connecting............ Detecting chip type... Unsupported detection protocol, switching and trying again... Connecting.... Detecting chip type... ESP32 Chip is ESP32-D0WD-V3 (revision v3.0) Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None Crystal is 40MHz MAC: 8c:4b:14:81:0b:ec Uploading stub... Running stub... Stub running... Warning: ESP32 has no Chip ID. Reading MAC instead. MAC: 8c:4b:14:81:0b:ec Hard resetting via RTS pin...


* The same trick after swap to ESP32 to /dev/cuaU1:

(venv3.9embedded) esptool.py --port-filter name=ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d chip_id esptool.py v4.9.dev1 Serial port /dev/cuaU0 Connecting... Detecting chip type... ESP32-C3 Chip is ESP32-C3 (QFN32) (revision v0.3) Features: WiFi, BLE, Embedded Flash 4MB (XMC) Crystal is 40MHz MAC: 60:55:f9:cc:e2:b8 Uploading stub... Running stub... Stub running... Warning: ESP32-C3 has no Chip ID. Reading MAC instead. MAC: 60:55:f9:cc:e2:b8 Hard resetting with RTC WDT...

(venv3.9embedded) esptool.py --port-filter name='ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d' chip_id esptool.py v4.9.dev1 Serial port /dev/cuaU0 Connecting... Detecting chip type... ESP32-C3 Chip is ESP32-C3 (QFN32) (revision v0.3) Features: WiFi, BLE, Embedded Flash 4MB (XMC) Crystal is 40MHz MAC: 60:55:f9:cc:e2:b8 Uploading stub... Running stub... Stub running... Warning: ESP32-C3 has no Chip ID. Reading MAC instead. MAC: 60:55:f9:cc:e2:b8 Hard resetting with RTC WDT...

(venv3.9embedded) esptool.py --port-filter name="ESP32DEVKITC-CeDeROM-76f24aad88225fedf7260bb4f54c525d" chip_id esptool.py v4.9.dev1 Serial port /dev/cuaU0 Connecting... Detecting chip type... ESP32-C3 Chip is ESP32-C3 (QFN32) (revision v0.3) Features: WiFi, BLE, Embedded Flash 4MB (XMC) Crystal is 40MHz MAC: 60:55:f9:cc:e2:b8 Uploading stub... Running stub... Stub running... Warning: ESP32-C3 has no Chip ID. Reading MAC instead. MAC: 60:55:f9:cc:e2:b8 Hard resetting with RTC WDT...


* Just to verify how the pyserial see the ports, maybe this is the problem? :-)

(venv3.9embedded) pip show pyserial Name: pyserial Version: 3.5 Summary: Python Serial Port Extension Home-page: https://github.com/pyserial/pyserial Author: Chris Liechti Author-email: cliechti@gmx.net License: BSD Location: /usr/home/cederom/.local/venv3.9embedded/lib/python3.9/site-packages Requires: Required-by: adafruit-ampy, bl60x-flash, esptool, mpremote, nrfutil, pyspinel, yoctools

(venv3.9embedded) pyserial-ports --verbose /dev/cuaU0 desc: n/a hwid: n/a /dev/cuaU1 desc: n/a hwid: n/a /dev/cuau0 desc: n/a hwid: n/a 3 ports found

cederom commented 1 week ago

If the port filter depends on pyserial I saw recenlty an update that extends the pyserial-ports verbose output. If you can provide your output or what is expected we will update pyserial so it works on FreeBSD too and provide necessary information for esptool port filter :-)

Dzarda7 commented 1 week ago

Thanks for noticing this and great report. I tried it on my system and it works great as you can see below:

(venv) ➜  esptool_test_freebsd pyserial-ports --verbose
/dev/ttyUSB0        
    desc: CP2102N USB to UART Bridge Controller
    hwid: USB VID:PID=10C4:EA60 SER=7c98d1065267ee11bcc4c8ab93cd958c LOCATION=3-6.1.7

(venv) ➜  esptool_test_freebsd esptool.py --port-filter serial=7c98d1065267ee11bcc4c8ab93cd958c flash_id
esptool.py v4.9.dev1
Found 1 serial ports
Serial port /dev/ttyUSB0
Connecting.....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 10:06:1c:98:2e:e0
Uploading stub...
Running stub...
Stub running...
Manufacturer: c8
Device: 4017
Detected flash size: 8MB
Flash voltage set by a strapping pin to 3.3V
Hard resetting via RTS pin...

(venv) ➜  esptool_test_freebsd esptool.py --port-filter serial=7c98d1065267ee11bcc4c8ab93cd958d flash_id
esptool.py v4.9.dev1
Found 0 serial ports

A fatal error occurred: Could not connect to an Espressif device on any of the 0 available serial ports.

I also tried it in docker with your version of python (3.9) using docker run --privileged --rm -it --device=/dev/ttyUSB0 python:3.9 bash which also works as expected:

root@e8ff0c799bc6:/# pyserial-ports --verbose
/dev/ttyUSB0        
    desc: CP2102N USB to UART Bridge Controller
    hwid: USB VID:PID=10C4:EA60 SER=7c98d1065267ee11bcc4c8ab93cd958c LOCATION=3-6.1.7

root@e8ff0c799bc6:/# esptool.py --port-filter serial=7c98d1065267ee11bcc4c8ab93cd958c flash_id
esptool.py v4.9.dev1
Found 1 serial ports
Serial port /dev/ttyUSB0
Connecting.....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 10:06:1c:98:2e:e0
Uploading stub...
Running stub...
Stub running...
Manufacturer: c8
Device: 4017
Detected flash size: 8MB
Flash voltage set by a strapping pin to 3.3V
Hard resetting via RTS pin...

root@e8ff0c799bc6:/# esptool.py --port-filter serial=7c98d1065267ee11bcc4c8ab93cd958d flash_id
esptool.py v4.9.dev1
Found 0 serial ports

A fatal error occurred: Could not connect to an Espressif device on any of the 0 available serial ports.

Yes, port filter depends on pyserial. As you already noticed, problem seems to be the pyserial that does not see port information. I am not really sure, what you mean by updating pyserial as version 3.5 is required by esptool:

esptool_test_freebsd pip show pyserial            
Name: pyserial
Version: 3.5
Summary: Python Serial Port Extension
Home-page: https://github.com/pyserial/pyserial
Author: Chris Liechti
Author-email: cliechti@gmx.net
License: BSD
Location: /home/jarda/Documents/Espressif/esptool_test_freebsd/venv/lib64/python3.13/site-packages
Requires: 
Required-by: esptool

cederom commented 1 week ago

Thank you @Dzarda7 :-)

Can you please double check on your Linux with two different devkits attached in different order and use port filter to one connected to /dev/ttyUSB1?

If that works for you on Linux that means we need to patch pyserial to work properly on FreeBSD :-)

Dzarda7 commented 1 week ago

ESP32H2 - /dev/ttyUSB0, ESP32 - /dev/ttyUSB1:

(venv) ➜  esptool_test_freebsd pyserial-ports --verbose
/dev/ttyUSB0        
    desc: CP2102N USB to UART Bridge Controller
    hwid: USB VID:PID=10C4:EA60 SER=14fefc9fe665ee119c09926293cd958c LOCATION=3-6.1.5
/dev/ttyUSB1        
    desc: CP2102N USB to UART Bridge Controller
    hwid: USB VID:PID=10C4:EA60 SER=7c98d1065267ee11bcc4c8ab93cd958c LOCATION=3-6.1.7

(venv) ➜  esptool_test_freebsd esptool.py --port-filter serial=14fefc9fe665ee119c09926293cd958c flash_id
esptool.py v4.9.dev1
Found 1 serial ports
Serial port /dev/ttyUSB0
Connecting.....
Detecting chip type... ESP32-H2
Chip is ESP32-H2 (revision v0.1)
Features: BLE, IEEE802.15.4
Crystal is 32MHz
MAC: 74:4d:bd:ff:fe:63:37:06
BASE MAC: 74:4d:bd:63:37:06
MAC_EXT: ff:fe
Uploading stub...
Running stub...
Stub running...
Manufacturer: c8
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

(venv) ➜  esptool_test_freebsd esptool.py --port-filter serial=7c98d1065267ee11bcc4c8ab93cd958c flash_id
esptool.py v4.9.dev1
Found 1 serial ports
Serial port /dev/ttyUSB1
Connecting.......
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting.....
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 10:06:1c:98:2e:e0
Uploading stub...
Running stub...
Stub running...
Manufacturer: c8
Device: 4017
Detected flash size: 8MB
Flash voltage set by a strapping pin to 3.3V
Hard resetting via RTS pin...

ESP32H2 - /dev/ttyUSB0, ESP32 - /dev/ttyUSB1:

(venv) ➜  esptool_test_freebsd pyserial-ports --verbose
/dev/ttyUSB0        
    desc: CP2102N USB to UART Bridge Controller
    hwid: USB VID:PID=10C4:EA60 SER=7c98d1065267ee11bcc4c8ab93cd958c LOCATION=3-6.1.7
/dev/ttyUSB1        
    desc: CP2102N USB to UART Bridge Controller
    hwid: USB VID:PID=10C4:EA60 SER=14fefc9fe665ee119c09926293cd958c LOCATION=3-6.1.5

(venv) ➜  esptool_test_freebsd esptool.py --port-filter serial=14fefc9fe665ee119c09926293cd958c flash_id
esptool.py v4.9.dev1
Found 1 serial ports
Serial port /dev/ttyUSB1
Connecting....
Detecting chip type... ESP32-H2
Chip is ESP32-H2 (revision v0.1)
Features: BLE, IEEE802.15.4
Crystal is 32MHz
MAC: 74:4d:bd:ff:fe:63:37:06
BASE MAC: 74:4d:bd:63:37:06
MAC_EXT: ff:fe
Uploading stub...
Running stub...
Stub running...
Manufacturer: c8
Device: 4016
Detected flash size: 4MB
Hard resetting via RTS pin...

(venv) ➜  esptool_test_freebsd esptool.py --port-filter serial=7c98d1065267ee11bcc4c8ab93cd958c flash_id
esptool.py v4.9.dev1
Found 1 serial ports
Serial port /dev/ttyUSB0
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting......
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 10:06:1c:98:2e:e0
Uploading stub...
Running stub...
Stub running...
Manufacturer: c8
Device: 4017
Detected flash size: 8MB
Flash voltage set by a strapping pin to 3.3V
Hard resetting via RTS pin...

Hope this helps, you can see that it works for me without the problem. Please beware that ends of both serial numbers are the same.

cederom commented 1 week ago

Thank you @Dzarda7 looks like a problem on our end will try to align pyserial on FreeBSD side :-)

espressif / esptool