search

espressif / esptool

Espressif SoC serial bootloader utility
https://docs.espressif.com/projects/esptool
GNU General Public License v2.0
5.51k stars 1.37k forks source link

MacOS Big Sur dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found (ESPTOOL-82) #540

Closed jeksys closed 3 years ago

jeksys commented 4 years ago

Full esptool.py command line as run:

esptool.py chip_id

Full output from esptool.py

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py", line 57, in <module>
    import serial.tools.list_ports as list_ports
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/serial/tools/list_ports.py", line 29, in <module>
    from serial.tools.list_ports_posix import comports
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/serial/tools/list_ports_posix.py", line 31, in <module>
    from serial.tools.list_ports_osx import comports
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/site-packages/serial/tools/list_ports_osx.py", line 32, in <module>
    kIOMasterPortDefault = ctypes.c_void_p.in_dll(iokit, "kIOMasterPortDefault")
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found

What is the expected behaviour?

Connected device info

Do you have any other information from investigating this?

I worked before the MacOS upgrade

Is there any other information you can think of which will help us reproduce this problem?

VladislavSmolyanoy commented 4 years ago

Same issue here!

projectgus commented 4 years ago

Hi @jeksys @SwiftySquirrel ,

Thanks for letting us know about this.

esptool relies on pyserial for serial port functionality, and this is where the exception is coming from. Ultimately whatever fix is required for MacOS Big Sur support will need to be applied there and then esptool will hopefully start working again.

Until then, the best we can do in esptool is apply a workaround so it's still possible to use esptool with a specific port specified on the command line, in the case that importing this utility function fails.

jeksys commented 4 years ago

@projectgus Thank you for the suggestion. I think the additional parameter will work for some cases. Though I think Arduio studio will still fail. When do you think it is possible to make a release with a fix?

I reported the issue to pyserial Though I think Arduio studio will still fail.

ehagerty commented 4 years ago

same issue here. forgive me if I'm being thick, but just putting --port /dev/tty.usbserial-xxxxxxxx flash-id for example gives the same error. did I misunderstand the suggestion of forcing the port parameter? Thanks so much, I should know better than to upgrade my primary machine!

projectgus commented 4 years ago

Yes, sorry I didn't explain clearly. The workaround is a potential one, it still involves a code change in esptool.

As a very quick hack you can try commenting this line in esptool.py:

  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py", line 57, in <module>
    import serial.tools.list_ports as list_ports

... and then always use the --port argument

...

I'm trying to get my hands on a Big Sur Preview install to do some more debugging.

xrolfex commented 4 years ago

@projectgus I can confirm that if you comment out the try: import block per your suggestion above it does work on Big Sur.

VladislavSmolyanoy commented 4 years ago

@xrolfex It doesn't for me...

XNinety9 commented 4 years ago

Same issue here when using esptool through PlatformIO: image

xrolfex commented 4 years ago

@p-vernaeckt Are you explicitly using the —port option after commenting out the four lines?

image
ehagerty commented 4 years ago

Hi, sorry for the delay, can confirm:

OS11 Beta (20A4299v) pyenv 1.2.19 pyenv-virtualenv 1.1.5 python 3.8.2 micropython esp32-idf4-20191220-v1.12.bin esp32 (huzzah32 esp32-wroom32)

commenting out lines 56-61 of esptool.py and using the explicit --port works. Thank you!

JuanEL0 commented 4 years ago

Hi, I have the same issue on my mac with BigSur.

I'm using Arduino IDE and when I try to verify my code, it shows the same message.

I am totally new in this world and don't know how to configure the port from the Arduino IDE so I can not verify and load my code into my ESP8266.

If somebody has any idea about how to do it, I would appreciate it.

jeksys commented 4 years ago

The suggested fix worked for me.

  1. Comment below lines in the file "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py"

    #try:
#    import serial.tools.list_ports as list_ports
#except ImportError:
#    print("The installed version (%s) of pyserial appears to be too old for esptool.py (Python interpreter %s). "
#          "Check the README for installation instructions." % (sys.VERSION, sys.executable))
#    raise

  2. List all available ports ls /dev/tty.*

  3. Use esptool.py with --port command esptool.py --port /dev/tty.usbserial-01EED907 --baud 921600 erase_flash

  4. Update esptool.py or env variable with a default port so esptool.py can be called without a parameter. TODO

hpwit commented 4 years ago

A have a question which is linked to that I have python 3.7.7 installed and that is my current python but esptools via Arduino keep on trying to get to 2.7 File "esptool.py", line 57, in File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module how to change that i have reinstalled Arduino and also es-tools

ehagerty commented 4 years ago

NB, in case anyone is interested, this works for the v3.0-dev version of esptool.py in case you are using that with your s2 modules...

shurshun commented 4 years ago

NB, in case anyone is interested, this works for the v3.0-dev version of esptool.py in case you are using that with your s2 modules...

I guess we just have to find v3.0-dev version of esptool.py... Could you provide a link to this version, please?

acmdorio commented 4 years ago

Yes, sorry I didn't explain clearly. The workaround is a potential one, it still involves a code change in esptool.

As a very quick hack you can try commenting this line in esptool.py:

  File "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py", line 57, in <module>
    import serial.tools.list_ports as list_ports

... and then always use the --port argument

...

I'm trying to get my hands on a Big Sur Preview install to do some more debugging.

@projectgus Thanks, it works for me on BS 11.0 Beta (20A5354i)! By the way, a big thanks for your teamwork on this tool! Maybe you should put this in esptool.py code as a test plus a warning before using the line 57, so other users barely note this issue. And brew update, then upgrade keep things working flawlessly. Just a humble idea. ;)

projectgus commented 4 years ago

Commit cb2060c is a workaround for this issue, so that esptool.py can run (without being able to detect the port). If you pull the latest master branch version of esptool.py (or follow the instructions in the README to install in "Development Mode" via GitHub) then no more modifications of the code should be required

It looks like the actual fix will come in a Python update once this issue is resolved: https://bugs.python.org/issue41100

Will keep this issue open, at least until the upstream issue is resolved.

R0dri commented 4 years ago

@projectgus Thankyou for the workaround. I can confirm esptool.py chip_id now catches the error instead of an abrupt termination of the process.

I've been trying to implement this to use arduino-cli or Arduinos own IDE with no luck. Inspecting the JSON board definition files i found arduino tries to compile the code not with python+esptool.py but with a binary stored in $arduino_packages/esp32/esptool_py/2.6.1/esptool which I guess its a compilation out of esptool.py.

I have never used esptool standalone and I don't know where to go next. Of course, I know I could wait for the actual python fix and keep using my raspberry pi for now. But I would love to learn a little more in the area. What would you recommend?

projectgus commented 4 years ago

@R0dri I'm not actually familiar with arduino-esp32 enough to give you an exact answer. However if you can get esptool.py master to run locally (by following the steps in the README for a development install) then you can probably update the Arduino platform.txt file here to specify a new path and command to run: https://github.com/espressif/arduino-esp32/blob/ed96d2a1b70ea02d925435d06b71617bd451ffec/platform.txt#L6

exrector commented 4 years ago

Hey. I tried to apply a workaround and I got an error. What advice can you give? 😭

bash-3.2$ git clone https://github.com/espressif/esptool.git xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun bash-3.2$

petykowski commented 4 years ago

Hey. I tried to apply a workaround and I got an error. What advice can you give? 😭

`bash-3.2$ git clone https://github.com/espressif/esptool.git

xcrun: error: invalid active developer path (/Library/Developer/CommandLineTools), missing xcrun at: /Library/Developer/CommandLineTools/usr/bin/xcrun

bash-3.2$ `

This message is unrelated to the fix provided by Expressif. This error indicates that you need to update to the latest Xcode developer tools build.

exrector commented 4 years ago

Xcode Thank you. But I have checked the versions and I have the latest 11.7. Big Sur beta 11.0 (20A5364e) Any variants? 😇 PS I will try to install and update additional tools and Command Line Tools

petykowski commented 4 years ago

@exrector I am running the same configuration as described in your comment. Funny enough, I was experiencing error when attempting to commit to an unrelated repo. To resolve the issue, you should download the Apple Command Line Tools via the Developer Downloads portal, or via the direct link to the latest Xcode 12 tools here.

AndreasRavn commented 4 years ago

I've been trying to implement this to use arduino-cli or Arduinos own IDE with no luck. Inspecting the JSON board definition files i found arduino tries to compile the code not with python+esptool.py but with a binary stored in $arduino_packages/esp32/esptool_py/2.6.1/esptool which I guess its a compilation out of esptool.py.

I have never used esptool standalone and I don't know where to go next. Of course, I know I could wait for the actual python fix and keep using my raspberry pi for now. But I would love to learn a little more in the area. What would you recommend?

I changed line 6 and 7 in platform.txt from

tools.esptool_py.path={runtime.platform.path}/tools/esptool tools.esptool_py.cmd=esptool

to

tools.esptool_py.path={runtime.platform.path}/tools tools.esptool_py.cmd=esptool.py

that did the trick for me, the arduino IDE is now using esptool.py instead of the compiled binary in the subfolder.

philletourneau commented 4 years ago

Commit cb2060c is a workaround for this issue, so that esptool.py can run (without being able to detect the port). If you pull the latest master branch version of esptool.py (or follow the instructions in the README to install in "Development Mode" via GitHub) then no more modifications of the code should be required

It looks like the actual fix will come in a Python update once this issue is resolved: https://bugs.python.org/issue41100

Will keep this issue open, at least until the upstream issue is resolved.

I've install the development version, but I'm still getting the same error...?

hpwit commented 3 years ago

stupid question but how to change the version of python used for the esp32 I have the 3.7 installed and everything seems to work well on a terminal but my Arduino keeps on using 2.6 version for esptools

projectgus commented 3 years ago

@philletourneau That's odd. Is the error exactly the same? Can you please check the commit of esptool.py you have? And also that the output of which esptool.py or (where esptool.py) is the development mode version and not another copy?

@hpwit This is determined by Arduino not esptool, and it depends on which OS you're using. One (hacky) option may be to edit the #!/usr/bin/env python line at the top of esptool.py to be #!/usr/bin/env python3.

hpwit commented 3 years ago

@projectgus Hello I couldn't change it but I have found the workaround 1=> switch to version 2.7 2=>install pySerial 3=>do the modification in esptools.py documented earlier 4=>change in the platform.txt

tools.esptool_py.path={runtime.platform.path}/tools
tools.esptool_py.cmd=esptool.py

Thank to all of you guys

bfaliszek commented 3 years ago

My temporary solution for macOS Big Sur(20A5395g) - ESP32 at ArduinoIDE problem: 1) copy /Users/XXXXX/Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/tools/esptool.py to /Users/XXXXX/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py 2) change in the platform.txt(/Users/XXXXX/Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/platform.txt):

tools.esptool_py.cmd=esptool >>> tools.esptool_py.cmd=esptool.py 3) save platform.txt, restart ArduinoIDE

lucasmarcelf commented 3 years ago

My temporary solution for macOS Big Sur(20A5395g) - ESP32 at ArduinoIDE problem:

  1. copy /Users/XXXXX/Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/tools/esptool.py to /Users/XXXXX/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py
  2. change in the platform.txt(/Users/XXXXX/Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/platform.txt):

tools.esptool_py.cmd=esptool >>> tools.esptool_py.cmd=esptool.py

  1. save platform.txt, restart ArduinoIDE

I have this error: fork/exec /Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py: permission denied Error compiling to the board ESP32 Dev Module

bfaliszek commented 3 years ago

@lucasmarcelf maybe try: chmod +x /Users/XXXXX/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py at Terminal

R0dri commented 3 years ago

@lucasmarcelf maybe try: chmod +x /Users/XXXXX/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py at Terminal

Yes, I had to do chmod 755 for it to work. Did not try chmod +x. I did not fully tested it, but I confirm i don't have any compilation errors. I came close to @bfaliszek solution earlier thanks to @projectgus but for some reason it was not working.

aungthawdar commented 3 years ago

A have a question which is linked to that I have python 3.7.7 installed and that is my current python but esptools via Arduino keep on trying to get to 2.7 File "esptool.py", line 57, in File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module how to change that i have reinstalled Arduino and also es-tools

Hi, I have the same issue on my mac with BigSur.

I'm using Arduino IDE and when I try to verify my code, it shows the same message.

I am totally new in this world and don't know how to configure the port from the Arduino IDE so I can not verify and load my code into my ESP8266.

If somebody has any idea about how to do it, I would appreciate it.

Hi, I have the same issue on my mac with BigSur.

I'm using Arduino IDE and when I try to verify my code, it shows the same message.

I am totally new in this world and don't know how to configure the port from the Arduino IDE so I can not verify and load my code into my ESP8266.

If somebody has any idea about how to do it, I would appreciate it.

I've tested on BigSur Beta (20A5384c) , with python 2.7 version. Read on. https://github.com/espressif/arduino-esp32/issues/4408

hawtdawg commented 3 years ago

Commit cb2060c is a workaround for this issue, so that esptool.py can run (without being able to detect the port). If you pull the latest master branch version of esptool.py (or follow the instructions in the README to install in "Development Mode" via GitHub) then no more modifications of the code should be required

It looks like the actual fix will come in a Python update once this issue is resolved: https://bugs.python.org/issue41100

Will keep this issue open, at least until the upstream issue is resolved.

I replaced my esptool.py with the commit you linked but now I get this error in Arduino when compiling: Traceback (most recent call last): File "/Users/Mate/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py", line 38, in <module> import serial ImportError: No module named serial exit status 1 Error compiling for board ESP32 Dev Module.

I did everything else, replacing the line in platform.txt, changing the permissions in terminal, etc... I even tried commenting out the line 37 and 38 in esptool.py to no avail. I'd really love some help :(

aungthawdar commented 3 years ago

Commit cb2060c is a workaround for this issue, so that esptool.py can run (without being able to detect the port). If you pull the latest master branch version of esptool.py (or follow the instructions in the README to install in "Development Mode" via GitHub) then no more modifications of the code should be required It looks like the actual fix will come in a Python update once this issue is resolved: https://bugs.python.org/issue41100 Will keep this issue open, at least until the upstream issue is resolved.

I replaced my esptool.py with the commit you linked but now I get this error in Arduino when compiling: Traceback (most recent call last): File "/Users/Mate/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py", line 38, in <module> import serial ImportError: No module named serial exit status 1 Error compiling for board ESP32 Dev Module.

I did everything else, replacing the line in platform.txt, changing the permissions in terminal, etc... I even tried commenting out the line 37 and 38 in esptool.py to no avail. I'd really love some help :(

I've tested on BigSur Beta (20A5384c) , with python 2.7 version. Read on. https://github.com/espressif/arduino-esp32/issues/4408

Elijahg commented 3 years ago

Building a new version of Python from source with brew fixes it without having to modify the esptool.py:

https://github.com/pyserial/pyserial/issues/509#issuecomment-726966076

Edit: turns out if using esptool.py in Arduino you still need to replace and edit it, but it’s a better fix than commenting out the port finding code. Details in link above.

brunohorta82 commented 3 years ago

The suggested fix worked for me.

  1. Comment below lines in the file "/Library/Frameworks/Python.framework/Versions/3.8/bin/esptool.py"
#try:
#    import serial.tools.list_ports as list_ports
#except ImportError:
#    print("The installed version (%s) of pyserial appears to be too old for esptool.py (Python interpreter %s). "
#          "Check the README for installation instructions." % (sys.VERSION, sys.executable))
#    raise
  1. List all available ports ls /dev/tty.*
  2. Use esptool.py with --port command esptool.py --port /dev/tty.usbserial-01EED907 --baud 921600 erase_flash
  3. Update esptool.py or env variable with a default port so esptool.py can be called without a parameter. TODO

WORK FOR ME ON PLATFORMIO

me-no-dev commented 3 years ago

problem is in the version of pyserial. Big Sur needs version 3.4. I have posted a recompiled binary for Mac here. Please give it a go

stephech commented 3 years ago

problem is in the version of pyserial. Big Sur needs version 3.4. I have posted a recompiled binary for Mac here. Please give it a go

This fixed the issue for me. I put this binary in my Library/Arduino15/packages/esp32/tools/esptool_py/3.0.0 directory

gretel commented 3 years ago

pip install -U --pre pyserial does make me happy on the big sur.

dobairoland commented 3 years ago

Dear all, there are at least two PRs on the pyserial site regarding BigSur issues (https://github.com/pyserial/pyserial/pull/546, https://github.com/pyserial/pyserial/pull/542). Maybe you could check those out and report the results there. They are preparing the release of 3.5. If a proper fix is possible and would it make into that release then it would be great.

problem is in the version of pyserial. Big Sur needs version 3.4. I have posted a recompiled binary for Mac here. Please give it a go

@me-no-dev Is in it by any chance anything worth submitting as PR to pyserial in top of the above two PRs?

me-no-dev commented 3 years ago

@dobairoland I did not have to change anything, only updated pyserial to 3.4 and recompiled the binary (BTW this happens on an old OS X 10.9 for compatibility) then tried it on Big Sur and all went well. TBH did not try flashing, just compiled fine.

gretel commented 3 years ago

@dobairoland mind my post

me-no-dev commented 3 years ago

@gretel just pip install pyserial did it for me as well. But that is to get the esptool.py to work on Big Sur. The binary that is the other discussion is another story :)

@dobairoland I tested upload as well. All good. So I guess 3.4 is fine if you do not list the ports.

gretel commented 3 years ago

ah sure binaries.. nevermind. regards

me-no-dev commented 3 years ago

@projectgus @dobairoland I am hitting reliably an issue that Arduino users reported. When trying to flash my Saola board (using external ESP-Prog), download is not able to start. Even though I see data back and fourth and entered Download manually, esptool still times out. Same ESP-Prog flashes another ESP32 based board no problem, and flashing S2 over CDC also works fine. Not sure how to debug this further.

ghost commented 3 years ago

These steps worked out for me running macOS Big Sur (11.0.1) and Arduino 1.8.13

  1. Install pyserial pip install -U --pre pyserial
  2. Open: /Users//Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/platform.txt
  3. Change: tools.esptool_py.cmd=esptool to: tools.esptool_py.cmd=esptool.py
  4. Copy: /Users//Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/tools/esptool.py to: /Users//Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1
  5. Open: /Users//Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py Replace first row with:

    !/usr/bin/env python3

me-no-dev commented 3 years ago

@machinegunjoeviterbo point was to just replace the binary from the zip above and not install pyserial or edit platform.txt I am glad it worked for you, but it does not serve as proof if that binary that is supposed to go into Arduino actually works.

billvoss commented 3 years ago

pip install -U --pre pyserial does make me happy on the big sur.

I'm using platformIO, I had to use pip3.8 install -U --pre pyserial to use ~/.platformio/penv/bin/pip3.8 instead of /usr/local/bin/pip. Then it worked!

SeanZhao1 commented 3 years ago

These steps worked out for me running macOS Big Sur (11.0.1) and Arduino 1.8.13

  1. Install pyserial pip install -U --pre pyserial
  2. Open: /Users//Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/platform.txt
  3. Change: tools.esptool_py.cmd=esptool to: tools.esptool_py.cmd=esptool.py
  4. Copy: /Users//Library/Arduino15/packages/esp32/hardware/esp32/1.0.4/tools/esptool.py to: /Users//Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1
  5. Open: /Users//Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py Replace first row with:

    !/usr/bin/env python3

after I did this I got another error:

fork/exec /Users/sean/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py: permission denied Error compiling for board Adafruit ESP32 Feather.

what should I do then?