arduino-esp32 projects fail on macOS 11 "Big Sur"

[adamwilt]

Hardware:

Board: DOIT ESP32 DEVKIT V1 Core Installation version: esp32 1.0.3 IDE name: Arduino 1.8.12, 1.8.13 Flash Frequency: N/A PSRAM enabled: N/A Upload Speed: N/A Computer OS: macOS 11.0 Beta (20A5384c) (and earlier betas, too)

Description:

Any attempt to compile an ESP32 project, even an empty one as shown below, fails with: ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found

The same projects compile and run without problems on macOS 10.5.X.

See also https://forum.arduino.cc/index.php?topic=699989.0 for another report of the same problem.

Sketch:

void setup() {
}

void loop() {
}

Debug Messages:

Traceback (most recent call last):
  File "esptool.py", line 57, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports.py", line 29, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_posix.py", line 31, in <module>
  File "/Library/Python/2.7/site-packages/PyInstaller/loader/pyimod03_importers.py", line 389, in load_module
  File "serial/tools/list_ports_osx.py", line 32, in <module>
ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found
Failed to execute script esptool
exit status 255
/Applications/Arduino.app/Contents/Java/arduino-builder returned 255
Error compiling for board DOIT ESP32 DEVKIT V1.

[lbernstone]

https://github.com/espressif/esptool/issues/540 https://github.com/pyserial/pyserial/issues/509

[huykaamaa]

me too :(

[aungthawdar]

When I compile and upload, I got the same error on my BigSir Beta version and solved with the following ways.

-esptool is dependent on pyserial (you need to install pyserial latest version) and

-Arduino's default preferences for ESPTool is binary file , it does not use the (esptool.py) source file. So need to replace default esptool binary file with esptool.py file in system.

How to replace?... :)

1. Click Arduino Menu => Preferences
2. Find [ArduinoLibraryPath]/preferences.txt at the bottom left of Preferences' Settings Tab, Click to Open ArduinoLibrary Folder.
3. Find & Open packages/esp32/hardware/esp32/1.x.x Folder and open platform.txt
4. At the line no 7. replace binary tools.esptool_py.cmd=esptool with tools.esptool_py.cmd=esptool.py and save.
5. Open packages/esp32/hardware/esp32/1.x.x/tools folder.
6. Copy esptool.py and paste to ArduinoLibraryFolder/packages/esp32/tools/esptool_py/2.x.x
7. Quit Arduino IDE and open again. have a nice coding. 👍

[robsworld]

@aungthawdar 👍

Thank you for this solution. It works great. The only think I have to add is you must change the permission of esptool.py with the terminal:

chmod +x /Users/XXXXX/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py

[lbernstone]

Does the executable work? If you open a command line and run it, do you get an error (perhaps about python2 not being installed)? We need a solution that can work on osx and big sur.

[SpoonS2K]

@aungthawdar tried your solution and got this error

Traceback (most recent call last): File "/Users/xxxxxxx/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py", line 37, in <module> import serial ImportError: No module named serial

[me-no-dev]

can you guys try the idf-release/v4.2 branch? It has updated esptool that should contain a fix

[TimoWielink]

@me-no-dev seems to be working with 4.2! Perfect Thnx!

[adamwilt]

When I tried replacing esptool with esptool.py in the platform.txt file, I got the serial ImportError: No module named serial error. So I tried the 4.2 branch, but I'm clearly missing some steps. I used the "installation instructions for Mac OS" at https://github.com/espressif/arduino-esp32/blob/idf-release/v4.2/docs/arduino-ide/mac.md, adding the --branch option: git clone --branch idf-release/v4.2 https://github.com/espressif/arduino-esp32.git esp32 --depth 1 Whether I do this on top of my existing installation, or as the initial ESP32 installation after wiping my existing library files, I still get ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found.

What am I doing wrong? What's the proper way to install the idf-release/v4.2 branch?

[Liferme]

When I compile and upload, I got the same error on my BigSir Beta version and solved with the following ways.

-esptool is dependent on pyserial (you need to install pyserial latest version) and

-Arduino's default preferences for ESPTool is binary file , it does not use the (esptool.py) source file. So need to replace default esptool binary file with esptool.py file in system.

How to replace?... :)

1. Click Arduino Menu => Preferences
2. Find [ArduinoLibraryPath]/preferences.txt at the bottom left of Preferences' Settings Tab, Click to Open ArduinoLibrary Folder.
3. Find & Open packages/esp32/hardware/esp32/1.x.x Folder and open platform.txt
4. At the line no 7. replace binary tools.esptool_py.cmd=esptool with tools.esptool_py.cmd=esptool.py and save.
5. Open packages/esp32/hardware/esp32/1.x.x/tools folder.
6. Copy esptool.py and paste to ArduinoLibraryFolder/packages/esp32/tools/esptool_py/2.x.x
7. Quit Arduino IDE and open again. have a nice coding. 👍

It really works, and we have to combine the method with the command @robsworld provided. When compiling, remember to choose "ESP32 Arduino", NOT "ESP32 Arduino(in sketch book)", and have fun coding!

[me-no-dev]

@TimoWielink thanks for the confirmation :) I have updated the version in Master, so it will go out in 1.0.5 in a few days.

[aungthawdar]

@aungthawdar tried your solution and got this error

Traceback (most recent call last): File "/Users/xxxxxxx/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py", line 37, in <module> import serial ImportError: No module named serial

@SpoonS2K You need to install pyserial package and be sure to import first

https://pyserial.readthedocs.io/en/latest/pyserial.html esptool.py used(import) serial library

Here is how to test, you've already installed serial or not.

1. Open terminal
2. type python in that terminal
3. type import serial
4. if no error, you've already installed serial, other wise you'll get "import serial ImportError: No module named serial

Have a nice code. :P

[tomtomhoward]

Thanks guys, just did the above and got it working on Big Sur for ESP32 in Arduino. I had to update to 4.2, install python serial, and modify the files.

There's also some playing about getting the CP210x driver working for Dev-Kit ESP32 variants, involving adding the Silicon Labs developed ID to kext in OSX.

[me-no-dev]

@tomtomhoward if you have updated to 4.2, then you should not have needed to change esptool. It is already a patched one.

[tomtomhoward]

I wouldn't trust my take on it, I'm working at the limits of my knowledge here to be honest. I think I might have messed up the 4.2 install as my Documents folder is on iCloud Drive.

[wick3r]

@aungthawdar tried your solution and got this error Traceback (most recent call last): File "/Users/xxxxxxx/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py", line 37, in <module> import serial ImportError: No module named serial

@SpoonS2K You need to install pyserial package and be sure to import first

https://pyserial.readthedocs.io/en/latest/pyserial.html esptool.py used(import) serial library

Here is how to test, you've already installed serial or not.

1. Open terminal
2. type python in that terminal
3. type import serial
4. if no error, you've already installed serial, other wise you'll get "import serial ImportError: No module named serial

Have a nice code. :P

Thank you so much for taking the time to explain! I did all this and got it to work. :)

[olivierbrcknr]

Thank all of you! I tried out different variants but still was not able to get it to work.

I reinstalled the Arduino IDE and deleted all my library folders to start from a clean slate. As far as I have understood, the current main branch has already the Big Sur patch included, right? But I unfortunately still get the ValueError: dlsym(RTLD_DEFAULT, kIOMasterPortDefault): symbol not found error.

After doing the recommended fix by @aungthawdar I get also the ImportError: No module named serial error which I am not able to fix through installing pyserial and serial in both, python2.7 and python3.8.

Do you have any ideas or recommendations? Would be very much appreciated!

[me-no-dev]

@olivierbrcknr did you install from git? to your [sketchbook]/hardware/espressif/esp32 folder?

[olivierbrcknr]

Yes, I followed the repo's instructions for macOS

[me-no-dev]

could you check the esptool version?

[me-no-dev]

You can now also try to install 1.0.5-rc2 through the board manager using https://raw.githubusercontent.com/espressif/arduino-esp32/gh-pages/package_esp32_dev_index.json

[olivierbrcknr]

So, I added the .json file you shared to reinstall it through the board manager. This didn't work, so I deleted and installed everything again to make sure I do not have any old file conflicting or accidentally being used, still the same error. I then figured out I had yet another python version, installed pyserial and serial there as well to test the esptool.py method again, but also no success.

I have the esp32 library version 1.0.5-rc2 installed and esptool version 3.0.0 if I see it correctly.

[me-no-dev]

hmmm that is strange. that version of esptool I read should be patched... I guess I need to figure out BigSur VM, because I can not now upgrade to it and risk messing up my work env :)

[olivierbrcknr]

No worries, I should have done the same ;) I will experiment a little more and try to figure out the issue, probably it's related to my individual settings/mess-ups. I will let you know if I solve it!

[aungthawdar]

@olivierbrcknr ,

In my opinion, whether it's latest ArduinoIDE / latest Arduino-esp32 version or not , dependency is python & pyserial. Your problem is that can't installed python serial(pyserial) package correctly for system default using python version.

I suggest to https://opensource.com/article/19/5/python-3-default-mac for right python and pip version. After set up the right

1. python ( recommended version is >3.7 , python 2.7 is already deprecated ) and
2. pip ( recommended version is pip3 )

install python -m pip install pyserial I think your ValueError:.... will be gone away soon.

Have a nice code. :P

[adamwilt]

It would be preferable to have a version of the toolchain that just worked with the Python version installed as part of the OS. As it currently stands, installing esp32 library version 1.0.5-rc2 on OS 11 fails exactly the same way as version 1.0.4 does.

Yes, one can install Homebrew, then install Python 3.7.x, then install pip3, and finally make the dotfile changes to run pyenv, but that's hardly a turnkey solution, especially by Arduino standards. And you'd only know about it if you'd followed this issue. Expect a lot of support requests if this convoluted path isn't straightened out. :-)

For me, it's just a lot less bother to develop in a 10.15 VM atop OS 11, because I'd already gotten that working a couple of weeks ago on the first try, with no false steps and no weird error messages. But that's just me...

[me-no-dev]

given that mac uses a prepackaged python into binary, it should not care for what is on your OS. I find it strange that it does. Whatever the fix is, it should not require the user to install anything else, just like it is on anything but BigSur. Arduino does not differentiate between the versions, so we can not provide two files for mac.

[rOOOObin]

@robsworld

@aungthawdar 👍

Thank you for this solution. It works great. The only think I have to add is you must change the permission of esptool.py with the terminal:

chmod +x /Users/XXXXX/Library/Arduino15/packages/esp32/tools/esptool_py/2.6.1/esptool.py

I´m not really good in using the terminal on mac. Can you tell me what to type in at XXXXX ?

Thank you!

[aungthawdar]

@rOOOObin , XXXXX means XXX^2 => porn square (Just kidding). XXXXX means current user's home directory(username) in unix system.

In your mac terminal, type chmod +x /Users/ then type the tab key. You will see the user home directory name.

Cheers

[me-no-dev]

Could you please try the esptool binary from the zip? No need to edit platform.txt

esptool.zip

[adamwilt]

I tried this after installing 1.0.5-rc2, and replacing esptool with the zipped version. I got much farther:

I had to first run esptool by right-clicking and selecting Open, then approving it (because it wasn't signed to Apple's liking!).

I then tried a compile & upload: `Arduino: 1.8.13 (Mac OS X), Board: "DOIT ESP32 DEVKIT V1, 80MHz, 921600, None"

Sketch uses 195784 bytes (14%) of program storage space. Maximum is 1310720 bytes. Global variables use 13016 bytes (3%) of dynamic memory, leaving 314664 bytes for local variables. Maximum is 327680 bytes. esptool.py v3.0-dev Serial port /dev/cu.usbserial-0001 Connecting......................................_____

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header ` During this time the ESP32 kept resetting, and when the upload finally failed, the existing code on the ESP32 ran, with proper Arduino serial monitor output, so the serial driver is clearly working as far as basic comms are concerned.

I had the same results at upload speeds of 115200. But if I press and hold the BOOT button on the ESP32 board when the IDE says "Connecting", the upload begins and the code loads and runs properly. Hooray!

I haven't had to press the BOOT button before, but I have to do that now, can live with that. ;-)

[me-no-dev]

hmmm that is not expected behaviour

[HomeSpan]

After updating to Big Sur I received the same errors reported here. Used the board manager to install 1.0.5-rc2 but when trying to compile I now receive the following error:

usage: cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file target_file
       cp [-R [-H | -L | -P]] [-fi | -n] [-apvXc] source_file ... target_directory
exit status 64
/Applications/Arduino.app/Contents/Java/arduino-builder returned 64
Error compiling for board Adafruit ESP32 Feather.

I assume this means I am missing a file somewhere and arduino-builder is trying to copy a non-existent file?

[me-no-dev]

it seems like some files are missing. Not an issue with esptool, so you might want to start clean (uninstall any Git version from your sketchbook/hardware folder)

[ivanisko]

Hi @me-no-dev, I try all things that you suggest and still have "Timed out waiting for packet header" message see below. Even press of boot button on ESP32 doesn't help.

Thanks for help Jan

`Sketch uses 957311 bytes (73%) of program storage space. Maximum is 1310720 bytes. Global variables use 36600 bytes (11%) of dynamic memory, leaving 291080 bytes for local variables. Maximum is 327680 bytes. esptool.py v3.0-dev Serial port /dev/cu.usbserial-0001 Connecting......................................_____

A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header A fatal error occurred: Failed to connect to ESP32: Timed out waiting for packet header`

[me-no-dev]

Works fine for me with FTDI. I was not able to find drivers for SiLabs that work on Big Sur

[ivanisko]

Ok, please what are the steps to get it work? Just replace esptool with your link? Thanks

[me-no-dev]

Yes, that is all I have done. 1.0.5-RC2 (though that should not matter) and replaced the binary with the one from the zip. Connected the ESP through FTDI, selected the port and hit Upload. What adapter are you using? I notice that your serial port has a different name, which means different driver.

[ivanisko]

OK, I try other ESP Dev kit and it's working normally after replace binary from zip. So problem is in ESP.

Thanks

[lrmate]

Could you please try the esptool binary from the zip? No need to edit platform.txt

esptool.zip

Thanks, this did the job for Lolin D32 Pro!

[ardu1]

Hello, I have also the problem, that I could not compile any sketch for my Wemos Lolin32 after I updated to Big Sur. I tried all things you describe but it will not works. This is the error report:

Traceback (most recent call last): File "/Applications/Arduino.app/Contents/Java/portable/packages/esp32/tools/esptool_py/2.6.1/esptool.py", line 37, in import serial ImportError: No module named serial exit status 1

Does anyone has an idea to fix it?

Thanks and best regards!

[HomeSpan]

Yes, after copying esptool.py into into esptool_py/2.6.1/ and editing platform.txt to point to esptool.py (instead of the binary esptool), I received the same error. To correct, I tried to use pip to install, but pip was not found. However, I found that my Mac did have pip3. So from the Terminal, type:

$ pip3 install pyserial

Depending on your permissions, you may need to run this as superuser as follows:

$ sudo pip3 install pyserial

This seemed to properly install pyserial into the Python3 environment. I then edited the first line in the esptool.py to use python3 instead of python (which on my Mac defaulted to python 2.7) as follows. The first line of esptool.py now reads.

!/usr/bin/env python3

This change to the the first line of esptool.py, as well the pip3 install of pyserial above did the trick for me. I am able to compile and upload to an Adafruit ESP32 board.

[ardu1]

Thanks, I tried it to do and got this answer: WARNING: The directory '/Users/xxxxx/Library/Caches/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. WARNING: The directory '/Users/xxxxx/Library/Caches/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag. Requirement already satisfied: pyserial in /Library/Python/3.8/site-packages (3.4) WARNING: You are using pip version 19.2.3, however version 20.2.4 is available. You should consider upgrading via the 'pip install --upgrade pip' command. What have I to do?

[HomeSpan]

Based on your diagnostics, it seems that pyserial is currently installed. Did you try editing esptool.py and setting the first line to point to python3 instead of python? When trying to compile do you get the same error (can't import serial)?

[ardu1]

Yes I did all the points. Yes, no matter what I try to compile, every time the same error.

[HomeSpan]

Maybe try updating pip3 itself per the diagnostic (I received the same suggestion and upgraded before installing pyserial). Maybe also try running the install with sudo -H as recommended?

[ardu1]

Ihm not very fit to use the terminal. Could you please write what commands I have to write? Thank you very much!

[HomeSpan]

Sure, from the terminal, first try:

$ sudo -H pip3 install --upgrade pip

Then, if that works:

$ pip3 install pyserial

If this second command does not work, try:

$ sudo -H pip3 install pyserial

[ardu1]

The installation was running and I have now the version 20.2.4

Defaulting to user installation because normal site-packages is not writeable Requirement already satisfied: pyserial in /Library/Python/3.8/site-packages (3.4)

Still the same error :-(

[HomeSpan]

Hmm. To make sure the correct file is being read, try this: Edit the esptool.py file and add a new second line with some nonsensical command, like GLOP or FOO. When you try to compile, this should report an error - something like GLOP not found, or unrecognizable command GLOP. If it does not, that means the build process is reading a different copy of esptool.py file (in other words, not the one you are editing).

The second thing to try is to make sure that pyserial is indeed properly loaded. From the terminal, enter python3 as follows:

$ python3

This should launch an interactive python session where you can then type at the >>> prompt:

import serial

If this results in an error, that means pyserial is not properly installed. If it does not result in any output, it suggests the library is indeed installed correctly. To exit the python, type quit().