ASN1 tag error when downloading components (IDFGH-8120)

josesimoes commented 2 years ago

Answers checklist.

[X] I have read the documentation ESP-IDF Programming Guide and the issue is not addressed there.
[X] I have updated my IDF branch (master or release) to the latest version and checked that the issue is present there.
[X] I have searched the issue tracker for a similar issue and not found a similar issue.

IDF version.

v4.4.2

Operating System used.

Windows

How did you build your project?

Command line with CMake

If you are using Windows, please specify command line type.

PowerShell

What is the expected behavior?

Being able to use install scripts

What is the actual behavior?

Error in download step

Steps to reproduce.

run install.bat or install.ps1

Build or installation Logs.

Downloading tools for win64: xtensa-esp32-elf, xtensa-esp32s2-elf, xtensa-esp32s3-elf, riscv32-esp-elf, esp32ulp-elf, esp32s2ulp-elf, cmake, openocd-esp32, ninja, idf-exe, ccache, dfu-util
Downloading xtensa-esp32-elf@esp-2021r2-patch3-8.4.0
Downloading xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
WARNING: Download failure [WinError -2146881269] ASN1 valor de tag inválido encontrado
WARNING: Failed to download https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
Downloading xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
WARNING: Download failure [WinError -2146881269] ASN1 valor de tag inválido encontrado
WARNING: Failed to download https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
Downloading xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
WARNING: Download failure [WinError -2146881269] ASN1 valor de tag inválido encontrado
WARNING: Failed to download https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
ERROR: Failed to download, and retry count has expired
Traceback (most recent call last):
  File "E:\GitHub\esp-idf\tools\idf_tools.py", line 1860, in <module>
    main(sys.argv[1:])
  File "E:\GitHub\esp-idf\tools\idf_tools.py", line 1856, in main
    action_func(args)
  File "E:\GitHub\esp-idf\tools\idf_tools.py", line 1413, in action_download
    tool_obj.download(tool_version)
  File "E:\GitHub\esp-idf\tools\idf_tools.py", line 722, in download
    raise DownloadError()
__main__.DownloadError

More Information.

The above error is in Portuguese "ASN1 valor de tag inválido encontrado" in English that should something close to " ASN1 bad tag value met"

igrr commented 2 years ago

Could you please try running the following to check if urllib is working correctly in this Python installation?

path\to\your\python3.exe -c "import urllib.request; print(len(urllib.request.urlopen('https://dl.espressif.com/').read()))"

Should print 329 if urllib.request works, or it might print the same ASN1 error.

josesimoes commented 2 years ago

I guess the above should be python.exe... named python3 I can only find a DLL.. Here's the output:

E:\GitHub\esp-idf>python -c "import urllib.request; print(len(urllib.request.urlopen('https://dl.espressif.com/').read()))"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 525, in open
    response = self._open(req, data)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 543, in _open
    '_open', req)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 503, in _call_chain
    result = func(*args)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1393, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\urllib\request.py", line 1318, in do_open
    h = http_class(host, timeout=req.timeout, **http_conn_args)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\http\client.py", line 1401, in __init__
    context = ssl._create_default_https_context()
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 589, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 490, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Users\JJSS\AppData\Local\Programs\Python\Python37\lib\ssl.py", line 474, in _load_windows_store_certs
    for cert, encoding, trust in enum_certificates(storename):
OSError: [WinError -2146881269] ASN1 valor de tag inválido encontrado

igrr commented 2 years ago

In that case I'm afraid this particular Python installation is not functioning correctly, as it fails while loading the certificates from the certificate store.

You can try to fix this by installing another copy of Python or trying to repair that one. Alternatively, please consider installing IDF using the installer: https://docs.espressif.com/projects/esp-idf/en/v4.4.2/esp32/get-started/windows-setup.html#esp-idf-tools-installer. Among other things, it comes with an "embedded" Python interpreter which significantly reduces the likelihood of installation failure due to an issue with the existing installation.

josesimoes commented 2 years ago

@igrr that was my initial go to to update to IDF 4.4.2! Thing is: the same error is happening. That's what prompted me to install Python...

Please look at the last part of the install log from running esp-idf-tools-setup-online-2.16.exe

2022-08-24 10:11:17.791   Installing tools:C:\Users\jjss\.espressif\tools\idf-python\3.8.7\python.exe "E:\GitHub\esp-idf\tools\idf_tools.py" --idf-path E:\GitHub\esp-idf\   install --targets=esp32,esp32-c3,esp32-s3,esp32-s2
2022-08-24 10:11:18.066   Done, exit code=1
2022-08-24 10:11:18.066   --------
2022-08-24 10:11:18.066   Running command: C:\Users\jjss\.espressif\tools\idf-python\3.8.7\python.exe "E:\GitHub\esp-idf\tools\idf_tools.py" --idf-path E:\GitHub\esp-idf\   install --targets=esp32,esp32-c3,esp32-s3,esp32-s2
                          Current system platform: win64
                          Selected targets are: esp32, esp32c3, esp32s3, esp32s2, esp32h2
                          Installing tools: xtensa-esp32-elf, xtensa-esp32s2-elf, xtensa-esp32s3-elf, riscv32-esp-elf, esp32ulp-elf, esp32s2ulp-elf, cmake, openocd-esp32, ninja, idf-exe, ccache, dfu-util
                          Installing xtensa-esp32-elf@esp-2021r2-patch3-8.4.0
                          Using GitHub assets mirror for URL: https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip => https://dl.espressif.com/github_assets/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip
                          Downloading xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
                          WARNING: Download failure [WinError -2146881269] ASN1 valor de tag inválido encontrado
                          WARNING: Failed to download https://dl.espressif.com/github_assets/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
                          Downloading xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
                          WARNING: Download failure [WinError -2146881269] ASN1 valor de tag inválido encontrado
                          WARNING: Failed to download https://dl.espressif.com/github_assets/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
                          Downloading xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
                          WARNING: Download failure [WinError -2146881269] ASN1 valor de tag inválido encontrado
                          WARNING: Failed to download https://dl.espressif.com/github_assets/espressif/crosstool-NG/releases/download/esp-2021r2-patch3/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip to C:\Users\jjss\.espressif\dist\xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch3-win64.zip.tmp
                          ERROR: Failed to download, and retry count has expired
                          Traceback (most recent call last):
                            File "E:\GitHub\esp-idf\tools\idf_tools.py", line 1860, in <module>
                              main(sys.argv[1:])
                            File "E:\GitHub\esp-idf\tools\idf_tools.py", line 1856, in main
                              action_func(args)
                            File "E:\GitHub\esp-idf\tools\idf_tools.py", line 1470, in action_install
                              tool_obj.download(tool_version)
                            File "E:\GitHub\esp-idf\tools\idf_tools.py", line 722, in download
                              raise DownloadError()
                          __main__.DownloadError

2022-08-24 10:11:18.066   --------
2022-08-24 10:11:18.066   Message box (OK):

igrr commented 2 years ago

@georgik PTAL at the installer log above, have you seen something like this before? Seems that the error happens for every Python installed on this system: both in the %USERPROFILE%\AppData\Local\Programs\Python\Python37 and in %USERPROFILE%.espressif\tools\idf-python\3.8.7\python.exe. Looks like _load_windows_store_certs is failing, and it seems that the embedded python is also trying to load "windows store certs"...

georgik commented 2 years ago

@igrr This looks very similar to what users were recently reporting for macOS https://github.com/esp-rs/rust-build/issues/136 Was there any recent change in certificates on dl.espressif.com?

@josesimoes Please, use Offline installer as an alternative to deploy whole toolchain https://github.com/espressif/idf-installer#espressif-ide-offline-installer

josesimoes commented 2 years ago

Looking at the cert from dl.espressif.com it seems that it was issued last month: 2022-07-20. Following the certificate chain seems that it's using a DigiCert Root CA, which is on the trust list for Windows.

@georgik I'll give it a try with the offline installer, thanks.

igrr commented 2 years ago

Was there any recent change in certificates on dl.espressif.com?

I don't think there was any recent change. The certificate was renewed, but it has the same root as previously.

And looking at the code (Python37\lib\http\client.py), the failure happens in context = ssl._create_default_https_context(), which is before the communication to the server actually starts.

josesimoes commented 2 years ago

Follow up on this: using the offline installer I was able to get the toolchains installed. At some point the installer stopped with an error which I haven't take note of (sorry).

Now after that I went to the install.bat and then I could see the toolchains being checked and it kept going. To my surprise it managed to install all the other stuff from dl.espressif.com.

Worth noting: I had uninstall the Python installation from my machine and I did one last try to use the install script before going with the offline installer. So... not sure what's going on with this...

josesimoes commented 2 years ago

Just tried to use install.ps1 (and install.bat) from v4.4.3 and I'm sorry to report that the issue remains...

WARNING: Download failure: [WinError -2146881269] ASN1 valor de tag inválido encontrado

I can't see the "help message" that I guess it was supposed to show when running on Windows... Here's the output for reference.

dobairoland commented 2 years ago

@josesimoes Because the issue is in your system and ESP-IDF cannot fix it. The best solution we could offer is to use the offline installer.

josesimoes commented 2 years ago

@dobairoland I did look for an offline installer for 4.4.3 and couldn't find one... can you please advise?

dobairoland commented 2 years ago

v4.4.3 was released just 2 hours ago. The installer will be available soon.

josesimoes commented 2 years ago

Not happy with the outcome I've made some investigations and local testing. Definitive fix in PR #10147.

djixon commented 1 year ago

The issue is related to some countries (usually in European union, but there are also other countries) as an additional prevention against attacks implemented in Python code, and is related to government certificates stored on the Windows system. This error is common on Windows platforms, for Python when there are government certificates (for reading government ID, driver license, smart cards or other sensitive data) installed on Windows.

Possible solutions: Removing government certificates, or installing whole ESP-IDF in virtual machine on blank Windows platform, or Offline installation.

The following ERROR is generated by using provided online installer, here is a setup.log:

Setup Log 2023-10-12.txt

espressif / esp-idf