"MUPCA Root" Certificates - SSLError when windows cert store contains bad certificates

[MDM-79]

Because I am pretty much a noob, better to direct you to other threads I have opened, after being redirected regarding the issue numerous times to finally here (for now!). First here (altho it all started with MPC-HC), then here, and finally here where I was directed to this place. If any additional files, tests, commands from me can help, please instruct me. Thanks'

[t8m]

Unfortunately it is quite unclear what you're asking about. Could you please be more specific? I do not see anything obvious OpenSSL related on the links above without trying to read the whole issues.

[MDM-79]

Yes I know sorry, I am IT but not a programmer, but here is the major reason:

Could you please open an OpenSSL bug on the projects bug tracker https://github.com/openssl/openssl/ and explain the issue there? They might be able to implement a workaround for the broken certificates or advise you how to handle the invalid certificates.

A workaround for Python would require a major rewrite of the Windows CA store integration. We don't have any capacity to work on that area. Even if we had capacity, a workaround would land in Python 3.11 earliest (October 2022).

Of the error I get using yt-dlp (but mainly a Python issue):

C:\Windows\system32>py
Python 3.9.7 (tags/v3.9.7:1016ef3, Aug 30 2021, 20:19:38) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import ssl
>>> len(ssl.enum_certificates('ROOT'))
68
>>> len(ssl.enum_certificates('CA'))
39
>>> ssl.create_default_context()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\ssl.py", line 750, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 574, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 566, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: not enough data: cadata does not contain a certificate (_ssl.c:4159)
>>> exit()

C:\Windows\system32>youtube-dl -v
[debug] System config: []
[debug] User config: []
[debug] Custom config: []
[debug] Command-line args: ['-v']
Traceback (most recent call last):
  File "C:\Program Files\Python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "C:\Program Files\Python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\Program Files\Python39\Scripts\youtube-dl.exe\__main__.py", line 7, in <module>
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\__init__.py", line 475, in main
    _real_main(argv)
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\__init__.py", line 442, in _real_main
    with YoutubeDL(ydl_opts) as ydl:
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\YoutubeDL.py", line 422, in __init__
    self._setup_opener()
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\YoutubeDL.py", line 2395, in _setup_opener
    https_handler = make_HTTPS_handler(self.params, debuglevel=debuglevel)
  File "C:\Program Files\Python39\lib\site-packages\youtube_dl\utils.py", line 2297, in make_HTTPS_handler
    context = ssl.create_default_context(ssl.Purpose.SERVER_AUTH)
  File "C:\Program Files\Python39\lib\ssl.py", line 750, in create_default_context
    context.load_default_certs(purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 574, in load_default_certs
    self._load_windows_store_certs(storename, purpose)
  File "C:\Program Files\Python39\lib\ssl.py", line 566, in _load_windows_store_certs
    self.load_verify_locations(cadata=certs)
ssl.SSLError: [ASN1] nested asn1 error (_ssl.c:4174)

Because my Government ID card "MUPCA Root 'xy'" certificates are erroneous (not complete - bad serial number...) and they wont (probably) get fixed ever but are also necessary and indispensable, Python applications have issues and just fail. So the Python guy (Christian Heimes) thinks you should be able to solve it somehow, alto the yt-dlp guy (pukkandan) thinks Python should solve it (bypass is) as:

A workaround for Python would require a major rewrite of the Windows CA store integration. We don't have any capacity to work on that area

In theory, the issue can be worked around by simply loading each certificate separately. See https://github.com/yt-dlp/yt-dlp/pull/1118/commits/599ca418ac75ab1c0baf97f184f32ac48aa759ed

So, can/should you or the Python guys solve it, I do not know (I feel like being led in circles)... 🙄 But I hope you do! Help! 😄

P.S. yt-dlp pukkandan made a workaround, but it just works for his application, not Python in general. And I can only guess more issues are going to follow in other apps and maybe elsewhere too, because of the raise of the use of those ID card certificates (signature, bank, tax, healthcare etc.) in Serbia...

[t8m]

Unfortunately I am afraid we are not going to help you. Accepting (ignoring?) broken CA certificates in the trust store would not be a bug fix but a feature which could in theory appear in the next minor release (3.1), however I do not think it would be a good idea to implement such feature.

But maybe I still do not understand the issue fully.

[MDM-79]

But maybe I still do not understand the issue fully.

That is quite possible, so the reason why I first gave all those links, is by being afraid of my lack of knowledge and thus lack of clearness and finally badly explaining the issue...

I do not think the issue is yours (it's my government's...), but that you just might help..? Like you said "ccepting (ignoring?) broken CA certificates"

The thing being, that those certificates are absolutely essential on a national level, are still fully valid; and if something isn't right with them, the chance of it being fixed (government/police side) are close to null... 🙄

The ones in questions are:

Some are valid, others not, but even if I delete some they get put back as soon as I insert my ID or lunch the ID reader app...

[kaduk]

I skimmed through many of the linked threads/reports. The issue seems to be that when python goes to establish a TLS connection, it ends up trying to load or process in some way all certificates from the OS root store, in order to add them to the trust store. If there is even one malformed certificate in the trust store, the whole operation fails, as opposed to just silently ignoring the malformed certificate(s) (i.e., not adding it to the trust store). So, the request seems to be to cause the "load all CA certificates" operation to be "load all valid CA certificates, ignoring any malformed ones".

I was not, however, able to determine where the iteration over CAs in the OS root store occurs -- libssl or python. That seems like the proper location to attempt to address the issue.

[MDM-79]

So, the request seems to be to cause the "load all CA certificates" operation to be "load all valid CA certificates, ignoring any malformed ones".

Yes that's it! (sorry for not being able to explain it myself, but that is the issue).

I was not, however, able to determine where the iteration over CAs in the OS root store occurs -- libssl or python. That seems like the proper location to attempt to address the issue.

What can I do about it test/report.. where? Advice please. 😃

[t8m]

I was not, however, able to determine where the iteration over CAs in the OS root store occurs -- libssl or python. That seems like the proper location to attempt to address the issue.

Apparently Python ssl module loads all the certificates from the Windows CA cert store into the OpenSSL at once. So this is a request to ignore any incorrect certificates when loading multiple certs from a file in OpenSSL. I am frankly not sure this is something we want to do.

[beldmit]

So this is a request to ignore any incorrect certificates when loading multiple certs from a file in OpenSSL. I am frankly not sure this is something we want to do.

I think it's worth to be at least an optional behavior...

[t8m]

How would you envision setting the option?

[beldmit]

I am afraid an API change will be needed :(

[MDM-79]

Seams I put you in a bind, sorry for that, but this is a non fixable but important and irreplaceable bad certificate example; more like these might happen in the the future too... So at least make some kind of behavior that wont break any program/app while loading.

Apparently Python SSL module loads all the certificates from the Windows CA cert store into the OpenSSL at once. So this is a request to ignore any incorrect certificates when loading multiple certs from a file in OpenSSL. I am frankly not sure this is something we want to do.

Maybe just solve it like pukkandan from yt-dlp suggested, by loading one by one instead off all at once? Then it fails silently instead of just crashing - or is this happening just in python and would still break other apps? Or maybe a way of still loading them if they are mainly correct (in this case some has a weird serial number) and valid, and are in fact working and even signing documents. If possible, just guessing around...

[levitte]

The question is where the pile of certificate is being handled.

Is a file just shoved to an OpenSSL function (not just the python wrapper, but that a function in our libraries ends up being called with that file), or does the python functionality actually collect the certs one by one before sending them to an OpenSSL function?

In the first case, it's reasonable to look at the OpenSSL function the file is shoved to and see if we can add some sort of filtering (I have no idea how for the moment). In the second case, it's probably more reasonable to ask the python functionality to do a basic check of each certificate (or ignore errors returned by the OpenSSL function that receives each certificate)

[t8m]

@levitte it is the first case.

[kaduk]

Is a file just shoved to an OpenSSL function (not just the python wrapper, but that a function in our libraries ends up being called with that file), or does the python functionality actually collect the certs one by one before sending them to an OpenSSL function?

it is the first case.

Okay, so which OpenSSL function is it?

[DjordjeMandic]

Is there any progress on this? Autodesk Fusion 360 installer just won't pass internet connection test because of this

[paulidale]

I don't see the core developers getting a chance to work on this anytime soon. If someone were to create a pull request, there is every chance it would be reviewed and accepted.

[DjordjeMandic]

Temporary fix is to export all these certs and delete them from the store. Then do what you need to do and reimport them later... Thats what i did to install Fusion 360.

Serbian gov wont fix this, trust me...

[levicki]

@t8m @levitte

How would you envision setting the option?

Perhaps an environment variable could be an option instead of changing the API?

The offending code that loads whole store at once is in _load_windows_store_certs.

The most common error messages are:

SSLError(58, '[ASN1] nested asn1 error (_ssl.c:4174)')
SSLError(0, 'not enough data: cadata does not contain a certificate (_ssl.c:4159)')

[MDM-79]

If it helps, here are all the certificates in question... Sertifikatri.zip