Could not find a suitable TLS CA certificate bundle, invalid path

[Rutikhal]

Time to time my Azure Function code in Python getting errors. Workaround to fix it just redeploying again to Azure, it works again sometime but then again shows that error.

FUNCTIONS_EXTENSION_VERSION =2 Python 3.6 requests==2.20.0

Could not find a suitable TLS CA certificate bundle, invalid path: /home/site/wwwroot/.python_packages/lib/python3.6/site-packages/certifi/cacert.pem Traceback (most recent call last): File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azure/storage/common/storageclient.py", line 333, in _perform_request File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/azure/storage/common/_http/httpclient.py", line 92, in perform_request File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/requests/sessions.py", line 524, in request File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/requests/sessions.py", line 637, in send File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/requests/adapters.py", line 416, in send File "/home/site/wwwroot/.python_packages/lib/python3.6/site-packages/requests/adapters.py", line 228, in cert_verify OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /home/site/wwwroot/.python_packages/lib/python3.6/site-packages/certifi/cacert.pem

[dsghi]

I am experiencing certificate verification errors with a dotnet function, which may be totally unrelated. But the reason I mention it is that it started a couple of weeks ago, and we didn't change any of our code, and I can call the API using postman (which also does certificate verification) on my local computer without issue, so I find it hard to believe it's not related to something on the Microsoft end.

I have a suspicion with all this hacking crap going on, that MS is changing out certificates all over the place, and though I can't prove anything is specifically wrong, I think azure functions have an issue in this area.

I did find this from this interesting note and wonder if this is in mix somehow: https://docs.microsoft.com/en-us/answers/questions/172717/action-required-for-attested-data-tls-with-azure-i.html

[chriszuercher]

I face exactly the same issue. The function works for some days. Then, suddently the error occures and the function stays broken until I restart or redeploy the function app. We run the function in out own APP Service Plan (Linux)

Result: Failure Exception: OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /home/site/wwwroot/.python_packages/lib/site-packages/certifi/cacert.pem

Python 3.8 requests-2.25.1 certifi-2020.12.5 Function Runtime Version 3 FUNCTIONS_EXTENSION_VERSION =3

Is there any idea how to fix this issue?

[kassem-tohme]

Just ran into this error too

• App Service Plan
• Python3.7 using requests
• Function Runtime Version 3
• FUNCTIONS_EXTENSION_VERSION = ~3

Any comment from MS on this issue?

[anthonychu]

@vrdmr @Hazhzeng Do you know what the problem could be?

[michael-kubion]

Using Azure CLI on my openSUSE Leap 15.2 environment I had a similar issue while downloading an artifact.

First I run the command that was given in Azure DevOps: az artifacts universal download --organization "https://dev.azure.com/***/" --feed "***" --name "***" --version "0.0.1" --path .

It returned the message that the extension "azure-devops" was missing and if I'd like to install it. answering this with a "yes", it started to download and install but returned an error message, like: An error occurred. Pip failed with status code 2. Use --debug for more information.

Then I tried to manually add the extension, and it returned the same error: az extension add --name azure-devops An error occurred. Pip failed with status code 2. Use --debug for more information.

Using the --debug option: az extension add --name azure-devops --debug It gave me more info. Between the error lines: OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /etc/pki/tls/certs/ca-bundle.crt ... cli.azure.cli.core.azclierror: An error occurred. Pip failed with status code 2. Use --debug for more information.

The information on these pages helped me to find the solution: https://github.com/Azure/azure-cli/issues/10079 https://stackoverflow.com/questions/31448854/how-to-force-requests-use-the-certificates-on-my-ubuntu-system

So I've searched my folders /etc/pki and /etc/ssl and found the /etc/ssl/ca-bundle.pem file. After running this export: export REQUESTS_CA_BUNDLE=/etc/ssl/ca-bundle.pem

I could successfully install the extension: az extension add --name azure-devops --debug

[HoLengZai]

I'm exactly on the same case as @chriszuercher

I face exactly the same issue. The function works for some days. Then, suddently the error occures and the function stays broken until I restart or redeploy the function app with Azure DevOPS. We also run the Function through an App Service Plan Linux (P1v2)

2021-02-25T10:40:40.594656713Z: [INFO]  Microsoft.Azure.WebJobs.Host.FunctionInvocationException: Exception while executing function: Functions.CDP_Ansible_HttpTrigger
2021-02-25T10:40:40.594669713Z: [INFO]   ---> Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException: Result: Failure
2021-02-25T10:40:40.594674213Z: [INFO]  Exception: OSError: Could not find a suitable TLS CA certificate bundle, invalid path: /home/site/wwwroot/.python_packages/lib/site-packages/certifi/cacert.pem
2021-02-25T10:40:40.594677713Z: [INFO]  Stack:   File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 355, in _handle__invocation_request
2021-02-25T10:40:40.594681413Z: [INFO]      call_result = await self._loop.run_in_executor(
2021-02-25T10:40:40.595800020Z: [INFO]    File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
2021-02-25T10:40:40.595811520Z: [INFO]      result = self.fn(*self.args, **self.kwargs)
2021-02-25T10:40:40.595815120Z: [INFO]    File "/azure-functions-host/workers/python/3.9/LINUX/X64/azure_functions_worker/dispatcher.py", line 542, in __run_sync_func
2021-02-25T10:40:40.595818320Z: [INFO]      return func(**params)

Python 3.9 Function Runtime Version 3 FUNCTIONS_EXTENSION_VERSION =3 I use VNet integration because my Azure function needs to connect to some hosts on my Vnet. It works but sometime it stops to work and I get the TLS error as mentioned

"state": "Running",
"version": "3.0.15405.0",
"versionDetails": "3.0.15405 Commit hash: c696322564f1f9dc9557bfa495c0485ddf71eeef",
"platformVersion": "91.0.10.166",
"instanceId": "7ae4b1b61969893e3245155e40b9d88f0de5614ab95b598ab06e1476e55d6453",
"computerName": "pl1sdlwk0000EC",
"processUptime": 4485881,
"extensionBundle": {
  "id": "Microsoft.Azure.Functions.ExtensionBundle",
  "version": "1.4.0"
}

Is there any idea how to fix this issue? I also got the issue (sometime) with my Azure Function when the version was: 3.0.15277.0 too

[Rutikhal]

I have reply from Microsoft Support.

The Microsoft Azure Team has investigated the issue you reported on Azure Functions and Azure App Services Web Apps for Linux using Python, that resulted in errors of  “Could not find a suitable TLS CA certificate bundle, invalid path: /home/site/wwwroot/.python_packages/lib/site-packages/certifi/cacert.pem”.   This issue was found to be related to an issue within the Kudulite container for Linux Apps and Azure Functions on dedicated App Service plans.    Upon investigation, engineers discovered during  routine maintenance a recycle of the Kudulite container may result in temporary loss of the underlying mount point, and this issue can be mitigated with a restart of the application. We have identified a fix for this issue and it will be rolling out in the next month.   We are continuously taking steps to improve the Azure Web App service and our processes to ensure such incidents do not occur in the future, and in this case it includes (but is not limited to):        • Implement the fix for this issue and review the tests to check for this condition.    We apologize for any inconvenience. Regards, The Microsoft Azure Team Privacy Statement

--

[allenhumphreys]

My team has experienced this issue with Antarres version 93, I have opened a support ticket and they say I'm not the only one.

[Ownmarc]

got the exact same issue here using the python 3.7

so the fix is to restart our app every couple of days ?

[allenhumphreys]

@Ownmarc (and whoever) You should open a support ticket if possible to make sure this issue gets resolved. But yes, restarting generally seems to fix it.

[nilavghosh]

My platform version is 94.* but still seeing this issue. One behaviour that i have seen is when the functions app scales up (As per a defined rule) in a premium plan then some of the requests see the error - Could not find a suitable TLS CA certificate bundle Attached a screenshot of the platform version.

[dsunjka]

Same problem here - Linux & Python 3.7 App Service.

Every night we get a couple of these in our logs and the App gets restarted automatically shortly after that. Highly irritating and unreliable.

[wallflower762]

I am regularly encountering this error when my app scales out to 30 instances. I have an open support ticket but have not arrived at a solution yet. I also find that restart/rerun of job resolves the issue.

I am also running Python 3.8 & Linux App Service Plan on a P1V3 instance.

[HRX101]

Does anyone solve it? then?

[v-bbalaiagar]

Apologies for the delayed response, the issue was somehow lost in the trace. We will investigate this further and update you with the findings.

[v-bbalaiagar]

Hi @pragnagopa / @vrdmr , Could you please look into this issue.

[v-bbalaiagar]

Similar scenarios were fixed/ tracked earlier as mentioned here. Let us know if the mentioned issue resolves your issue. If not, could you share the app name using https://github.com/Azure/azure-functions-host/wiki/Sharing-Your-Function-App-name-privately to check the cause.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.

[mike-leuer]

I also have gotten this issue, I have reached out to AZURE Support.

[ghost]

This issue has been automatically marked as stale because it has been marked as requiring author feedback but has not had any activity for 4 days. It will be closed if no further activity occurs within 3 days of this comment.