Linux agent 2.144.2 config.sh thrown exception: --> Interop+Crypto+OpenSslCryptographicException: error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table

[spashx]

Agent version and OS: Agent version 2.144.2 OS: Ubuntu 16.04.1 LTS

OpenSSL version 1.0.2g 1 Mar 2016 Curl version is 7.47.0 dpkg -l | grep libcurl gives:

ii  libcurl3:amd64                              7.47.0-1ubuntu2.13                                          amd64        easy-to-use client-side URL transfer library (OpenSSL flavour)
ii  libcurl3-gnutls:amd64                       7.47.0-1ubuntu2.13                                          amd64        easy-to-use client-side URL transfer library (GnuTLS flavour)
ii  python3-pycurl                              7.43.0-1ubuntu1                                             amd64        Python bindings to libcurl (Python 3)

Server: Azure Devops Server 2019.01

Symptoms: config .sh crash after entering credentials with default auth with error:

Unhandled Exception: System.TypeInitializationException: The type initializer for 'Ssl' threw an exception. ---> System.TypeInitializationException: The type initializer for 'SslInitializer' threw an exception. ---> Interop+Crypto+OpenSslCryptographicException: error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table
   at Interop.SslInitializer..cctor()
   --- End of inner exception stack trace ---
   at Interop.Ssl..cctor()
   --- End of inner exception stack trace ---
   at Interop.Ssl.SetProtocolOptions(IntPtr ctx, SslProtocols protocols)
   at System.Net.Http.CurlHandler.SslProvider.SslCtxCallback(IntPtr curl, IntPtr sslCtx, IntPtr userPointer)
   at Interop.Http.MultiPerform(SafeCurlMultiHandle multiHandle)
   at System.Net.Http.CurlHandler.MultiAgent.PerformCurlWork()
   at System.Net.Http.CurlHandler.MultiAgent.WorkerBodyLoop()
./config.sh : ligne 86 : 49449 Abandon                 (core dumped) ./bin/Agent.Listener configure "$@"

dotnet core dependencies are up to date (config.sh is not asking for update).

with CURL_TRACE=1, VSTS_AGENT_HTTPTRACE=true defined on CLI, log in _diag says:

[2019-06-04 14:44:33Z INFO Terminal] WRITE: Entrez nom d'utilisateur >
[2019-06-04 14:44:33Z INFO Terminal] READ LINE
[2019-06-04 14:44:40Z INFO Terminal] Read value: '<DOM>\<USER>'
[2019-06-04 14:44:40Z INFO CommandSettings] Arg 'password': ''
[2019-06-04 14:44:40Z INFO CommandSettings] Flag 'unattended': 'False'
[2019-06-04 14:44:40Z INFO PromptManager] ReadValue
[2019-06-04 14:44:40Z INFO Terminal] WRITE: Entrez mot de passe >
[2019-06-04 14:44:40Z INFO Terminal] READ SECRET
[2019-06-04 14:44:47Z INFO Terminal] Read value: '***'
[2019-06-04 14:44:47Z INFO NegotiateCredential] GetVssCredentials
[2019-06-04 14:44:47Z INFO NegotiateCredential] User name retrieved.
[2019-06-04 14:44:47Z INFO NegotiateCredential] Password retrieved.
[2019-06-04 14:44:47Z INFO NegotiateCredential] URL retrieved: https://<SERVER>
[2019-06-04 14:44:47Z INFO ConfigurationManager] cred retrieved
[2019-06-04 14:44:47Z INFO VisualStudioServices] Starting operation Location.GetConnectionData
[2019-06-04 14:44:47Z INFO HttpTrace] Trace System.Net.Http.HttpRequestOut.Start event:
{ Request = Method: GET, RequestUri: 'https://<SERVER>/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1', Version: 1.1, Content: <null>, Headers:
{
  Accept: application/json
  User-Agent: VSServices/16.145.28329.0
  User-Agent: (NetStandard; Linux 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14 UTC 2019)
  User-Agent: VstsAgentCore-linux-x64/2.144.2
  User-Agent: (Linux 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14 UTC 2019)
  X-VSS-E2EID: 61a03968-ab31-4da9-83c7-2f716f5c0932
  Accept-Language: fr-FR
  X-TFS-FedAuthRedirect: Suppress
  X-TFS-Session: c3d0157e-ce52-4a81-9c4b-cb44247e163c
  Expect: 100-continue
} }
[2019-06-04 14:44:47Z INFO HttpTrace] Trace System.Net.Http.Request event:
{ Request = Method: GET, RequestUri: 'https://<SERVER>/_apis/connectionData?connectOptions=1&lastChangeId=-1&lastChangeId64=-1', Version: 1.1, Content: <null>, Headers:
{
  Accept: application/json
  User-Agent: VSServices/16.145.28329.0
  User-Agent: (NetStandard; Linux 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14 UTC 2019)
  User-Agent: VstsAgentCore-linux-x64/2.144.2
  User-Agent: (Linux 4.4.0-148-generic #174-Ubuntu SMP Tue May 7 12:20:14 UTC 2019)
  X-VSS-E2EID: 61a03968-ab31-4da9-83c7-2f716f5c0932
  Accept-Language: fr-FR
  X-TFS-FedAuthRedirect: Suppress
  X-TFS-Session: c3d0157e-ce52-4a81-9c4b-cb44247e163c
  Expect: 100-continue
}, LoggingRequestId = 1ae9957d-5a87-4fda-91e4-20b1038169d9, Timestamp = 1303786126509837 }

but nothing more.

I'm able to connect to server with openssl, so not a certificate issue. openssl s_client -connect :443

most important, I'm able to run the VSTS Agent version 2.111.1 configuration without issues, so it's seems not a OS configuration issue.

Please advice. Thank you.

[ndunn990]

We're experiencing something similar. The only recent change was that we installed a new certificate (update) on the server so it could be trusted for another tool. @spashx did you make any similar changes?

No solution or workaround yet, but I'll update if we find anything.

[spashx]

@ndunn990 : no we didn't update our cert files onto the server. Basically, we had a working 2.111.1 agent, and wanted to upgrade to 2.144.2. I assume the issue may be in the dependencies to openssl/libopenssl the 2.144.2 agent uses in it's dotnet core version compared to the 2.111.1 agent. I did not investigate more so far, the 2.111.1 still working against our Devops 2019 Server.

[tjhowse]

I am also hitting this issue attempting to configure the Linux build agent in Debian WSL. I was not able to fix the problem by falling back to 2.111.1.

[ndunn990]

So, I wound up having to uninstall and reinstall the entire certificate store. There's a chance my issue was different since mine occurred around the time I installed new ca certificates on the server. However, it might be worth a try for those unable to find another solution.

[tjhowse]

Thanks for the suggestion! I removed ca-certificates and reinstalled it but unfortunately I'm still getting the same error when configuring.

[iricigor]

I am experiencing the same with agent 2.155.1

[juliobbv]

We've gotten multiple reports of this issue happening, so I started to look at it. The common factors appear to be: an Ubuntu 16.04 agent that with a version equal or later than 2.141.0, targeting an on-prem Azure DevOps Server with HTTPS enabled.

The theory is that our move to build the agent with .NET Core 2.1 has caused this issue to surface, due to a (currently unknown) way the CA certificates are laid out on the affected machines. We currently don't have a repro, so I'll need some help from you: @tjhowse, @iricigor, @ndunn990.

@ndunn990: looks like your particular issue started to occur when you installed new CA certificates on the server. What were the commands that you used to uninstall and reinstall the certificate store on the agent machine that allowed you to go past the OpenSSL error?

@iricigor, @tjhowse: I'm curious at of how did you set up your agent environments, especially regarding what steps you took to add the certs to the store. Did you use a particular guide/tutorial to generate the CA certs? Also, if possible, can you try configuring an agent against a completely-clean Linux install, and see if that makes the issue still repro?

Thanks for the help, and sorry for the inconvenience, Julio

[iricigor]

@juliobbv You are on a right track! I did configure an agent without any issues if I first configure the agent and then run my setup on a VM. So yes, something in our setup is causing conflicts, but I do not know yet what (we have huge runbooks for it...)

Some technical details: I had the issue on Debian 9 machine. I tried three different versions of agent (can't recall atm exact ones, but I presume it was 155, 151 and 141) and the issue was happening on all three of them.

[juliobbv]

@iricigor thanks for the heads-up, let me know how the investigation goes. I'd be interested to know how the certificate store is set up on the affected machine that's causing the .NET Core networking framework to trip, so I'd look for things like duplicate certs (e.g. one in the local store, one in global), or two certs describing the same server, but with different validity periods.

Hopefully if we can find out what's weird with the machine, we can repro on a clean machine and file a bug against the .NET Core team to get this fixed. 😄

You might find this man-page helpful to find out which certs are currently installed in the certificate store..

/etc/ca-certificates.conf
A configuration file.
/etc/ssl/certs/ca-certificates.crt
A single-file version of CA certificates. This holds all CA certificates that you activated in /etc/ca-certificates.conf.
/usr/share/ca-certificates
Directory of CA certificates.
/usr/local/share/ca-certificates
Directory of local CA certificates (with .crt extension).

[ndunn990]

@juliobbv my solution was relatively simple. I removed the certificates signed by my organization and uninstalled 'ca-certificates'. I then reinstalled 'ca-certificates' and added my organization's certificates once more.

We're currently running Ubuntu 16.04 (Xenial)

So, after deleting my organization's certificates, I ran: sudo apt remove ca-certificates

And then: sudo apt install ca-certificates

Finally, I placed my organization's certificates back on the server.

[spashx]

@juliobbv: we had the same issue on another Linux machine running Debian 9, on which I was able to make it works again.

Non working state: /usr/local/share/ca-certificates/ contained a root certificate and another certificate (let's call it blah) containing a server certifcate+the root certificate. both certificates beeing self signed. /etc/ssl/certs/ contained symlink to these certificates (due to call to sudo update-ca-certificates).

Action performed: I remove the root certificate file from /usr/local/share/ca-certificates/ then removed manually the symlink in /etc/ssl//certs/ pointing to it. then called sudo update-ca-certificates wich returned:

1 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
Replacing debian:blah.pem  
done.
done.

I really don't know what update-ca-certificates means by "1 added" in this case.

Result: After that, I was able to configure the agent again.

My understanding is that the version of libcurl used in the 2.144.2 agent is not able to consider two certificates if another one is containing the first one.

Configuration details:

Build agent: vsts-agent-linux-x64-2.144.2.tar.gz

lsb_release -d
Debian GNU/Linux 9.9 (stretch)

 curl --version
curl 7.52.1 (x86_64-pc-linux-gnu) libcurl/7.52.1 OpenSSL/1.0.2s zlib/1.2.8 libidn2/0.16 libpsl/0.17.0 (+libidn2/0.16) libssh2/1.7.0 nghttp2/1.18.1 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: AsynchDNS IDN IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz TLS-SRP HTTP2 UnixSockets HTTPS-proxy PSL

[jtpetty]

closing this issue as I believe this issue was addressed. If you are still seeing problems, please open up a new issue.

[Nirav-Bhadradiya]

I see this today with version 2.166.4 , i don't think there is anything changed in terms of certificates for my org

    agent v2.166.4             |_|          (commit efdfb40)

End User License Agreements: Building sources from a TFVC repository requires accepting the Team Explorer Everywhere End User License Agreement. This step is not required for building sources from Git repositories. A copy of the Team Explorer Everywhere license agreement can be found at: /azp/agent/externals/tee/license.html Connect: Unhandled exception. System.TypeInitializationException: The type initializer for 'Ssl' threw an exception. ---> System.TypeInitializationException: The type initializer for 'SslInitializer' threw an exception. ---> Interop+Crypto+OpenSslCryptographicException: error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table at Interop.SslInitializer..cctor() --- End of inner exception stack trace --- at Interop.Ssl..cctor() --- End of inner exception stack trace --- at Interop.Ssl.SetProtocolOptions(IntPtr ctx, SslProtocols protocols) at System.Net.Http.CurlHandler.SslProvider.SslCtxCallback(IntPtr curl, IntPtr sslCtx, IntPtr userPointer) at Interop.Http.MultiPerform(SafeCurlMultiHandle multiHandle) at System.Net.Http.CurlHandler.MultiAgent.PerformCurlWork() at System.Net.Http.CurlHandler.MultiAgent.WorkerBodyLoop() Fatal error. Internal CLR error. (0x80131506) at Interop+Http.MultiPerform(SafeCurlMultiHandle) at Interop+Http.MultiPerform(SafeCurlMultiHandle) at System.Net.Http.CurlHandler+MultiAgent.PerformCurlWork() at System.Net.Http.CurlHandler+MultiAgent.WorkerBodyLoop() at System.Net.Http.CurlHandler+MultiAgent.WorkerBody() at System.Net.Http.CurlHandler+MultiAgent+<>c.b20_0(System.Object) at System.Threading.Tasks.Task.InnerInvoke() at System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread) at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread) at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b10_0(System.Object) at System.Threading.ThreadHelper.ThreadStart_Context(System.Object) at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object) at System.Threading.ThreadHelper.ThreadStart(System.Object) ./config.sh: line 86: 207 Aborted (core dumped) ./bin/Agent.Listener configure "$@"

[raghureddy45]

Facing the same issue.

Platform: ubuntu 16 Agent version: v2.166.1

A copy of the Team Explorer Everywhere license agreement can be found at:
  /var/adoagent/agent01/externals/tee/license.html

>> Connect:
STDERR: Unhandled exception. System.TypeInitializationException: The type initializer for 'Ssl' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'SslInitializer' threw an exception.
 ---> Interop+Crypto+OpenSslCryptographicException: error:0B07C065:x509 certificate routines:X509_STORE_add_cert:cert already in hash table
   at Interop.SslInitializer..cctor()
   --- End of inner exception stack trace ---
   at Interop.Ssl..cctor()
   --- End of inner exception stack trace ---
   at Interop.Ssl.SetProtocolOptions(IntPtr ctx, SslProtocols protocols)
   at System.Net.Http.CurlHandler.SslProvider.SslCtxCallback(IntPtr curl, IntPtr sslCtx, IntPtr userPointer)
   at Interop.Http.MultiPerform(SafeCurlMultiHandle multiHandle)
   at System.Net.Http.CurlHandler.MultiAgent.PerformCurlWork()
   at System.Net.Http.CurlHandler.MultiAgent.WorkerBodyLoop()
Fatal error. Internal CLR error. (0x80131506)
   at Interop+Http.MultiPerform(SafeCurlMultiHandle)
   at Interop+Http.MultiPerform(SafeCurlMultiHandle)
   at System.Net.Http.CurlHandler+MultiAgent.PerformCurlWork()
   at System.Net.Http.CurlHandler+MultiAgent.WorkerBodyLoop()
   at System.Net.Http.CurlHandler+MultiAgent.WorkerBody()
   at System.Net.Http.CurlHandler+MultiAgent+<>c.<EnsureWorkerIsRunning>b__20_0(System.Object)
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task+<>c.<.cctor>b__274_0(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
   at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
   at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
   at System.Threading.ThreadHelper.ThreadStart_Context(System.Object)
   at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
   at System.Threading.ThreadHelper.ThreadStart(System.Object)
Aborted (core dumped)