search

EventStore / EventStore

EventStoreDB, the event-native database. Designed for Event Sourcing, Event-Driven, and Microservices architectures
https://eventstore.com
Other
5.24k stars 639 forks source link

Interop+Crypto+OpenSslCryptographicException: error:10000080:BIO routines::no such file #4186

Open knowitall12 opened 6 months ago

knowitall12 commented 6 months ago

Describe the bug We are deploying the event store db on docker. It's a single node deployment on a AWS EC2 instance. We are using the certificate file generated for EC2 instance. We are getting following error:

[ 1, 1,09:57:30.426,FTL] Host terminated unexpectedly. System.FormatException: The input is not a valid Base-64 string as it contains a non-base 64 character, more than two padding characters, or an illegal character among the padding characters.

To Reproduce Steps to reproduce the behavior:

  1. Execute the following command sudo docker run --name esdb-node-security -it -p 2113:2113 -p 1113:1113 -v /home/eventStoreDB/eventStore/logs:/var/log/eventstore -v /home/eventStoreDB/eventStore:/var/lib/eventstore -v /etc/pki/tls/:/etc/pki/tls/ -e ASPNETCORE_Kestrel__Certificates__Default__Path="/etc/pki/tls/certs/file_name.pfx" -e ASPNETCORE_Kestrel__Certificates__Default__Password="password_masked" eventstore/eventstore:latest --run-projections=All --enable-external-tcp --enable-atom-pub-over-http --certificate-reserved-node-common-name 10.241.126.84 --trusted-root-certificates-path /etc/pki/tls/certs/ --certificate-file certificate.pem --certificate-private-key-file=/etc/pki/tls/certs/key.pem

Expected behavior Docker container should have started.

Actual behavior Docker container is failing.

Config/Logs/Screenshots [ 1, 1,09:57:29.854,INF] "ES VERSION:" "23.10.1.0" ("oss-v23.10.1"/"3ce7f59f2", "2024-01-17T12:51:15+00:00") [ 1, 1,09:57:29.862,INF] "OS ARCHITECTURE:" X64 [ 1, 1,09:57:29.889,INF] "OS:" Linux ("Unix 5.10.209.198") [ 1, 1,09:57:29.893,INF] "RUNTIME:" ".NET 6.0.27/80de56dad" (64-bit) [ 1, 1,09:57:29.895,INF] "GC:" "3 GENERATIONS" "IsServerGC: False" "Latency Mode: Interactive" [ 1, 1,09:57:29.895,INF] "LOGS:" "/var/log/eventstore" [ 1, 1,09:57:29.959,INF] MODIFIED OPTIONS: CERTIFICATE OPTIONS: CERTIFICATE RESERVED NODE COMMON NAME: 10.x.x.x (Command Line) TRUSTED ROOT CERTIFICATES PATH: /etc/pki/tls/certs/ (Command Line)

CERTIFICATE OPTIONS (FROM FILE):
     CERTIFICATE FILE:                              file.pfx (Command Line)
     CERTIFICATE PASSWORD:                          ******** (Command Line)
     CERTIFICATE PRIVATE KEY FILE:                  /etc/pki/tls/certs/file.pfx (Command Line)
     CERTIFICATE PRIVATE KEY PASSWORD:              ******** (Command Line)

INTERFACE OPTIONS:
     ENABLE ATOM PUB OVER HTTP:                     true (Command Line)
     ENABLE EXTERNAL TCP:                           true (Command Line)
     NODE IP:                                       0.0.0.0 (Yaml)
     REPLICATION IP:                                0.0.0.0 (Yaml)

PROJECTION OPTIONS:
     RUN PROJECTIONS:                               All (Command Line)

DEFAULT OPTIONS: APPLICATION OPTIONS: ALLOW ANONYMOUS ENDPOINT ACCESS: False () ALLOW ANONYMOUS STREAM ACCESS: False () ALLOW UNKNOWN OPTIONS: False () CONFIG: /etc/eventstore/eventstore.conf () DISABLE HTTP CACHING: False () ENABLE HISTOGRAMS: False () HELP: False () INSECURE: False () LOG FAILED AUTHENTICATION ATTEMPTS: False () LOG HTTP REQUESTS: False () MAX APPEND SIZE: 1048576 () OVERRIDE ANONYMOUS ENDPOINT ACCESS FOR GOSSIP: True () SKIP INDEX SCAN ON READS: False () STATS PERIOD SEC: 30 () TELEMETRY OPTOUT: False () VERSION: False () WHAT IF: False () WORKER THREADS: 0 ()

AUTHENTICATION/AUTHORIZATION OPTIONS:
     AUTHENTICATION CONFIG:                          (<DEFAULT>)
     AUTHENTICATION TYPE:                           internal (<DEFAULT>)
     AUTHORIZATION CONFIG:                           (<DEFAULT>)
     AUTHORIZATION TYPE:                            internal (<DEFAULT>)
     DISABLE FIRST LEVEL HTTP AUTHORIZATION:        False (<DEFAULT>)

CERTIFICATE OPTIONS (FROM STORE):
     CERTIFICATE STORE LOCATION:                     (<DEFAULT>)
     CERTIFICATE STORE NAME:                         (<DEFAULT>)
     CERTIFICATE SUBJECT NAME:                       (<DEFAULT>)
     CERTIFICATE THUMBPRINT:                         (<DEFAULT>)
     TRUSTED ROOT CERTIFICATE STORE LOCATION:        (<DEFAULT>)
     TRUSTED ROOT CERTIFICATE STORE NAME:            (<DEFAULT>)
     TRUSTED ROOT CERTIFICATE SUBJECT NAME:          (<DEFAULT>)
     TRUSTED ROOT CERTIFICATE THUMBPRINT:            (<DEFAULT>)

CLUSTER OPTIONS:
     CLUSTER DNS:                                   fake.dns (<DEFAULT>)
     CLUSTER GOSSIP PORT:                           2113 (<DEFAULT>)
     CLUSTER SIZE:                                  1 (<DEFAULT>)
     DEAD MEMBER REMOVAL PERIOD SEC:                1800 (<DEFAULT>)
     DISCOVER VIA DNS:                              True (<DEFAULT>)
     GOSSIP ALLOWED DIFFERENCE MS:                  60000 (<DEFAULT>)
     GOSSIP INTERVAL MS:                            2000 (<DEFAULT>)
     GOSSIP SEED:                                    (<DEFAULT>)
     GOSSIP TIMEOUT MS:                             2500 (<DEFAULT>)
     LEADER ELECTION TIMEOUT MS:                    1000 (<DEFAULT>)
     NODE PRIORITY:                                 0 (<DEFAULT>)
     QUORUM SIZE:                                   1 (<DEFAULT>)
     READ ONLY REPLICA:                             False (<DEFAULT>)
     STREAM INFO CACHE CAPACITY:                    0 (<DEFAULT>)
     UNSAFE ALLOW SURPLUS NODES:                    False (<DEFAULT>)

DATABASE OPTIONS:
     ALWAYS KEEP SCAVENGED:                         False (<DEFAULT>)
     CACHED CHUNKS:                                 -1 (<DEFAULT>)
     CHUNK INITIAL READER COUNT:                    5 (<DEFAULT>)
     CHUNK SIZE:                                    268435456 (<DEFAULT>)
     CHUNKS CACHE SIZE:                             536871424 (<DEFAULT>)
     COMMIT TIMEOUT MS:                             2000 (<DEFAULT>)
     DB:                                            /var/lib/eventstore (<DEFAULT>)
     DB LOG FORMAT:                                 V2 (<DEFAULT>)
     DISABLE SCAVENGE MERGING:                      False (<DEFAULT>)
     HASH COLLISION READ LIMIT:                     100 (<DEFAULT>)
     INDEX:                                          (<DEFAULT>)
     INDEX CACHE DEPTH:                             16 (<DEFAULT>)
     INDEX CACHE SIZE:                              0 (<DEFAULT>)
     INITIALIZATION THREADS:                        1 (<DEFAULT>)
     MAX AUTO MERGE INDEX LEVEL:                    2147483647 (<DEFAULT>)
     MAX MEM TABLE SIZE:                            1000000 (<DEFAULT>)
     MAX TRUNCATION:                                268435456 (<DEFAULT>)
     MEM DB:                                        False (<DEFAULT>)
     MIN FLUSH DELAY MS:                            2 (<DEFAULT>)
     OPTIMIZE INDEX MERGE:                          False (<DEFAULT>)
     PREPARE TIMEOUT MS:                            2000 (<DEFAULT>)
     READER THREADS COUNT:                          0 (<DEFAULT>)
     REDUCE FILE CACHE PRESSURE:                    False (<DEFAULT>)
     SCAVENGE BACKEND CACHE SIZE:                   67108864 (<DEFAULT>)
     SCAVENGE BACKEND PAGE SIZE:                    16384 (<DEFAULT>)
     SCAVENGE HASH USERS CACHE CAPACITY:            100000 (<DEFAULT>)
     SCAVENGE HISTORY MAX AGE:                      30 (<DEFAULT>)
     SKIP DB VERIFY:                                False (<DEFAULT>)
     SKIP INDEX VERIFY:                             False (<DEFAULT>)
     STATS STORAGE:                                 File (<DEFAULT>)
     STREAM EXISTENCE FILTER SIZE:                  256000000 (<DEFAULT>)
     UNBUFFERED:                                    False (<DEFAULT>)
     UNSAFE DISABLE FLUSH TO DISK:                  False (<DEFAULT>)
     UNSAFE IGNORE HARD DELETE:                     False (<DEFAULT>)
     USE INDEX BLOOM FILTERS:                       True (<DEFAULT>)
     WRITE STATS TO DB:                             False (<DEFAULT>)
     WRITE THROUGH:                                 False (<DEFAULT>)
     WRITE TIMEOUT MS:                              2000 (<DEFAULT>)

DEFAULT USER OPTIONS:
     DEFAULT ADMIN PASSWORD:                        ******** (<DEFAULT>)
     DEFAULT OPS PASSWORD:                          ******** (<DEFAULT>)

DEV MODE OPTIONS:
     DEV:                                           False (<DEFAULT>)
     REMOVE DEV CERTS:                              False (<DEFAULT>)

GRPC OPTIONS:
     KEEP ALIVE INTERVAL:                           10000 (<DEFAULT>)
     KEEP ALIVE TIMEOUT:                            10000 (<DEFAULT>)

INTERFACE OPTIONS:
     ADVERTISE HOST TO CLIENT AS:                    (<DEFAULT>)
     ADVERTISE HTTP PORT TO CLIENT AS:              0 (<DEFAULT>)
     ADVERTISE NODE PORT TO CLIENT AS:              0 (<DEFAULT>)
     ADVERTISE TCP PORT TO CLIENT AS:               0 (<DEFAULT>)
     CONNECTION PENDING SEND BYTES THRESHOLD:       10485760 (<DEFAULT>)
     CONNECTION QUEUE SIZE THRESHOLD:               50000 (<DEFAULT>)
     DISABLE ADMIN UI:                              False (<DEFAULT>)
     DISABLE EXTERNAL TCP TLS:                      False (<DEFAULT>)
     DISABLE GOSSIP ON HTTP:                        False (<DEFAULT>)
     DISABLE INTERNAL TCP TLS:                      False (<DEFAULT>)
     DISABLE STATS ON HTTP:                         False (<DEFAULT>)
     ENABLE TRUSTED AUTH:                           False (<DEFAULT>)
     ENABLE UNIX SOCKET:                            False (<DEFAULT>)
     EXT HOST ADVERTISE AS:                          (<DEFAULT>)
     EXT IP:                                        127.0.0.1 (<DEFAULT>)
     EXT TCP HEARTBEAT INTERVAL:                    2000 (<DEFAULT>)
     EXT TCP HEARTBEAT TIMEOUT:                     1000 (<DEFAULT>)
     EXT TCP PORT:                                  1113 (<DEFAULT>)
     EXT TCP PORT ADVERTISE AS:                     0 (<DEFAULT>)
     GOSSIP ON SINGLE NODE:                          (<DEFAULT>)
     HTTP PORT:                                     2113 (<DEFAULT>)
     HTTP PORT ADVERTISE AS:                        0 (<DEFAULT>)
     INT HOST ADVERTISE AS:                          (<DEFAULT>)
     INT IP:                                        127.0.0.1 (<DEFAULT>)
     INT TCP HEARTBEAT INTERVAL:                    700 (<DEFAULT>)
     INT TCP HEARTBEAT TIMEOUT:                     700 (<DEFAULT>)
     INT TCP PORT:                                  1112 (<DEFAULT>)
     INT TCP PORT ADVERTISE AS:                     0 (<DEFAULT>)
     NODE HEARTBEAT INTERVAL:                       2000 (<DEFAULT>)
     NODE HEARTBEAT TIMEOUT:                        1000 (<DEFAULT>)
     NODE HOST ADVERTISE AS:                         (<DEFAULT>)
     NODE PORT:                                     2113 (<DEFAULT>)
     NODE PORT ADVERTISE AS:                        0 (<DEFAULT>)
     NODE TCP PORT:                                 1113 (<DEFAULT>)
     NODE TCP PORT ADVERTISE AS:                    0 (<DEFAULT>)
     REPLICATION HEARTBEAT INTERVAL:                700 (<DEFAULT>)
     REPLICATION HEARTBEAT TIMEOUT:                 700 (<DEFAULT>)
     REPLICATION HOST ADVERTISE AS:                  (<DEFAULT>)
     REPLICATION PORT:                              1112 (<DEFAULT>)
     REPLICATION TCP PORT ADVERTISE AS:             0 (<DEFAULT>)

LOGGING OPTIONS:
     DISABLE LOG FILE:                              False (<DEFAULT>)
     LOG:                                           /var/log/eventstore (<DEFAULT>)
     LOG CONFIG:                                    logconfig.json (<DEFAULT>)
     LOG CONSOLE FORMAT:                            Plain (<DEFAULT>)
     LOG FILE INTERVAL:                             Day (<DEFAULT>)
     LOG FILE RETENTION COUNT:                      31 (<DEFAULT>)
     LOG FILE SIZE:                                 1073741824 (<DEFAULT>)
     LOG LEVEL:                                     Default (<DEFAULT>)

PROJECTION OPTIONS:
     FAULT OUT OF ORDER PROJECTIONS:                False (<DEFAULT>)
     PROJECTION COMPILATION TIMEOUT:                500 (<DEFAULT>)
     PROJECTION EXECUTION TIMEOUT:                  250 (<DEFAULT>)
     PROJECTION THREADS:                            3 (<DEFAULT>)
     PROJECTIONS QUERY EXPIRY:                      5 (<DEFAULT>)
     START STANDARD PROJECTIONS:                    False (<DEFAULT>)

[ 1, 1,09:57:29.964,WRN] DEPRECATED The Legacy TCP Client Interface has been deprecated as of version 20.6.0. It is recommended to use gRPC instead. AtomPub over HTTP Interface has been deprecated as of version 20.6.0. It is recommended to use gRPC instead 1, 1,11:39:10.486,INF] Cannot find plugins path: "/usr/share/eventstore/plugins" [ 1, 1,11:39:10.780,DBG] MessageHierarchy initialization took 00:00:00.2687165. [ 1, 1,11:39:10.888,INF] Loading the node's certificate(s) from file: "certificate.pem" [ 1, 1,11:39:10.978,FTL] Host terminated unexpectedly. Interop+Crypto+OpenSslCryptographicException: error:10000080:BIO routines::no such file at Interop.Crypto.CheckValidOpenSslHandle(SafeHandle handle) at Internal.Cryptography.Pal.OpenSslX509CertificateReader.FromFile(String fileName, SafePasswordHandle password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(String fileName, String password, X509KeyStorageFlags keyStorageFlags) at System.Security.Cryptography.X509Certificates.X509Certificate2..ctor(String fileName, String password) at EventStore.Core.CertificateUtils.LoadFromFile(String certificatePath, String privateKeyPath, String certificatePassword, String certificatePrivateKeyPassword) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/Certificates/CertificateUtils.cs:line 148 at EventStore.Core.ClusterVNodeOptionsExtensions.LoadNodeCertificate(ClusterVNodeOptions options) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/ClusterVNodeOptionsExtensions.cs:line 239 at EventStore.Core.Certificates.OptionsCertificateProvider.LoadCertificates(ClusterVNodeOptions options) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/Certificates/OptionsCertificateProvider.cs:line 17 at EventStore.Core.ClusterVNode1.ReloadCertificates(ClusterVNodeOptions options) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/ClusterVNode.cs:line 1862 at EventStore.Core.ClusterVNode1..ctor(ClusterVNodeOptions options, ILogFormatAbstractorFactory1 logFormatAbstractorFactory, AuthenticationProviderFactory authenticationProviderFactory, AuthorizationProviderFactory authorizationProviderFactory, IReadOnlyList1 additionalPersistentSubscriptionConsumerStrategyFactories, CertificateProvider certificateProvider, MetricsConfiguration metricsConfiguration, IExpiryStrategy expiryStrategy, Nullable1 instanceId, Int32 debugIndex) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/ClusterVNode.cs:line 249 at EventStore.Core.ClusterVNode.Create[TStreamId](ClusterVNodeOptions options, ILogFormatAbstractorFactory1 logFormatAbstractorFactory, AuthenticationProviderFactory authenticationProviderFactory, AuthorizationProviderFactory authorizationProviderFactory, IReadOnlyList1 factories, CertificateProvider certificateProvider, MetricsConfiguration metricsConfiguration, Nullable1 instanceId, Int32 debugIndex) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/ClusterVNode.cs:line 85 at EventStore.ClusterNode.ClusterVNodeHostedService..ctor(ClusterVNodeOptions options, CertificateProvider certificateProvider, MetricsConfiguration metricsConfiguration) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/ClusterVNodeHostedService.cs:line 105 at EventStore.ClusterNode.Program.Main(String[] args) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.ClusterNode/Program.cs:line 171

EventStore details

knowitall12 commented 6 months ago

I have tried:

  1. Adding -e ASPNETCORE_KestrelCertificatesDefault__Path="name_of_file.pfx" -e ASPNETCORE_KestrelCertificatesDefault__Password="password_here"
  2. Checking the certificate path in docker container.
hayley-jean commented 4 months ago

Hi @knowitall12, does your certificate file contain empty lines? The error looks similar to the one reported here: https://github.com/EventStore/EventStore/issues/3312

marcelvini commented 1 month ago

This happened to me as well and it started suddenly(two months working fine and then this happened) : {"@t":"2024-07-25T12:13:30.7090903+00:00","@mt":"[S{remoteEndPoint}, L{localEndPoint}]: Authentication exception on AuthenticateAsServerAsync.","@l":"Information","@i":3445664483,"@x":"System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.\n ---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.\n ---> Interop+Crypto+OpenSslCryptographicException: error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol\n --- End of inner exception stack trace ---\n at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan1 input, Byte[]& sendBuf, Int32& sendCount)\n at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)\n --- End of inner exception stack trace ---\n at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)\n at EventStore.Transport.Tcp.TcpConnectionSsl.AuthenticateAsServerAsync(Func1 serverCertificateSelector, Func1 intermediatesSelector) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Transport.Tcp/TcpConnectionSsl.cs:line 189","remoteEndPoint":"167.94.146.58:57834","localEndPoint":"10.158.15.245:1112","SourceContext":"EventStore.Transport.Tcp.TcpConnectionSsl","ProcessId":1294,"ThreadId":110}

{"@t":"2024-07-25T12:13:23.3126284+00:00","@mt":"[S{remoteEndPoint}, L{localEndPoint}]: Exception on AuthenticateAsServerAsync.","@l":"Information","@i":990159251,"@x":"System.ArgumentNullException: Value cannot be null. (Parameter 'cert')\n at System.Security.Cryptography.X509Certificates.X509Certificate..ctor(X509Certificate cert)\n at EventStore.Common.Utils.CertificateExtensions.ConvertToCertificate2(X509Certificate certificate, X509Certificate2& certificate2) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Common/Utils/CertificateExtensions.cs:line 192\n at EventStore.Core.ClusterVNode1.ValidateClientCertificate(X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors, Func1 intermediateCertsSelector, Func1 trustedRootCertsSelector) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Core/ClusterVNode.cs:line 1781\n at EventStore.Transport.Tcp.TcpConnectionSsl.ValidateClientCertificate(Object sender, X509Certificate certificate, X509Chain chain, SslPolicyErrors sslPolicyErrors) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Transport.Tcp/TcpConnectionSsl.cs:line 285\n at System.Net.Security.SecureChannel.VerifyRemoteCertificate(RemoteCertificateValidationCallback remoteCertValidationCallback, SslCertificateTrust trust, ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus)\n at System.Net.Security.SslStream.CompleteHandshake(ProtocolToken& alertToken, SslPolicyErrors& sslPolicyErrors, X509ChainStatusFlags& chainStatus)\n at System.Net.Security.SslStream.CompleteHandshake(SslAuthenticationOptions sslAuthenticationOptions)\n at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)\n at EventStore.Transport.Tcp.TcpConnectionSsl.AuthenticateAsServerAsync(Func1 serverCertificateSelector, Func1 intermediatesSelector) in /home/runner/work/TrainStation/TrainStation/build/oss-eventstore/src/EventStore.Transport.Tcp/TcpConnectionSsl.cs:line 189","remoteEndPoint":"167.94.146.58:39064","localEndPoint":"10.158.15.245:1112","SourceContext":"EventStore.Transport.Tcp.TcpConnectionSsl","ProcessId":1294,"ThreadId":183} `