search

LINBIT / linstor-server

High Performance Software-Defined Block Storage for container, cloud and virtualisation. Fully integrated with Docker, Kubernetes, Openstack, Proxmox etc.
https://docs.linbit.com/docs/linstor-guide/
GNU General Public License v3.0
984 stars 76 forks source link

Remote satellite peer has closed the connection. Unhandled IllegalStateException #327

Open kvaps opened 2 years ago

kvaps commented 2 years ago

Seems same as https://github.com/LINBIT/linstor-server/issues/288

Here we are again. Linstor-controller can't establish SSL connection with linstor-node for some reason:

Linstor-controller log:

09:51:32.377 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.118:3367 has closed the connection.
09:51:32.385 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.119:3367 has closed the connection.
09:51:39.348 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.119:3367 has closed the connection.
09:51:39.381 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.122:3367 has closed the connection.
09:51:53.340 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.121:3367 has closed the connection.
09:52:01.968 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.115:3367 has closed the connection.
09:52:02.325 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.115:3367 has closed the connection.
09:52:03.322 [SslConnector] INFO  LINSTOR/Controller - SYSTEM - Remote satellite peer /10.51.32.115:3367 has closed the connection.

Linstor-satellite log:

12:43:08.201 [MainWorkerPool-1] INFO  LINSTOR/Satellite - SYSTEM - Controller connected and authenticated (172.16.10.53:52480)
12:43:08.207 [MainWorkerPool-1] INFO  LINSTOR/Satellite - SYSTEM - Node 'spb99-dev-dkm02' created.
12:43:08.208 [MainWorkerPool-1] INFO  LINSTOR/Satellite - SYSTEM - Storage pool 'DfltDisklessStorPool' created.
12:43:15.196 [MainWorkerPool-1] INFO  LINSTOR/Satellite - SYSTEM - Controller connected and authenticated (172.16.10.53:52482)
12:43:15.200 [MainWorkerPool-1] INFO  LINSTOR/Satellite - SYSTEM - Node 'spb99-dev-dkm02' created.
12:43:15.201 [MainWorkerPool-1] INFO  LINSTOR/Satellite - SYSTEM - Storage pool 'DfltDisklessStorPool' created.
12:43:22.135 [SSLNetComService] ERROR LINSTOR/Satellite - SYSTEM - Unhandled IllegalStateException [Report number 637A4A60-5ADAA-000137]

12:43:29.134 [SSLNetComService] ERROR LINSTOR/Satellite - SYSTEM - Unhandled IllegalStateException [Report number 637A4A60-5ADAA-000138]

Error reports from sattelite (sorry for formating):

ERROR REPORT 637A4A60-5ADAA-000137============================================================Application:                        LINBIT? LINSTOR
Module:                             Satellite
Version:                            1.19.1
Build ID:                           a758bf07796c374fd2004465b0d8690209b74356
Build time:                         2022-07-28T04:54:55+00:00
Error time:                         2022-11-21 12:43:22
Node:                               spb99-dev-dkm02
Peer:                               172.16.10.53:52484============================================================Reported error:
===============Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method ‘run’, Source file ‘TcpConnectorService.java’, Line #734Error message:                      Unhandled IllegalStateExceptionCall backtrace:    Method                                   Native Class:Line number
    run                                      N      com.linbit.linstor.netcom.TcpConnectorService:734
    run                                      N      java.lang.Thread:829Caused by:
==========Category:                           RuntimeException
Class name:                         IllegalStateException
Class canonical name:               java.lang.IllegalStateException
Generated at:                       Method ‘doHandshake’, Source file ‘SslTcpConnectorHandshaker.java’, Line #103Error message:                      com.linbit.linstor.netcom.ssl.SslTcpConnectorService indicates requiring a handshake, but the sun.security.ssl.SSLEngineImpl instance is not in handshake modeCall backtrace:    Method                                   Native Class:Line number
    doHandshake                              N      com.linbit.linstor.netcom.ssl.SslTcpConnectorHandshaker:103
    read                                     N      com.linbit.linstor.netcom.ssl.SslTcpConnectorPeer:162
    run                                      N      com.linbit.linstor.netcom.TcpConnectorService:543
    run                                      N      java.lang.Thread:829END OF ERROR REPORT.
ERROR REPORT 637A4A60-5ADAA-000138============================================================Application:                        LINBIT? LINSTOR
Module:                             Satellite
Version:                            1.19.1
Build ID:                           a758bf07796c374fd2004465b0d8690209b74356
Build time:                         2022-07-28T04:54:55+00:00
Error time:                         2022-11-21 12:43:29
Node:                               spb99-dev-dkm02
Peer:                               172.16.10.53:52486============================================================Reported error:
===============Category:                           Error
Class name:                         ImplementationError
Class canonical name:               com.linbit.ImplementationError
Generated at:                       Method ‘run’, Source file ‘TcpConnectorService.java’, Line #734Error message:                      Unhandled IllegalStateExceptionCall backtrace:    Method                                   Native Class:Line number
    run                                      N      com.linbit.linstor.netcom.TcpConnectorService:734
    run                                      N      java.lang.Thread:829Caused by:
==========Category:                           RuntimeException
Class name:                         IllegalStateException
Class canonical name:               java.lang.IllegalStateException
Generated at:                       Method ‘doHandshake’, Source file ‘SslTcpConnectorHandshaker.java’, Line #103Error message:                      com.linbit.linstor.netcom.ssl.SslTcpConnectorService indicates requiring a handshake, but the sun.security.ssl.SSLEngineImpl instance is not in handshake modeCall backtrace:    Method                                   Native Class:Line number
    doHandshake                              N      com.linbit.linstor.netcom.ssl.SslTcpConnectorHandshaker:103
    read                                     N      com.linbit.linstor.netcom.ssl.SslTcpConnectorPeer:162
    run                                      N      com.linbit.linstor.netcom.TcpConnectorService:543
    run                                      N      java.lang.Thread:829END OF ERROR REPORT.

Restart of linstor-controller solves the situation for a short a while.

beornf commented 1 year ago

Same SSL handshake exception as #161