search

CactuseSecurity / firewall-orchestrator

Network Security data repository for automation, reporting and compliance of firewall rules
https://fwo.cactus.de
Apache License 2.0
29 stars 10 forks source link

UI and MW dying because of nightly apache reload #2452

Open tpurschke opened 3 weeks ago

tpurschke commented 3 weeks ago

this happens at large customer site f1 only on prod server but has not been seen anywhere else The next reload usually (24h later) fixes the problem and everyhting works after that.

Logs

(logs from 2024-06-16 before midnight look normal)

2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  ---
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: Exception thrown:
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  WebSocketException
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: Message:
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: Stack Trace:
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: --- End of stack trace from previous location ---
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.456+02:00 orcprocust201 fworch.middleware-server[406468]:  ---
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]: Exception thrown:
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:  WebSocketException
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]: Message:
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]: Stack Trace:
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-06-17T00:00:28.458+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: --- End of stack trace from previous location ---
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:  ---
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: Exception thrown:
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:  WebSocketException
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: Message:
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-06-17T00:00:28.466+02:00 orcprocust201 fworch.middleware-server[406468]: Stack Trace:
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]: --- End of stack trace from previous location ---
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]: 2024-06-17T00:00:28+02:00 Error - Config Subscription (Config.cs in line 160), Config subscription lead to error.
2024-06-17T00:00:28.467+02:00 orcprocust201 fworch.middleware-server[406468]:  ---

-----------------------------------

2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-16T00:00:30+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  ---
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: Exception thrown:
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  WebSocketException
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: Message:
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: Stack Trace:
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-03-16T00:00:30.407+01:00 orcprocust1201 fworch-ui[2804]: --- End of stack trace from previous location ---
2024-03-16T00:00:30.408+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-16T00:00:32+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  ---
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: Exception thrown:
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  WebSocketException
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: Message:
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  The server returned status code '503' when status code '101' was expected.
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]: Stack Trace:
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:  at System.Net.WebSockets.WebSocketHandle.ConnectAsync(Uri uri, CancellationToken cancellationToken, ClientWebSocketOptions options)
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ClientWebSocket.ConnectAsyncCore(Uri uri, CancellationToken cancellationToken)
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ConnectAsync(CancellationToken token) in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 432
2024-03-16T00:00:33.003+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLWSProtocolHandler.<>c__DisplayClass7_0`1.<<CreateSubscriptionObservable>b__0>d.MoveNext() in /_/src/GraphQL.Client/Websocket/GraphQLWSProtocolHandler.cs:line 134
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-18T00:00:30+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  ---
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: Exception thrown:
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  WebSocketException
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: Message:
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  The remote party closed the WebSocket connection without completing the close handshake.
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: Stack Trace:
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:  at System.Net.WebSockets.ManagedWebSocket.ThrowIfEOFUnexpected(Boolean throwOnPrematureClosure)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.EnsureBufferContainsAsync(Int32 minimumRequiredBytes, CancellationToken cancellationToken, Boolean throwOnPrematureClosure)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource.GetResult(Int16 token)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Net.WebSockets.ManagedWebSocket.ReceiveAsyncPrivate[TResult](Memory`1 payloadBuffer, CancellationToken cancellationToken)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Runtime.CompilerServices.PoolingAsyncValueTaskMethodBuilder`1.StateMachineBox`1.System.Threading.Tasks.Sources.IValueTaskSource<TResult>.GetResult(Int16 token)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at System.Threading.Tasks.ValueTask`1.ValueTaskSourceAsTask.<>c.<.cctor>b__4_0(Object state)
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]: --- End of stack trace from previous location ---
2024-03-18T00:00:31.108+01:00 orcprocust1201 fworch-ui[2804]:    at GraphQL.Client.Http.Websocket.GraphQLHttpWebSocket.ReceiveWebsocketMessagesAsync() in /_/src/GraphQL.Client/Websocket/GraphQLHttpWebSocket.cs:line 509
2024-03-18T00:00:32.788+01:00 orcprocust1201 fworch-ui[2804]: 2024-03-18T00:00:32+01:00 Error - Config Subscription (Config.cs in line 150), Config subscription lead to error.
2024-03-18T00:00:32.788+01:00 orcprocust1201 fworch-ui[2804]:  ---
tpurschke commented 3 weeks ago

as the config subscription is also involved, might there be a connection to other issues like https://github.com/CactuseSecurity/firewall-orchestrator/issues/2404