[BUG] [v2023.7] Complete system failure - All data corrupted

[JeneralPain]

Describe the bug During a game, the jam ended and we went into a time out (I was the IPR at the time). The ePLTs were validating data and I noticed that the scoreboard went "P2 UPCOMING". On the screen you could see that all the "TIMEOUTS" (including for the entire game) were UNTYPED TIMEOUT, all the scores for the trips performed were 0 and the system was literally stuck.

An attempt to change the state of the system (ie: Start timeout, stop timeout) were unresponsive.

An attempt to restart the CRG instance yielded no response and brought us back to the same broken state.

To Reproduce It's not one I could ever even consider how to reproduce, its like the whole java platform decided to blow 4 whistles and just walk off track without a care in the world.

Expected behavior Not broken

Screenshots see attachments.

Version: v2023.7

If this issue occurred on a computer, please complete the following information:

• SERVER: Ubuntu 22.04LTS
• JAVA: openjdk version "21.0.3" 2024-04-16 OpenJDK Runtime Environment (build 21.0.3+9-Ubuntu-1ubuntu1) OpenJDK 64-Bit Server VM (build 21.0.3+9-Ubuntu-1ubuntu1, mixed mode, sharing)

Additional context It was very weird and the SBO advised "they did nothing different".

See some attachments. STATS-2024-06-09_UnnamedTeam_vs_UnnamedTeam_3.xlsx STATS-2024-06-09_UnnamedTeam_vs_UnnamedTeam_3.json

[JeneralPain]

crg.log scoreboard-50-secs-ago.json scoreboard-40-secs-ago.json scoreboard-10-secs-ago.json

[JeneralPain]

Complete folder upload here: https://drive.google.com/drive/folders/1lkkLEnKJwgzsZRN6nKOhn2SiwK2a78BF?usp=sharing

[JeneralPain]

It happened again, this time it didn't corrupt but threw this error.

Logs are backed up and I'll get them uploaded soon.

But here is the snip:

Exception handling update for Game(7316bada-5d4b-46d1-ba77-7830a28fa2cb).Timeout() - true: java.lang.IllegalArgument> java.lang.IllegalArgumentException: Timeout is not a property of com.carolinarollergirls.scoreboard.core.game.GameIm> at com.carolinarollergirls.scoreboard.event.ScoreBoardEventProviderImpl.checkProperty(ScoreBoardEventProvide> at com.carolinarollergirls.scoreboard.event.ScoreBoardEventProviderImpl.isWritable(ScoreBoardEventProviderIm> at com.carolinarollergirls.scoreboard.event.ScoreBoardEventProviderImpl.add(ScoreBoardEventProviderImpl.java> at com.carolinarollergirls.scoreboard.event.ScoreBoardEventProviderImpl.add(ScoreBoardEventProviderImpl.java> at com.carolinarollergirls.scoreboard.core.game.GameImpl._startTimeout(GameImpl.java:753) at com.carolinarollergirls.scoreboard.core.game.GameImpl.timeout(GameImpl.java:620) at com.carolinarollergirls.scoreboard.core.game.GameImpl.execute(GameImpl.java:429) at com.carolinarollergirls.scoreboard.json.ScoreBoardJSONSetter.set(ScoreBoardJSONSetter.java:204) at com.carolinarollergirls.scoreboard.json.ScoreBoardJSONSetter.set(ScoreBoardJSONSetter.java:224) at com.carolinarollergirls.scoreboard.json.ScoreBoardJSONSetter.set(ScoreBoardJSONSetter.java:172) at com.carolinarollergirls.scoreboard.jetty.WS$ScoreBoardWebSocket$1.run(WS.java:152) at com.carolinarollergirls.scoreboard.event.ScoreBoardEventProviderImpl.runInBatch(ScoreBoardEventProviderIm> at com.carolinarollergirls.scoreboard.jetty.WS$ScoreBoardWebSocket.onMessage(WS.java:149) at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) at java.base/java.lang.reflect.Method.invoke(Method.java:580) at org.eclipse.jetty.websocket.common.events.annotated.CallableMethod.call(CallableMethod.java:70) at org.eclipse.jetty.websocket.common.events.annotated.OptionalSessionCallableMethod.call(OptionalSessionCal> at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextMessage(JettyAnnotatedEventDriv> at org.eclipse.jetty.websocket.common.message.SimpleTextMessage.messageComplete(SimpleTextMessage.java:69) at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.appendMessage(AbstractEventDriver.java:67) at org.eclipse.jetty.websocket.common.events.JettyAnnotatedEventDriver.onTextFrame(JettyAnnotatedEventDriver> at org.eclipse.jetty.websocket.common.events.AbstractEventDriver.incomingFrame(AbstractEventDriver.java:152) at org.eclipse.jetty.websocket.common.WebSocketSession.incomingFrame(WebSocketSession.java:326) at org.eclipse.jetty.websocket.common.extensions.AbstractExtension.nextIncomingFrame(AbstractExtension.java:> at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.nextIncomingFrame(PerMe> at org.eclipse.jetty.websocket.common.extensions.compress.CompressExtension.forwardIncoming(CompressExtensio> at org.eclipse.jetty.websocket.common.extensions.compress.PerMessageDeflateExtension.incomingFrame(PerMessag> at org.eclipse.jetty.websocket.common.extensions.ExtensionStack.incomingFrame(ExtensionStack.java:202) at org.eclipse.jetty.websocket.common.Parser.notifyFrame(Parser.java:225) at org.eclipse.jetty.websocket.common.Parser.parseSingleFrame(Parser.java:259) at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.> at org.eclipse.jetty.websocket.common.io.AbstractWebSocketConnection.onFillable(AbstractWebSocketConnection.> at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) at java.base/java.lang.Thread.run(Thread.java:1583)

[frank-weinberg]

This is another instance of the same bug as #704 and #705. We also had at least one instance of it at European Regional Championships. The CRG managers there were taking snapshots of the data every 30s and I should get this data for the game in question. I hope that this will allow me to figure out what the actual root cause is.

[JeneralPain]

Yeah I've got full back ups on the server, just need to power the box back up. Sent from my iPhoneOn 11 Jun 2024, at 00:19, frank-weinberg @.***> wrote: This is another instance of the same bug as #704 and #705. We also had at least one instance of it at European Regional Championships. The CRG managers there were taking snapshots of the data every 30s and I should get this data for the game in question. I hope that this will allow me to figure out what the actual root cause is.

—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you authored the thread.Message ID: @.***>