GQ Client/Server 0.10.0 Pre 4 - Stations disappear from display after a few minutes

[andyaz]

GQ Client/Server 0.10.0 Pre 4 - Stations disappear from display after a few minutes. Console window shows "Station Index Has Changed!" when it happens. Restart the client and connect to server and stations return for a few minutes until it happens again.

[andyaz]

Additional info. I have 5200 stations feeding, perhaps related to large number of stations?

[andyaz]

I am unsure if this is related but the console is full of these error messages

[xspanger3770]

This really looks like a bug. I will try to investigate later today.

[xspanger3770]

Isn't there some error also in the server's console?

[andyaz]

Hi Jakub - There is no indication of issues in the server side console. Below is a capture from the console on the server. Here you see when I closed the client we got the expected socket error. Then you see the client reconnect as Client 8. About 2 minutes after the reconnect all the stations dissappeared from the client display but nothing was shown in the console window:

[10:59:28] ERROR: java.net.SocketException: Connection reset at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:328) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:355) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:808) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:966) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:961) at java.base/java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2886) at java.base/java.io.ObjectInputStream$BlockDataInputStream.peek(ObjectInputStream.java:3213) at java.base/java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:3223) at java.base/java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1695) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:514) at java.base/java.io.ObjectInputStream.readObject(ObjectInputStream.java:472) at gqserver.api.ServerClient.readPacket(ServerClient.java:53) at gqserver.server.ClientReader.run(ClientReader.java:22) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:842) [10:59:31] WARN: Not updating because: PREVIOUS_WAS_BETTER [10:59:31] INFO: Hypocenter finding finished in: 56 ms [10:59:31] INFO: Client #7 disconnected due to timeout [10:59:37] INFO: A new client is joining... [10:59:37] INFO: Performing handshake for client #8 [10:59:38] INFO: Client #8 handshake successfull [10:59:40] INFO: Hypocenter finding finished in: 73 ms [10:59:40] INFO: Hypocenter finding finished in: 71 ms [10:59:44] INFO: Hypocenter finding finished in: 96 ms [10:59:46] INFO: Hypocenter finding finished in: 93 ms [10:59:53] WARN: Not updating because: PREVIOUS_WAS_BETTER [10:59:53] INFO: Hypocenter finding finished in: 130 ms [10:59:55] INFO: Hypocenter finding finished in: 89 ms [10:59:56] WARN: Not updating because: PREVIOUS_WAS_BETTER [10:59:56] INFO: Hypocenter finding finished in: 114 ms [11:01:26] WARN: Not updating because: PREVIOUS_WAS_BETTER [11:01:26] INFO: Hypocenter finding finished in: 96 ms [11:01:54] WARN: Not updating because: PREVIOUS_WAS_BETTER [11:01:54] INFO: Hypocenter finding finished in: 108 ms [11:02:00] WARN: Not updating because: PREVIOUS_WAS_BETTER [11:02:00] INFO: Hypocenter finding finished in: 138 ms [11:02:44] INFO: Hypocenter finding finished in: 170 ms [11:03:10] INFO: Hypocenter finding finished in: 34 ms [11:04:00] INFO: Hypocenter finding finished in: 78 ms [11:04:01] INFO: Hypocenter finding finished in: 114 ms [11:04:01] WARN: Not updating because: PREVIOUS_WAS_BETTER [11:04:01] INFO: Hypocenter finding finished in: 147 ms [11:04:01] INFO: Hypocenter finding finished in: 119 ms

[andyaz]

Here are some screenshots of the server window to provide additional context for you.

[xspanger3770]

Hmm that's really weird, I never had this issue happen to me and I'm running 3000 stations on my local server... There is really no error in the server console, but some errors might be hidden in the code by mistake..

[xspanger3770]

Also the "station indexing reset" message should only appear once in the client unless the server has been restarted, so if it appears more times there is definitely something wrong

[andyaz]

Here is what the client showed after another client restart. The Client restart was at 11:29:50 or so and you can see the indexing message at 11:29:51. The triangles all disappeared from the client display at 11:34:54 and there were no further messages on the console even after 12+ minutes. BTW, I have also tried restart of the server last night with the same behavior on the part of the client just to see if there was something strange on that end. I guess we will need some form of debug logging to root cause this for you if you do not see it on your test instance. Also may be of importance, this is running on Windows 10 on both server/client side. Here is the JDK information:

IMPLEMENTOR="Oracle Corporation" JAVA_RUNTIME_VERSION="17.0.9+11-LTS-201" JAVA_VERSION="17.0.9" JAVA_VERSION_DATE="2023-10-17" LIBC="default" MODULES="java.base java.compiler java.datatransfer java.xml java.prefs java.desktop java.instrument java.logging java.management java.security.sasl java.naming java.rmi java.management.rmi java.net.http java.scripting java.security.jgss java.transaction.xa java.sql java.sql.rowset java.xml.crypto java.se java.smartcardio jdk.accessibility jdk.internal.jvmstat jdk.attach jdk.charsets jdk.compiler jdk.crypto.ec jdk.crypto.cryptoki jdk.crypto.mscapi jdk.dynalink jdk.internal.ed jdk.editpad jdk.hotspot.agent jdk.httpserver jdk.incubator.foreign jdk.incubator.vector jdk.internal.le jdk.internal.opt jdk.internal.vm.ci jdk.internal.vm.compiler jdk.internal.vm.compiler.management jdk.jartool jdk.javadoc jdk.jcmd jdk.management jdk.management.agent jdk.jconsole jdk.jdeps jdk.jdwp.agent jdk.jdi jdk.jfr jdk.jlink jdk.jpackage jdk.jshell jdk.jsobject jdk.jstatd jdk.localedata jdk.management.jfr jdk.naming.dns jdk.naming.rmi jdk.net jdk.nio.mapmode jdk.random jdk.sctp jdk.security.auth jdk.security.jgss jdk.unsupported jdk.unsupported.desktop jdk.xml.dom jdk.zipfs" OS_ARCH="x86_64" OS_NAME="Windows" SOURCE=".:git:9272770fe13f open:git:3b8ba2bb0043"

From the Client console window:

D:\GlobalQuake>java -jar GlobalQuake_v0.10.0_pre4.jar Failed to load CUDA: no gq_hypocs in java.library.path: C:\Program Files\Java\jdk-17\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files\Common Files\Oracle\Java\javapath;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\Program Files\Zulu\zulu-11\bin\;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files (x86)\NTP\bin;C:\WSJT\wsjtx\bin;C:\Program Files\IDM Computer Solutions\UltraEdit;C:\Program Files\PuTTY\;C:\Program Files\xLights;C:\Program Files\dotnet\;C:\Program Files (x86)\dotnet\;C:\Program Files (x86)\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Intel\Intel(R) Management Engine Components\DAL;C:\Program Files\Java\jdk-17;D:\apache-maven-4.0.0-alpha-8\bin;;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Users\andy\AppData\Local\Microsoft\WindowsApps;C:\Users\andy\AppData\Local\GitHubDesktop\bin;. Iteration #0 difficulty: 90000.00K Iteration #1 difficulty: 22500.00K Iteration #2 difficulty: 15000.00K Iteration #3 difficulty: 15000.00K Iteration #4 difficulty: 15000.00K Iteration #0 space H 4.55km V 50.000000km Iteration #1 space H 4.07km V 10.000000km Iteration #2 space H 1.41km V 5.000000km Iteration #3 space H 0.45km V 2.000000km Iteration #4 space H 0.22km V 0.500000km Cuda loaded: false Intensity tables filled in 14 ms. [11:29:51] INFO: Station indexing has changed!

[andyaz]

Jakub, some new info. If I uncheck the hide stations with no data on the client all the triangles are there but they are gray... so it looks like it thinks there is no station data.

[andyaz]

Also with that box unchecked I see flashing activity indications on the grayed out stations.

[xspanger3770]

Oh I see.. maybe you have different timezone settings on client and server?

[xspanger3770]

or maybe the time itself got desynchronized either on the server or in client

[andyaz]

Both computers are in America Los Angeles tz and time is synchronized to my local stratum 1 GPS NTP so clocks should not be an issue. If the client server is reliant on time zones that would be bad for the scenario of server in one tz and clients in others. I would think you should have based all timestamps on UTC...

Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Jakub Španger @.> Sent: Friday, November 10, 2023 12:24:12 PM To: xspanger3770/GlobalQuake @.> Cc: Andrew Hansis @.>; Author @.> Subject: Re: [xspanger3770/GlobalQuake] GQ Client/Server 0.10.0 Pre 4 - Stations disappear from display after a few minutes (Issue #167)

or maybe the time itself got desynchronized either on the server or in client

— Reply to this email directly, view it on GitHubhttps://github.com/xspanger3770/GlobalQuake/issues/167#issuecomment-1806386446, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEVC6DPTFEX6RLQ3UPCPE5LYD2EOZAVCNFSM6AAAAAA7FS76WWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGM4DMNBUGY. You are receiving this because you authored the thread.Message ID: @.***>

[xspanger3770]

Can you try hovering your mouse over a station and see the delay displayed below?

[andyaz]

I will try that when I get back from lunch.

Sent from my Verizon, Samsung Galaxy smartphone Get Outlook for Androidhttps://aka.ms/AAb9ysg

---

From: Jakub Španger @.> Sent: Friday, November 10, 2023 12:34:35 PM To: xspanger3770/GlobalQuake @.> Cc: Andrew Hansis @.>; Author @.> Subject: Re: [xspanger3770/GlobalQuake] GQ Client/Server 0.10.0 Pre 4 - Stations disappear from display after a few minutes (Issue #167)

Can you try hovering your mouse over a station and see the delay displayed below?

— Reply to this email directly, view it on GitHubhttps://github.com/xspanger3770/GlobalQuake/issues/167#issuecomment-1806398052, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AEVC6DIMUUAPBABJCAPTF5LYD2FVXAVCNFSM6AAAAAA7FS76WWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMBWGM4TQMBVGI. You are receiving this because you authored the thread.Message ID: @.***>

[andyaz]

Zoom and right click shows no data when all are grayed out.. When I restart the client and look while they are blue the delays are counting up until they go gray. Even my lowest delay stations on the AM network from my own seedlink in local mode only show about 1 to 4 seconds delay, the delay just ticks up. It looks like it gets the initial data indexed and then times out like it never sees any station data again. Every other part of the client display appears to function properly including the green flashing boxes for triggered stations come and go. If I enable "Last Data Time" on the client it is visibly in sync with the system clock to the second by eye.

FYI - I did verify that both server and client are in sync with the Stratum 1 GPS clock so if there is a clock difference it is in milliseconds...

[andyaz]

II take back the statement about the station trigger flashing squares. They are stuck in the state last seen and are not changing over time. If they were flashing trigger at the time of the data timeout they are remaining in that state.

[YacineBoussoufa]

Have you tried seeing if the Firewall is blocking GlobalQuake? The flashing beein stuck in the last data recorded, happens to me when my internet crashes. So probably something either blocking the connection (like the firewall), or another service using the same port of GlobalQuake, and creating a conflict.

[andyaz]

Have you tried seeing if the Firewall is blocking GlobalQuake? The flashing beein stuck in the last data recorded, happens to me when my internet crashes. So probably something either blocking the connection (like the firewall), or another service using the same port of GlobalQuake, and creating a conflict.

Arrrgggghhh!!!! You are a genius Yacine... When the port changed to 38000 something in one of the boxes windows firewalls must have been partially blocking things. I set it back to 12345 on both ends and now it works. I will investigate further to get it on the new recommended port. Great catch I had not thought about since both boxes are on the same subnet with no firewalls between them. I did not think to look at the Win Firewalls. DOH!

[andyaz]

I will close this...