search

Kriptikz / ore-hq-server

34 stars 37 forks source link

Continuously disconnecting and reconnecting #3

Open wstdn opened 1 month ago

wstdn commented 1 month ago

2024-08-12T17:26:29.961701Z ERROR ore_hq_server: Client tried to ready up with expired signed message 2024-08-12T17:26:29.961772Z INFO ore_hq_server: Client

quite a lot: 2024-08-12T17:27:14.986301Z DEBUG request{method=GET uri=/?timestamp=1723483635 version=HTTP/1.1 headers={"host": "192.168.6.156", "connection": "upgrade", "upgrade": "websocket", "sec-websocket-version": "13", "sec-websocket-key": "v07mM1sfAp1w5tuzbaTv4w==", "authorization": "Basic

wstdn commented 1 month ago

Thank you for sharing.

wstdn commented 1 month ago

Hello! First of all, I would like to express my sincere gratitude for developing and sharing the excellent tools, ore-hq-server and ore-hq-client. They have greatly assisted us in our cluster mining work, and we deeply appreciate your contribution.

In our recent tests, we used 32 and 100 clients for experiments but encountered some issues during operation. Typically, after running for ten to thirty minutes, the system starts to exhibit the following errors:

1.  Authentication Failure: We noticed frequent status=401 errors in the logs, indicating authentication failures. We have verified that the authentication credentials on both the client and server sides are correct, but the problem persists.
2.  WebSocket Connection Issues: During testing, we found that the clients sometimes experienced unstable or disconnected WebSocket connections when attempting to connect to the server.
3.  Expired Signature Messages: We found the error Client tried to ready up with expired signed message in the logs, which seems to be caused by the expiration of the client’s signed message.
4.  Processing Lag: In some cases, the server gets stuck at Sending signed tx... with attempt: 3 in the logs and cannot continue processing.

These issues have occurred in tests with different scales (32 and 100 clients), significantly impacting system stability. We have tried various solutions, but the problems remain unresolved. I hope this test can help you, as many others in the Discord community have also encountered similar issues.

Once again, thank you for your contributions to the community, and we look forward to your assistance and guidance.

Wishing you all the best!

Kriptikz commented 1 month ago

2024-08-12T17:26:29.961701Z ERROR ore_hq_server: Client tried to ready up with expired signed message 2024-08-12T17:26:29.961772Z INFO ore_hq_server: Client

quite a lot: 2024-08-12T17:27:14.986301Z DEBUG request{method=GET uri=/?timestamp=1723483635 version=HTTP/1.1 headers={"host": "192.168.6.156", "connection": "upgrade", "upgrade": "websocket", "sec-websocket-version": "13", "sec-websocket-key": "v07mM1sfAp1w5tuzbaTv4w==", "authorization": "Basic

This issue should resolved on latest version. Let me know it it's still happening after updates.

Kriptikz commented 1 month ago

Hello! First of all, I would like to express my sincere gratitude for developing and sharing the excellent tools, ore-hq-server and ore-hq-client. They have greatly assisted us in our cluster mining work, and we deeply appreciate your contribution.

In our recent tests, we used 32 and 100 clients for experiments but encountered some issues during operation. Typically, after running for ten to thirty minutes, the system starts to exhibit the following errors:

1.    Authentication Failure: We noticed frequent status=401 errors in the logs, indicating authentication failures. We have verified that the authentication credentials on both the client and server sides are correct, but the problem persists.
2.    WebSocket Connection Issues: During testing, we found that the clients sometimes experienced unstable or disconnected WebSocket connections when attempting to connect to the server.
3.    Expired Signature Messages: We found the error Client tried to ready up with expired signed message in the logs, which seems to be caused by the expiration of the client’s signed message.
4.    Processing Lag: In some cases, the server gets stuck at Sending signed tx... with attempt: 3 in the logs and cannot continue processing.

These issues have occurred in tests with different scales (32 and 100 clients), significantly impacting system stability. We have tried various solutions, but the problems remain unresolved. I hope this test can help you, as many others in the Discord community have also encountered similar issues.

Once again, thank you for your contributions to the community, and we look forward to your assistance and guidance.

Wishing you all the best!

Thanks for the details! There is definitely a lot of room for optimizations on server side. Any chance you can show cpu usages and memory? Curious how those look with 100 clients. Also what are the specs of the server?

I still need to finish a few things with database and client side claims. But then I will be looking into making it more efficient overall.