search

rchain-community / mainnet-outage

0 stars 0 forks source link

Unexpected Error - observer down #33

Closed JayeshNaphade closed 1 year ago

JayeshNaphade commented 1 year ago
Questions Needed Information
Date June 20/2023
Outage Type Rchain developer interface explore check balance is not working.Unexpected error while capturing results from Rholang: java.lang.RuntimeException: unknown rootWallet Outage deploy explore fails.
Nature of Problem TBD
Time Circuit Went Down DD: 20 HH:0900 MM:47
Time Circuit Was Restored DD: HH: MM:
Site (https://rholang.io/oc-mainnet-wallet/)
Operator Hetzner
Current Status Down
Action Taken Explore Test
Bill-Kunj commented 1 year ago

Fixed: rebooted traefik and observers.

JayeshNaphade commented 1 year ago

Screenshot (129) Rchain developer interface explore check balance is not working and the oc-mainnet-wallet is also giving an error.

Bill-Kunj commented 1 year ago

Rchain developer interface explore check balance is not working and the oc-mainnet-wallet is also giving an error.

Please provide procedure to reproduce the error

Bill-Kunj commented 1 year ago

I note that the dropdown menus include testnet, which hasn't existed in many months. It seems the software for the wallet is intolerably out-of-date.

Selecting a mainnet server from the dropdown and clicking on status provides a correct response from the observer. Maybe something is wrong with your wallet code?

https://observer.services.mainnet.rchain.coop/status

jimscarver commented 1 year ago

the Asian observer is working fine, so we are not down completly. image

azazime commented 1 year ago

The Observer node needs to be restarted

Bill-Kunj commented 1 year ago

There is now a casper failure on observer.services.mainnet.rchain.coop. L3 is working the problem

Bill-Kunj commented 1 year ago

@JayeshNaphade @jimscarver I note that your screenshots include https://rholang.io/oc-mainnet-wallet/. I thought the mainnet wallet was running elsewhere. What is rholang.io, and why are the testnet servers listed in the dropdowns? This really looks like a non-RevDev wallet

Bill-Kunj commented 1 year ago

Reviewing the history, it appears the observer ran out of memory about a week ago. Which probably isn't possible under normal operation. The observer is NOT down, but it is operating with a casper failure

Bill-Kunj commented 1 year ago

https://observer.services.mainnet.rchain.coop/api/blocks is now functional.

Please retest your failure scenario and close if all is fixed @JayeshNaphade

JayeshNaphade commented 1 year ago

I have tested the failure scenario thoroughly, and the issue has been resolved completely. I have verified the fix and can confirm that everything is functioning as expected. Therefore, I am closing the issue now.