blocks not advancing on mainnet

[johnreitano]

Type:

Issue

What happened? / What do you need?:

block number is stuck at 3078951. This is shown on the console by entering kcoin.blockNumber and also on the stats page https://kusd.kowala.tech/

If it's a request, what do you need it for?:

To get the system working

If it's an issue, how can we reproduce it?:

1. Download kcoin client.
2. run kcoin init then run kcoin --nat none console
3. At console, enter: kcoin.syncing until it returns false
4. check value of kcoin.blockNumber

Affected servers or services:

kcoin client

Priority:

High

• High = We should stop anything we are doing and do this. If you're writing this calmly, you should probably choose another option. If it's really that urgent, please make sure we read this.
• Medium = It's important that we do this within a few days.
• Low = We will consider this on our next sprint planning.

(Please be aware that your priority may not match ours, we'll use this as guidance only).

Other info we may need:

[rgeraldes]

Additionally, we need to add monitoring/alert services.

[acroca]

The genesis validator was down but we lost its logs, so we don't know why. I think the instance ran out of disk space because it was logging locally instead of sending logs to cloudwatch. Now it's fixed. We have to wait for the validator to sync up with the network and start validating to see if it works again.

[acroca]

Working again