near / stakewars-iii

Stake Wars: Episode 3 challenges and place to report issues
87 stars 177 forks source link

Missing chunk after update lastest code #76

Open tranhoaison opened 2 years ago

tranhoaison commented 2 years ago

Hello,

I have just updated my node:

sudo systemctl stop neard
cd ~/nearcore
git fetch
git checkout 68bfa84ed1455f891032434d37ccad696e91e4f5
cargo build -p neard --release --features shardnet
sudo systemctl start neard

But it gets error: Missing chunk....

image

P/s: My vps: 4vCpu, 8GB RAM, 160GB SSD - Ubuntu 22.04

scholtz commented 2 years ago

These are my logs..

logs-from-stakewars-2-nonarchival-deployment-in-stakewars-2-nonarchival-deployment-5996f6b5dd-xtx8p.log

Notice repeated processed block and over 30 sec processing times..

image

joesixpack commented 2 years ago

P/s: My vps: 4vCpu, 8GB RAM, 160GB SSD - Ubuntu 22.04

Highly likely this is underpowered. 1 core = 2 vCPU's.

tranhoaison commented 2 years ago

Highly likely this is underpowered. 1 core = 2 vCPU's.

not only me, most testers gets same error with me after update.

scholtz commented 2 years ago

logs-from-stakewars-nonarchival-deployment-in-stakewars-nonarchival-deployment-6b877f9579-bzqkn.zip

I am running it in kubernetes, these logs are from stakewars-fi-2 validator.. Btw it would be nice at the start of the log to print out validator public key.

image

Notice that with image g68bf.. it still has missed producing the blocks and chunks even it was uptime 100% and ram, cpu and network usage is highly underused..

resources section in deployment setup:

          resources:
            requests:
              memory: "10000Mi"
              cpu: "1000m"
            limits:
              memory: "20000Mi"
              cpu: "8000m"

https://ludovit-scholtz.gitbook.io/stakewars-iii/challange-2/run-node

scholtz commented 2 years ago

image

doma2k commented 2 years ago

I have same issue after updating config.json

Screen Shot 2022-08-04 at 10 16 48

"build":"1.1.0-2565-g68bfa84ed","rustc_version":"1.62.1" VDS 16cpu 64ram 1.8TB NVME

DDeAlmeida commented 2 years ago

@mina86 @bowenwang1996

scholtz commented 2 years ago

With commit https://github.com/near/nearcore/commit/7395551f9e556b504a3e040da2797f53115e3480 ( trunk/1.1.0-2570-g7395551f9 )

logs-from-stakewars-de-1-testnet-soldier-app-deployment-in-stakewars-de-1-testnet-soldier-app-deployment-56b4cc94c8-fbbrs.log

image

1100x times is in the logs WARN sync: sync_hash `J4mpf6yZ5c119dY1dz7abuSxGjoVDVmG1iUPSfi8Fx89` didn't pass validation, possible malicious behavior

image

ayakashi86 commented 2 years ago

missing chunk

Seem like many people on NEAR discord get the same problems with missing chunk...

scholtz commented 2 years ago

logs-from-stakewars-2-nonarchival-deployment-in-stakewars-2-nonarchival-deployment-5dc9798465-bf7mw.log

For the record logs with the latest build ( 1.1.0-2580-g78ef2f558 ) (from scholtz.factory.shardnet.near validator)

Still flooded by repeated "Processed in progress" logs

scholtz commented 2 years ago

logs-from-stakewars-2-nonarchival-deployment-in-stakewars-2-nonarchival-deployment-5dc9798465-bf7mw (1).log

image

Just before the error was forwarded multiple times is usually WARN network: Peer stream error err=Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }

and that might result in non producing blocks?

image