Closed alex-drocks closed 1 month ago
same issue here +1
This issue came back again after wiping my folders and using the rust based subtensor instead of docker š¦ Node breaks after a certain point š¦
I got the problem with a ryzen 7950x Debian using Docker subtensor (original issue) AND then wiped out disk to reinstall with Rust based subtensor also.
Just also found two other servers on different providers local subtensor out of sync.
All same cause
Facing the same problem
This is a huge problem for us. We've been seeing this on our own subtensor node, and across many validators on the subnet, i.e. this is a bittensor-breaking issue which should be addressed ASAP to maintain the integrity of the entire chain.
So with that said, let's try to be constructive here:
Currently, latest main is not runnable on our machines, that's using docker or directly compiling from main...
@distributedstatemachine any ideas?
For anyone still experiencing this issue, I pulled Release v1.1.1, cleaned blockchain data, rebuilt cargo and started the subtensor again and it works fine.
I am using the subtensor only to fetch data from the chain and for no other reason. If you are using the subtensor for any other reason, I can't guarantee this release is gonna work smoothly for you.
Workaround until this issue is fixed.
For anyone still experiencing this issue, I pulled Release v1.1.1, cleaned blockchain data, rebuilt cargo and started the subtensor again and it works fine.
I am using the subtensor only to fetch data from the chain and for no other reason. If you are using the subtensor for any other reason, I can't guarantee this release is gonna work smoothly for you.
Workaround until this issue is fixed.
My node will run for about 24h normally before it encounters the error again when starting from scratch with a clean install. Have yours been working for longer than this?
The past week that's what was happening for me as well, but yesterday it wouldn't even work for a while, it immediately started spamming the same errors.
With the old Release, no issues for 24 hours so far, and the node is syncing fine.
With the old Release, no issues for 24 hours so far, and the node is syncing fine.
Yep can also confirm 1.1.1
is working for me:
cd subtensor
git fetch --all
git reset --hard 703bdbd178c5d6e64edd4ffcf69aa49d63349290
cargo build --release --features pow-faucet
... Restart
https://github.com/opentensor/subtensor/commit/703bdbd178c5d6e64edd4ffcf69aa49d63349290
Confirming a workaround is to use an old node version, it is fully compatible with the latest runtime. We're using a625834e
on a test machine and it syncs fine.
We're continuing to investigate this, thanks for bearing with us.
This is fixed in #714
A node will attempt to use its native runtime in substitute for the on-chain Wasm runtime if all of spec_name, spec_version, and authoring_version are the same between the Wasm and native binaries. This was causing nodes compiled from main to use a runtime slightly different compared to what is on-chain, and write invalid stuff to storage.
In the future, CI should prevent us from ever pushing to main a spec version that matches what is on-chain (cc @sam0x17)
Describe the bug
Local subtensor node found out of sync this morning, with an error log like this:
Then restarted it and it was throwing tons of errors like this after the restart:
To Reproduce
Running local subtensor node on ryzen 7950x Debian with this:
Expected behavior
No panick error logs
Screenshots
Environment
Linux Debian 12
Additional context
Node was up and running for a few days.
This morning at 5am EST, I tried btcli w balance command and saw outdated data so I inspected the subtensor logs.
I had this error in the logs:
After seeing that it was broken (didn't sync to the chain properly afterwards) I just restarted the node and then it had thousands of lines per second containing the error: