search

thetatoken / guardian-mainnet-guide

Guide for setting up the Guardian Node for the mainnet
https://docs.thetatoken.org/
147 stars 34 forks source link

ValidateSourceAdvanced failed #78

Open gn02960889 opened 3 years ago

gn02960889 commented 3 years ago

I found a lot of error message on log file. Is this normal?

[2021-03-11 18:10:38.112] [error] [2021-03-11 18:10:38] WARN [ledger] validateSourceAdvanced failed on 0xAE17b59b2587D01AFa10361b5045B14DB627B2E8: Result{code:100002, message:ValidateInputAdvanced: Got 4, expected 5. (acc.seq=4)} [2021-03-11 18:10:38.112] [info] [2021-03-11 18:10:38] WARN [ledger] validateSourceAdvanced failed on 0xAE17b59b2587D01AFa10361b5045B14DB627B2E8: Result{code:100002, message:ValidateInputAdvanced: Got 4, expected 5. (acc.seq=4)}

Larix-Laricina commented 3 years ago

I also have lots of these, would be useful to know what they mean.

nexster commented 3 years ago

My power went out a few days ago and now I just get this messege in log and no Tfuel. So much for passive income.

OzymandiasOwl75 commented 3 years ago

I have the same Guardian node WARN [ledger] validateSourceAdvanced failed error messages. I have only being running Guardian Node for 24 hours. Still no TFuel. Connectivity says good with 23 Guardian Peers. I assume lack of TFuel earned relates to this message

nexster commented 3 years ago

I unstaked and now I am waiting waiting 3 days for theta back. Will wipe node computer and restake. Will update results then.

OzymandiasOwl75 commented 3 years ago

I unstaked and now I am waiting waiting 3 days for theta back. Will wipe node computer and restake. Will update results then.

I am not certain its a problem with node setup. Noticed on my Edge Node I have pending earnings of 0.0015 TFuel. I am new to this, but its clear guardian node is showing TFuel rounded to a whole number. I am only staking 1000 Theta, which is minimum requirement. Maybe there is'nt an issue, but it would be interesting to know what TFuel earning should be expected on 1000 Theta staked over 24 hours

nexster commented 3 years ago

I have 1100 Theta and it was making 1.2 Tfuel a day. But 0 per day for last 3 days since my power went 9out.

OzymandiasOwl75 commented 3 years ago

I have 1100 Theta and it was making 1.2 Tfuel a day. But 0 per day for last 3 days since my power went 9out.

OK, thanks for that. I will leave mine running and see it it resolves itself. I will let you know how I get on

gn02960889 commented 3 years ago

@jieyilong I saw many "ValidateSourceAdvanced failed" message and no Tfuel rewards.Any ideas about this case?

AquaWookie commented 3 years ago

Same issue. After approximately 2 months of a functional node, 2 days ago I stopped receiving TFuel and now the logs are telling me the same shenanigans:

[2021-03-12 07:23:49.109] [info] [2021-03-12 07:23:49] WARN [ledger] validateSourceAdvanced failed on 0x147F2Dd6A52Be02ecBCc5488fbc6636AD28FA989: Result{code:100002, message:ValidateInputAdvanced: Got 5, expected 6. (acc.seq=5)}

image

Not having any luck troubleshooting...

Waldojesus commented 3 years ago

I'm actually having something strange happen as well. I think I'm receiving too much Tfuel. I had been averaging about one a day and last night I earned 12. I wonder if there is some weird redistribution happening from failed nodes.

gn02960889 commented 3 years ago

@AquaWookie That is what I see. Health check failed

sirbrokealot commented 3 years ago

I have the same problem with 2.2.10.

Interesting for me ist visiting https://guardianmonitor.io/ and scroll way down and search for my Guardian Node address. It actually shows an increasing 1DTf number while in the GN app there is a frozen 0 session earnings.

In the log I get for example: [2021-03-13 12:07:04.018] [error] [2021-03-13 12:07:04] WARN [ledger] validateSourceAdvanced failed on 0xb9bc591387b58a269ee94EB0602E54E78FfaFfb2: Result{code:100002, message:ValidateInputAdvanced: Got 125, expected 126. (acc.seq=125)}

The stopping, starting bug could maybe be resolved with clicking Node -> Deleting Node Database

Some people said, that the distribution from Tfuel has changed recently and we could check some "medium" article.

Please clarify where can we read about it and how to fix the 0 session earning which might be connected to the validateSourceAdvanced failed error.

OzymandiasOwl75 commented 3 years ago

I have the same problem with 2.2.10.

Interesting for me ist visiting https://guardianmonitor.io/ and scroll way down and search for my Guardian Node address. It actually shows an increasing 1DTf number while in the GN app there is a frozen 0 session earnings.

In the log I get for example: [2021-03-13 12:07:04.018] [error] [2021-03-13 12:07:04] WARN [ledger] validateSourceAdvanced failed on 0xb9bc591387b58a269ee94EB0602E54E78FfaFfb2: Result{code:100002, message:ValidateInputAdvanced: Got 125, expected 126. (acc.seq=125)}

The stopping, starting bug could maybe be resolved with clicking Node -> Deleting Node Database

Some people said, that the distribution from Tfuel has changed recently and we could check some "medium" article.

Please clarify where can we read about it and how to fix the 0 session earning which might be connected to the validateSourceAdvanced failed error.

I have heard that Theta have changed the way TFuel is accrued. You still get the same amount of TFuel, but the payments to your account are accumulated.

I have heard it relates to the Theta MainNet 3.0 upgrade. I suspect what you are seeing is the TFuel has been accrued. I assume a Gaurdian Node upgrade maybe available for this next month https://www.thetatoken.org/

sirbrokealot commented 3 years ago

Thanks is there anything to read about accumulation thresholds?

ATTENTION DONT FALL FOR SCAMS IN THIS MATTER. I got some chat inquiries from the scammer thetatokens (https://www.reddit.com/user/thetatokens) and his scam site https://thetawebwalletsio.com BE ADVISED DONT FALL FOR THIS SCAM

the-quantum-rabbit commented 3 years ago

I'm getting the same issue after the node dropping 2 or 3 days ago. On restarting theta , the sync stopped at Thu Mar 11 07:49:21 CET 2021 { "address": "0x99d5AA6C86E690054f5B63DED992fB8E48ba6dAA", "chain_id": "mainnet", "current_epoch": "9585082", "current_height": "9508409", "current_time": "1615445361", "latest_finalized_block_epoch": "9574152", "latest_finalized_block_hash": "0xf8a92d717c75647142263913b6080feb7cdc34c1453a665cb89ccf9f2fe8fdea", "latest_finalized_block_height": "9497499", "latest_finalized_block_time": "1615370641", "peer_id": "0x99d5AA6C86E690054f5B63DED992fB8E48ba6dAA", "syncing": true }

Converting finalised_block_time 1615370641 timestamp in seconds to readable date with https://www.epochconverter.com/

Assuming that this timestamp is in seconds: GMT: Wednesday 10 March 2021 10:04:01 Your time zone: mercredi 10 mars 2021 11:04:01 GMT+01:00 Relative: 21 hours ago

I first tried a restart but the sync stopped again, so deleted the db and snapshot, got a new snapshot and restarted. sync again stops on another block. I see a lot of WARN [ledger] validateSourceAdvanced failed messages with sequence errors as reported before. I have tried differnt snapshots with the same results. :-(

OzymandiasOwl75 commented 3 years ago

Thanks is there anything to read about accumulation thresholds?

ATTENTION DONT FALL FOR SCAMS IN THIS MATTER. I got some chat inquiries from the scammer thetatokens (https://www.reddit.com/user/thetatokens) and his scam site https://thetawebwalletsio.com BE ADVISED DONT FALL FOR THIS SCAM

I really am very new to all this, but its clear Theta MainNet is currently going through a period of change and expansion. That's what I like about it.

https://medium.com/theta-network/theta-protocol-code-v2-2-0-released-with-important-performance-related-updates-6b9e2261d0ae

The more I read the more I like it.

https://medium.com/theta-network/theta-network-2021-roadmap-69053f16a870

rchandi commented 3 years ago

Hi All,

I am having the same problem.

I started up my guardian node (on a desktop) about 1 day ago and currently I am showing 0.00 Session earnings on the Guardian node, however; on https://guardianmonitor.io/ my node is said to be online and it looks like it is accruing TFuel rewards.

The logs are also showing:

WARN [ledger] validateSourceAdvanced failed on 0x2FeF30e4cAfddDC10b06FCC07459F54A559c1A3B: Result{code:100002, message:ValidateInputAdvanced: Got 6, expected 7. (acc.seq=6)}

I am going to let it run for a few days and maybe it works itself out. If not I will have to try to un-install and re-install the entire node...not sure what else I can do.

nexster commented 3 years ago

I have not. Does not matter to me my theta is already unstaked waiting for it to return to wallet. Install new guardians node and try that.

rchandi commented 3 years ago

@DavidGarry7654 thanks for the tip. I have reached out and will see what they recommend.

nexster commented 3 years ago

I deleted my guardian node, reinstalled, restaked still getting same errors in log, but it looks like it’s working from UI. No Tfuel yet. Will update if I get any Tfuel,

rchandi commented 3 years ago

I emailed the Theta support team. Will keep everyone updated here.

phyx1u5 commented 3 years ago

im also getting the same problem, syncing is flase, but keep getting these validaorSourceAdvanced failed errors, im using the Linux CLI version

rrln commented 3 years ago

I found a lot of error message on log file. Is this normal?

[2021-03-11 18:10:38.112] [error] [2021-03-11 18:10:38] WARN [ledger] validateSourceAdvanced failed on 0xAE17b59b2587D01AFa10361b5045B14DB627B2E8: Result{code:100002, message:ValidateInputAdvanced: Got 4, expected 5. (acc.seq=4)} [2021-03-11 18:10:38.112] [info] [2021-03-11 18:10:38] WARN [ledger] validateSourceAdvanced failed on 0xAE17b59b2587D01AFa10361b5045B14DB627B2E8: Result{code:100002, message:ValidateInputAdvanced: Got 4, expected 5. (acc.seq=4)}

Following are my observations (I'm not an expert att all). It looks like there are two diffrent log entries on each row in the original post above. I.e first an entry "[2021-03-11 18:10:38.112] [error] " for one event. Then an entry, on the same row but for another event "[2021-03-11 18:10:38] WARN [ledger] validateSourceAdvanced failed ... "

I do think that the problem is that there are two theta processes running after a restart. I have several nodes and have seen this problem. Solved on Windows nodes by closing down the GN, kill all processes associated to GN still running (The Guardian Node, theta) via Task manager. Then start the GN (no need to clean DB/snapshoot. I had to reboot my linux servers at one point, this could be the reason for that as well, didn't check the processes before reboot.

I don't know for sure, but I do think "WARN [ledger] validateSourceAdvanced failed ... " is ok and per design of the protocol before consencus is reached. It is a warning, not an error.

Ijacques commented 3 years ago

Me to I am having several error codes showing on my node log, also my session earnings shows a constant zero yet I see that I am earning a lil tfuel because the total tfuel balance has increased a lil. I hope they work these bugs out soon.

nexster commented 3 years ago

Not getting error messages anymore now just getting WARN ledger messages. No Tfuel yet.

rchandi commented 3 years ago

I emailed the Theta support team. Will keep everyone updated here.

Hi Everyone,

I still have not heard back from the support team but I am starting to see TFuel earnings on my guardian node.

I suspect the log error:

"WARN [ledger] validateSourceAdvanced failed on 0x90a5C68EB3A4391fb48f9e7bd7B2d39Fb32601CD: Result{code:100002, message:ValidateInputAdvanced: Got 60, expected 61. (acc.seq=60)}"... Is unrelated to the tfuel sessions earning issue and may be a natural part of the guardian node and not a "bug".

Kuehn commented 3 years ago

@DavidGarry7654 So I also am having the same error messages and experience @rchandi appears to be having. I did go to telegram for the first time. Last message was on Feb 11th appeared to be stating it's on discord. Maybe I went to the wrong telegram group? Either way I asked my question there and everyone just stated to stake with Gpool to not have those issues anymore. So, I am going to keep looking into it. I'd rather not go stake with Gpool. Will post if I find a resolution to this error as well.

Kuehn commented 3 years ago

@DavidGarry7654 So I did reach out to theta support via telegram you sent, and they were going to help me out, however I went to my log and the error stopped showing up, had some warnings but the ValidateSourceAdvanced failed error was no longer there. So I am letting it run for a while and going to see if it comes back. Thanks for all your help so far.

sirbrokealot commented 3 years ago

If you got it fixed can you please tell us how you did it and what was suggested by theta support.

I still have errors and 0 session earnings.

nexster commented 3 years ago

Day 3 no Tfuel. Now getting errors and warnings.

Kuehn commented 3 years ago

Hey everyone, sorry for the slow reply. I will go through the run down, unfortunately the log error do appear to come back for me every once in while, however today I did get my first tFuel reward since trying to stake. Here is everything I did (unfortunately I don't think there is anything different that what you guys may have tried)

Thursday: Staked saw it was having errors, but left it for two days since I wasn't sure what was going on and thought it might take some time.

Saturday: Attempt #1: I decided to restart my node. This time when it loaded I was getting constant health checks failures in the recent activity and didn't let it run for that long as that seemed like an issue.

Attempt #2: I decided to delete my node database, delete my log file, and restart/sync my node database (I was reassured from reading other posts that deleting a node will not lose my current stake). At this point I was back at my original position of having the validate error, but no health check errors in the recent activity.

Sunday: I decided to delete my node database, delete my log file for a third time but this time I changed my performance tier to start to be Standard instead of Platinum (made sure I did it when I deleted my node as I didn't want the health check errors and that appeared to fix that.) This time I still got the errors in the log and came here, and once I reached out to telegram I was kind of embarrassed to find the errors were no longer there in the most recent part of the log. So The told me to likely delete my node database and resync if I have the issue again, but we didn't go into what it could have been since I wasn't producing it at the time. However I noticed later last night the errors do sometimes come back and go away, however today I got tfuel so i'm just going to monitor it for now and will update if I find an actual fix for it.

Sorry this all I got right now hope it helps someone, but currently I'm not sure why I got tfuel today but nothing before, and nor do I know why the error message goes away and comes back. I did realize I don't exactly meet the minimum free space in my hard drive, so I might move it all to a different SSD that is 1 terabyte to see if that resolves it, if it appears not to give me tfuel anymore.

sirbrokealot commented 3 years ago

I have the same thought in mind @Kuehn . Maybe it needs the 1TB SSD storage and/or the PC might be too slow for it. On Windows I tried to move the Guardian Node (GN) to a free 1 TB SSD. I went to %appdata% (AppData\Roaming\Theta Guardian Node) and saved the ThetaGN folder without the db folder as backup. I deleted all content in AppData\Roaming\Theta Guardian Node including the Theta Guardian Node folder and attempted a reinstall of GN. This means I put the setup file on my fresh SSD and ran it. On install I pointed GN to install the db in the very same fresh SSD. I let it download. When it turned towards the initial state of verifying close it and put your ThetaGN back in the AppData\Roaming\Theta Guardian Node. I hope that this will reinstall the GN in my fresh SSD without the trouble of restaking and getting rid of performance issues regarding to free disk space. I will post here soon if it works.

AquaWookie commented 3 years ago

I see some workarounds to try to trick a failing node into working, but is there any official explanation or guidance on this matter?

AquaWookie commented 3 years ago

2.2.10

I've attempted to update, changed performance tiers etc. as well. Any suggestions?

AquaWookie commented 3 years ago

According to what I see Theta has no official telegram. Can you show me where this is documented? I found about 4 scam 'official' telegrams, including people trying to pose as 'support/admins' offering bogus URLs that I won't post here that try to get you to unlock your wallet to 'fix the issue'.

I mean thanks if you're actually trying to help but the amount of scams in this space is gross, so many people will fall for them.

Beware scammers folks, they are abundant.

https://twitter.com/Theta_Network/status/1359593775983513602

Their Telegram was discontinued in February.

image

nexster commented 3 years ago

Day 4 no Tfuel.... last time I emailed theta customer support it took a week to get a response. I figured it was fishy because they are busy with Mainnet 3.0. No time for us peons. But that’s ok as long as Launch goes smoothly.

AquaWookie commented 3 years ago

Day 4 no Tfuel.... last time I emailed theta customer support it took a week to get a response. I figured it was fishy because they are busy with Mainnet 3.0. No time for us peons. But that’s ok as long as Launch goes smoothly.

Yeah that's basically what I'm hearing in the official Discord- it's a WIP and no one is coming to save us until after Main Net 3.0- that and to ignore all of these scammer cunts pushing Telegrams since they don't exist and are not official.

(Looking at you DavidGarry)

sirbrokealot commented 3 years ago

I just deleted the link too. I didn’t about it and I apologise

DAVID IS A SCAMMER @DavidGarry7654

BE ADVISED

the-quantum-rabbit commented 3 years ago

I found a lot of error message on log file. Is this normal?

[2021-03-11 18:10:38.112] [error] [2021-03-11 18:10:38] WARN [ledger] validateSourceAdvanced failed on 0xAE17b59b2587D01AFa10361b5045B14DB627B2E8: Result{code:100002, message:ValidateInputAdvanced: Got 4, expected 5. (acc.seq=4)} [2021-03-11 18:10:38.112] [info] [2021-03-11 18:10:38] WARN [ledger] validateSourceAdvanced failed on 0xAE17b59b2587D01AFa10361b5045B14DB627B2E8: Result{code:100002, message:ValidateInputAdvanced: Got 4, expected 5. (acc.seq=4)}

After much deleting database and snapshots and node restarts I finally fixed this issue in Ubuntu Linux by upgrading to the 2.2.0 theta process version from 2.1.0 by the process outlined in the manual.

curl -k --output bin/theta curl -k 'https://mainnet-data.thetatoken.org/binary?os=linux&name=theta' curl -k --output bin/thetacli curl -k 'https://mainnet-data.thetatoken.org/binary?os=linux&name=thetacli' chmod +x bin/theta chmod +x bin/thetacli

and restarting the node.

Maybe just lucky ;-)

As other people who have commented here the issue can happen with 2.2.0 and I do still see the messages, though less, and there is no loop with not getting sync'd. I finally get to a sync'd state OK.

So I conclude that the warning messages are normal but there is still some issue if people have to fiddle like we have to get our nodes up and running. This could impact the robustness of the net.

THERE SHOULD BE A DOCUMENTED RECOVERY PROCEDURE HERE, MAINTAINING STAKED THETA.

nexster commented 3 years ago

Day 5. No errors, warnings persist. No Tfuel.

sirbrokealot commented 3 years ago

Will try this now https://www.theta.tv/video/vid2sgb6k2fbnfjj9tk

edit: It it still completely broken. Damn this is f''''' up! I just want to support it. Don't even care about the tfuel money much. Get it together plz

nexster commented 3 years ago

Day 6. No Tfuel. Gonna give this a good 3-4 weeks before I do something. Node is showing up as active so maybe it’s just not getting selected for blocks.

nexster commented 3 years ago

Day 7 Tfuel no Tfuel yet.

Dahveed77 commented 3 years ago

Just dropping in to say that I have the exact same problem, with the same errors in my log.

I have been happily earning Tfuel for a couple months now, only once did I have a synching problem which I resolved by resetting, reinstalling etc...

Now I keep earning 0 Tfuel since a few days ago, which is disappointing. Nothing I try seems to change anything, including changing tiers, deleting all files and reinstalling, restaking node, resetting peers, dancing naked while sacrificing children to c'thulu, etc.

Some sort of "official" answer would be reassuring.

nexster commented 3 years ago

Minimum payout is now 12 Tfuel. I will start worrying after 14 days for my small stake. Just updating daily to keep thread alive.

nexster commented 3 years ago

Theta does not have a telegram.

phyx1u5 commented 3 years ago

Just dropping in to say that I have the exact same problem, with the same errors in my log.

I have been happily earning Tfuel for a couple months now, only once did I have a synching problem which I resolved by resetting, reinstalling etc...

Now I keep earning 0 Tfuel since a few days ago, which is disappointing. Nothing I try seems to change anything, including changing tiers, deleting all files and reinstalling, restaking node, resetting peers, dancing naked while sacrificing children to c'thulu, etc.

Some sort of "official" answer would be reassuring.

I'm having the exact same problem. would be good to get an official response. from what I can gather there is now a minimum threshold to payout so I'll keep waiting for a week or so and see if I get anything.

nexster commented 3 years ago

Day 8 no Tfuel. My stake should make 12 every 10 days. Crossing my fingers.

Kuehn commented 3 years ago

Update, after I earned Tfuel five days ago. The errors persist and I haven't received any since then, however I agree it's probably just the time to be patient as it appears that the rewards are less often but in bigger amounts as my reward last time was 12 tFuel. Good luck for everyone struggling with this issue as well. @sirbrokealot thanks for sharing the video that was very useful and I recommend everyone at least saving their keys off (as they demonstrate in the demo). I agree this is an awesome project and hope they give more support and hopefully a bit after the mainnet 3.0 launch we will get further clarification for our issues.

sirbrokealot commented 3 years ago

@nexster thank you for showing us the "doing" nothing "don't change a running system" side of this issue! keep it on! They don't have Telegram anymore. I REPEAT: THETA DOES NOT HAVE TELEGRAM. But they have an official discord. @nexter have you tried the official discord @AquaWookie posted the channel link above.

@Dahveed77 I feel you. I also did change tiers, delete peers, delete database, reinstall several times with backup key insert (like they guy showed in the vid above).

Today I will finally be able to restake. This time I will carefully be downloading the most recent software. Stake via different browser. Install without crash or reboot. Choosing not the system drive as database storage. Then I will also dance naked while sacrificing children, pets and my left pinky to c'thulu.

I/we are running out of options so I urge you all to write this issue to them in discord.

This goes out to any Theta Guardian Node dev or representative: Should we just let it run buggy until you fix it via update or did we really mess up something?