search

thetatoken / guardian-mainnet-guide

Guide for setting up the Guardian Node for the mainnet
https://docs.thetatoken.org/
147 stars 34 forks source link

Guardian Node is taking a long time to sync to the Theta Network #28

Closed clutchnshift closed 2 years ago

clutchnshift commented 3 years ago

Hello,

I had setup a Guardian Node on my PC since September 2020, running Windows 10 Pro. Just before Thanksgiving, the Guardian Node was pulling a lot of resources and my PC was locking up. I let a few weeks pass and then attempted several times to start up the Guardian Node, no luck. The GN is taking a long time to sync. It does the snapshot quickly, no change, but the actual sync-to-network part will start off at about approximately 95.5% and it will take about a day to get to approximately 95.6%. At this rate, it will take over a month to sync, assuming it remains constant. Not only that, but if I allow the GN to run (attempt to sync) for more than 24 hours, it will really slow down the PC, I would need to reboot, especially by the 36th hour...never syncing to the network.

Since I was able to run the GN in September with a sync time of about 1 hour, give or take, why is it taking longer now? My PC is not optimal, since it locks up pretty good, but can someone explain why the Theta Network is harder to sync to?

Did the network get bogged down because of the additional Guardian Nodes (reaching 2,200+)?

I would figure that a solid machine wouldn't lock up at all, may not even flinch or skip a bit, but with a slow machine you can notice a difference between September and December.

In late September we were at ~50.5% total staked Theta, with ~1,380 Guardian Nodes.

On 14NOV20, we were at 52.35% total staked Theta, with 1,472 Guardian Nodes.

Right now we are at 55.5% total staked Theta, with 2,264 Guardian Nodes.

clutchnshift commented 3 years ago

17APR21, my Guardian Node (GN) used up the resources from my PC, but good. It had been running for approximately 10 days, without a reboot, and on the 9th day the GN was down momentarily while it re-synced, just missed about 6 blocks, so it was quick. However, on the 10th day up, about 24 hours later, I happened to check guardianmonitor.io for a quick look at my 30DMB (Missed Blocks in the last 30 days) and found my status as ONLINE/UNSTABLE. I went to my PC and took a look at my GN, found it holding 18 PEERS (normally runs on 24/25 peers) and I realized that the GN was going down. Root cause? The PC had froze and I had to do a HARD RESET (hold the power button down till the PC shuts down, then restart).

I had seen this happen to me once before, exact same situation, computer freezes, PEERS drop, then I can't re-sync the GN, it does all of the following steps, except it gets stuck at STEP 5, currently hodling to 99.70% (give or take 1%).

image

Normally, it takes my GN setup approximately 75 minutes from Start to Synced.

I have heard that 99.99% can still get stuck, not syncing to the network, but a good rule of thumb is 99.95% and above is a good sign that your GN will sync, less than that and you should consider "delete node database", to start.

But, in my case now, I have tried the following and still no change from 99.70%

  1. Delete Node Database (multiple times in different performance tiers)
  2. Reverting to Standard (performance tier)
  3. Returning to Platinum (performance tier)
  4. Restarting my PC (yet again)
  5. Resetting my Modem (no help either)

The only thing I haven't tried yet is MANUALLY deleting my GN's database info, but the last time I did that I had to unstake and restake to the fresh re-installation of the GN...I knew no other way at the time (or now).

I want to avoid that, I'll try one more thing before I go nuclear on my GN...I'll post my results.

Going on personal experience here, DON'T LET YOUR PC FREEZE (lock-up), don't run it hard by using Platimum unless you are OK with doing frequent reboots, less frequent reboots for Standard. Freezing your PC it will cause problems that apparently a simple "Delete Node Database" will not fix. Know your machine's capabilities and if needs it, reboot it on a regular basis - DON'T LET IT FREEZE!**** If you have a super-solid machine ($$$), this may not even be an issue for you to worry about.

If you just started the Guardian Node (GN), expect the following...

  1. Downloading Snapshot (0-100% in about a minute)

  2. Starting Node...(quick)

  3. Validating Snapshot (this runs around 30 mins, give or take, percentage increments by 5% at a time).

  4. Loading Snapshot (this runs around 15 mins, give or take, percentage increments by 5% at a time).

  5. Syncing Theta Network (this runs around 15-30 mins, give or take) * This is where people’s GNs can really get stuck or just take too long, if so “delete node database” and monitor the GN.

NOTE: A successful action to “delete node database” during these 5 steps, will prompt you with a small window: "This will cause your guardian node to re-sync the Theta Network and will result in downtime of your guardian node." In short, you will stop earning TFUEL while your GN restarts and re-syncs (so you will be down about 75 minutes, assuming it comes back up on the first try).

image

clutchnshift commented 3 years ago

Alrighty then, here's the short version - my GN is back up & running! It took exactly 60 minutes from Double-Click to Sync!

image

Here's how I recovered WITHOUT having to MANUALLY delete any files/folders and then re-installing the GN software, which would start you off from scratch (like a clean, first-time install).

  1. Uninstall the Theta Guardian Node program (in Windows, right-click on the GN shortcut and left-click on Uninstall) * You can also use the Uninstall features from the Control Panel (in Windows), so long as you let the machine do the uninstalling and you don't go around deleting files/folders manually. image

  2. Reboot the machine (after uninstalling the GN). image

  3. Download/Install the Theta Guardian Node software from thetatoken.org (to make sure you have the latest software version). image

  4. Run the Theta Guardian Node (as soon as you install it, why wait?!). image

  5. In my case, it took exactly 60 minutes to have my GN back online. image

In the last ~2 days I missed about 264 blocks, that means I have to run minimal downtime for the next 30 days so that I can come close to SILVER or at the very least BRONZE status. There's no way I can get to ELITE status with my current machine, I need to reboot it about once a week to keep it from crashing on me, like mentioned below.

For some of you, the best move is to join Gpool and the like, stake your THETA using their servers, they're always running 100% on the 30DMB, which means ELITE status (can't miss a single block in 30 days). Checkout https://guardianmonitor.io/ your GN should be there.

I hope this helps, even me for future reference.

P.S. One last thing to mention regarding this "issue". It happened to me when I could have sworn I was on the GN version 2.2.10 (I think), then we are now on 2.2.26....if I remember correctly, the last time this happened to me back in early March, when I ended up manually deleting the GN, there had also been a software upgrade. What am I getting at? Perhaps it was coincidence that my PC froze and that the software update happened the same day??? Or perhaps that's why my PC froze, because the software needed updating??? Anyway, just follow the steps above, it's essentially a simple software re-installation.

17APR21, my Guardian Node (GN) used up the resources from my PC, but good. It had been running for approximately 10 days, without a reboot, and on the 9th day the GN was down momentarily while it re-synced, just missed about 6 blocks, so it was quick. However, on the 10th day up, about 24 hours later, I happened to check guardianmonitor.io for a quick look at my 30DMB (Missed Blocks in the last 30 days) and found my status as ONLINE/UNSTABLE. I went to my PC and took a look at my GN, found it holding 18 PEERS (normally runs on 24/25 peers) and I realized that the GN was going down. Root cause? The PC had froze and I had to do a HARD RESET (hold the power button down till the PC shuts down, then restart).

I had seen this happen to me once before, exact same situation, computer freezes, PEERS drop, then I can't re-sync the GN, it does all of the following steps, except it gets stuck at STEP 5, currently hodling to 99.70% (give or take 1%).

image

Normally, it takes my GN setup approximately 75 minutes from Start to Synced.

I have heard that 99.99% can still get stuck, not syncing to the network, but a good rule of thumb is 99.95% and above is a good sign that your GN will sync, less than that and you should consider "delete node database", to start.

But, in my case now, I have tried the following and still no change from 99.70%

  1. Delete Node Database (multiple times in different performance tiers)
  2. Reverting to Standard (performance tier)
  3. Returning to Platinum (performance tier)
  4. Restarting my PC (yet again)
  5. Resetting my Modem (no help either)

The only thing I haven't tried yet is MANUALLY deleting my GN's database info, but the last time I did that I had to unstake and restake to the fresh re-installation of the GN...I knew no other way at the time (or now).

I want to avoid that, I'll try one more thing before I go nuclear on my GN...I'll post my results.

Going on personal experience here, DON'T LET YOUR PC FREEZE (lock-up), don't run it hard by using Platimum unless you are OK with doing frequent reboots, less frequent reboots for Standard. Freezing your PC it will cause problems that apparently a simple "Delete Node Database" will not fix. Know your machine's capabilities and if needs it, reboot it on a regular basis - DON'T LET IT FREEZE!**** If you have a super-solid machine ($$$), this may not even be an issue for you to worry about.

If you just started the Guardian Node (GN), expect the following...

  1. Downloading Snapshot (0-100% in about a minute)
  2. Starting Node...(quick)
  3. Validating Snapshot (this runs around 30 mins, give or take, percentage increments by 5% at a time).
  4. Loading Snapshot (this runs around 15 mins, give or take, percentage increments by 5% at a time).
  5. Syncing Theta Network (this runs around 15-30 mins, give or take) * This is where people’s GNs can really get stuck or just take too long, if so “delete node database” and monitor the GN.

NOTE: A successful action to “delete node database” during these 5 steps, will prompt you with a small window: "This will cause your guardian node to re-sync the Theta Network and will result in downtime of your guardian node." In short, you will stop earning TFUEL while your GN restarts and re-syncs (so you will be down about 75 minutes, assuming it comes back up on the first try).

image

RandallRay commented 3 years ago

Thanks for the detailed explanation.About six weeks ago I read about THETA.  I was hooked.After messing around with the Guardian Node installation, I became less enthusiastic.  Even more so when I allowed myself to be scammed out of 1,400 THETA.  I'm back on the GN, a wiser and more cautious user.  I'm prepared to endure more troubles as this very exciting, world-changing solution becomes mainstream.I'm saving your email.Thanks again. On Tuesday, April 20, 2021, 04:25:37 PM PDT, clutchnshift @.***> wrote:

Alrighty then, here's the short version - my GN is back up & running! It took exactly 60 minutes from Double-Click to Sync!

Here's how I recovered WITHOUT having to MANUALLY delete any files/folders and then re-installing the GN software, which would start you off from scratch (like a clean, first-time install).

In the last ~2 days I missed about 264 blocks, that means I have to run minimal downtime for the next 30 days so that I can come close to SILVER or at the very least BRONZE status. There's no way I can get to ELITE status with my current machine, I need to reboot it about once a week to keep it from crashing on me, like mentioned below.

For some of you, the best move is to join Gpool and the like, stake your THETA using their servers, they're always running 100% on the 30DMB, which means ELITE status (can't miss a single block in 30 days). Checkout https://guardianmonitor.io/ your GN should be there.

I hope this helps, even me for future reference.

P.S. One last thing to mention regarding this "issue". It happened to me when I could have sworn I was on the GN version 2.2.10 (I think), then we are now on 2.2.26....if I remember correctly, the last time this happened to me back in early March, when I ended up manually deleting the GN, there had also been a software upgrade. What am I getting at? Perhaps it was coincidence that my PC froze and that the software update happened the same day??? Or perhaps that's why my PC froze, because the software needed updating??? Anyway, just follow the steps above, it's essentially a simple software re-installation.

17APR21, my Guardian Node (GN) used up the resources from my PC, but good. It had been running for approximately 10 days, without a reboot, and on the 9th day the GN was down momentarily while it re-synced, just missed about 6 blocks, so it was quick. However, on the 10th day up, about 24 hours later, I happened to check guardianmonitor.io for a quick look at my 30DMB (Missed Blocks in the last 30 days) and found my status as ONLINE/UNSTABLE. I went to my PC and took a look at my GN, found it holding 18 PEERS (normally runs on 24/25 peers) and I realized that the GN was going down. Root cause? The PC had froze and I had to do a HARD RESET (hold the power button down till the PC shuts down, then restart).

I had seen this happen to me once before, exact same situation, computer freezes, PEERS drop, then I can't re-sync the GN, it does all of the following steps, except it gets stuck at STEP 5, currently hodling to 99.70% (give or take 1%).

Normally, it takes my GN setup approximately 75 minutes from Start to Synced.

I have heard that 99.99% can still get stuck, not syncing to the network, but a good rule of thumb is 99.95% and above is a good sign that your GN will sync, less than that and you should consider "delete node database", to start.

But, in my case now, I have tried the following and still no change from 99.70%

The only thing I haven't tried yet is MANUALLY deleting my GN's database info, but the last time I did that I had to unstake and restake to the fresh re-installation of the GN...I knew no other way at the time (or now).

I want to avoid that, I'll try one more thing before I go nuclear on my GN...I'll post my results.

Going on personal experience here, DON'T LET YOUR PC FREEZE (lock-up), don't run it hard by using Platimum unless you are OK with doing frequent reboots, less frequent reboots for Standard. Freezing your PC it will cause problems that apparently a simple "Delete Node Database" will not fix. Know your machine's capabilities and if needs it, reboot it on a regular basis - DON'T LET IT FREEZE! If you have a super-solid machine ($$$), this may not even be an issue for you to worry about.

If you just started the Guardian Node (GN), expect the following...

NOTE: A successful action to “delete node database” during these 5 steps, will prompt you with a small window: "This will cause your guardian node to re-sync the Theta Network and will result in downtime of your guardian node." In short, you will stop earning TFUEL while your GN restarts and re-syncs (so you will be down about 75 minutes, assuming it comes back up on the first try).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.

deanotuck commented 3 years ago

Just an FYI...I had tried everything to get the Theta Guardian Node to Sync and Run and had pretty much given up but I tried one last thing. When I changed the priority on the three Executables running to Realtime, it ran and synced quickly. To change the priority:

  1. Go to the Task Manager
  2. Click on the Details tab
  3. Look for the Theta Guardian Node executables
  4. Right click on the executable and select Set Priority
  5. Select Realtime and confirm the change

You can try lower than Realtime but the Realtime worked amazingly.

Hope this helps!

clutchnshift commented 2 years ago

A lot of us were still sporting older computer systems when we started running Guardian Nodes. Last August, after a year of slow-syncing-torture, I replaced my old computer (HDD) with a new computer (SSD). THETA's requirements are to run on a Solid State Drive (SSD), among a couple more hardware requirements (read up on them). The difference between operating a HDD vs a SSD was night and day, the GN syncs up fast and along with a halfway decent amount of memory (32GB), I have had ZERO computer issues in the last 12 months when running the GN, EN and THETA TV all at once, and for extended periods of time. I typically just run the GN and EN simultaneously and once in a while I'll tune into THETA TV. Just make sure that you are meeting THETA's minimum hardware requirements, and apparently you should be good.

Processor 11th Gen Intel(R) Core(TM) i9-11900F @ 2.50GHz 2.50 GHz Installed RAM 32.0 GB Storage Micron 2300 1.0 TB SSD with NVMe System type 64-bit operating system, x64-based processor