search

Chia-Network / chia-blockchain

Chia blockchain python implementation (full node, farmer, harvester, timelord, and wallet)
Apache License 2.0
10.81k stars 2.02k forks source link

[BUG] Since 1.1.5 Consistent Syncing Issues (more details inside) [Repost from GUI page #402] #5464

Closed Brittle73 closed 3 years ago

Brittle73 commented 3 years ago

Reposting this here since it seems to get more traction that the blockchain-gui page. Just copying and pasting as is.

Hello all,

I started plotting in early April and worked out a bunch of user kinks trying to learn the system, but I got it all going and cleared any errors. Until I updated to 1.1.5, which is weird because supposedly it fixes a lot of issues for other people.

Almost immediately after updating I started having syncing issues. I have not had enough time to get really into managing through PowerShell and primarily use the GUI for now. Basically it looked like it was consistently about an hour behind, sometimes more. I started to notice that it primarily looked like it was having issues whilst plotting. So I finished my current plot and set out to troubleshoot. I:

manually added peer node.chia.net:8444 restarted GUI/virtual machine many times (this sometimes got it to sync briefly, but then went not synced) set up port-forwarding (more on this in a moment) killed services/apps and deleted the database and let it rebuild full restart - uninstall, reboot, reinstall, resync (which took all weekend...)

I have been super busy, so I haven't been able to sit and watch it, but as far as I can tell it had no problems syncing and farming up until this morning when I made a little space on my Synology to throw a few more plots down and started plotting. Not synced, again. No farming. Essentially just burning power for nothing, so not super green. I have seen very similar reports from users on 1.1.4 and 1.1.5 all over the internet. Before I started plotting again it dropped a couple of times, but if I rebooted or just restarted the GUI (letting it close all the way first) it seemed to come back. The port forwarding honestly seems like it may have done more bad than good. I disabled the rule in my USG and no sync issues since, so that could have been wrong, but I had one of my senior engineers check my work and mirror his and he said it looks right, so I am leaving that as it is for now.

So I basically have to abandon hope of winning any Chia at all if I want to add more plots to my tiny farm at this point and I am not sure what to do with it. I've done everything I have read online related to the errors I was receiving. Since starting plotting this morning I have no errors in the log, but some warnings:

2021-05-18T12:24:25.911 full_node chia.full_node.full_node: WARNING Invalid response for slot None

Prior to starting to plot all I had was INFO logs. Anything I am missing here?

Thank you!

W10 VM Chia 1.1.5 mostly GUI Static IP, port-forwarded set up, but abandoned OS on datastore, temp on dedicated 2.5" SSD (only thing using it), and a passthrough USB 3.0 portable 4TB HDD / networked Synology share for plots (not getting any plotting errors and all are connected and farm when syncing)

Edit 1 - added specs Edit 2 - added most recent log file (...no errors OR warnings now): debug.log Edit 3 - typed this Edit 4 - I also disabled AV and Windows firewall to test Edit 5 - I have since reinitiated port-forwarding to see if it goes better... It randomly started syncing again, but for the last week or two it'll do that eventually and then get up to about an hour behind and get dunked again. I am at 91% with my current plot and will not start another one to see if it catches all the way up. < UPDATE - this appeared to gunk things up again... I finished that last plot and it still stayed about an hour to 1.5 hours behind. Finished plotting, disabled the port-forward, and rebooted and am currently synced, but need to plot more and scared to start.

debug.log

sparkyyy277 commented 3 years ago

I have the same issue. Gui hasn't synced for 4 days now. Updated to 1.1.5 still having the same issue. Wish they would get it fixed!

joelrb commented 3 years ago

Add you comment to my request to just decouple plotting from the rest so we can restart Chia anytime without losing plotting in progress.

https://github.com/Chia-Network/chia-blockchain/discussions/5467

Brittle73 commented 3 years ago

Update - I stubbornly started plotting again and sure enough, stopped syncing. I haven't had this many issues with this since like 1.1.0 or something.

I haven't researched enough to find out how it works, but I may have to share out the plot drive and add a second VM just to plot to get around this... it's getting frustrating for sure. If anyone has any easy guides/info on this please comment here!

What's killing me is the no errors/warnings, but clearly broken part...

Brittle73 commented 3 years ago

@joelrb honestly if that is a concern then I would just plot from PowerShell. It's not so much needing to restart the GUI without losing plotting that bugs me, it's not knowing if it's syncing or farming right WHILE plotting.

joelrb commented 3 years ago

@joelrb honestly if that is a concern then I would just plot from PowerShell. It's not so much needing to restart the GUI without losing plotting that bugs me, it's not knowing if it's syncing or farming right WHILE plotting.

You're right. It's a pain not syncing. At least I see in the logs it's a websockets exception. So, a full restart is needed but I can't just yet as it's 76% plotting. I'll just plot next time from powershell.

dvd101x commented 3 years ago

I have been in the project since last year and just recently at 1.1.3, 1.1.4 and 1.1.5 I’ve been having constant syncing issues. I enabled port forwarding for 8444 on the router since the betas and it doesn’t seem to be working at all. Introducers also do nothing and I have to manually add peers, but they seem to disconnect my peer as soon as I connect to them.

Brittle73 commented 3 years ago

@joelrb I know that frustration :/. Just gotta let it finish and keep an eye on it and not queue more. You could also set up a scheduled task if you're using Windows and can roughly estimate when it might be done, give it a couple of hours' headroom, and tell it to restart on its own. Chia won't fully log into the wallet even if you put it in shell:startup, but it'll at least launch and be ready for you to click it.

@dvd101x so this is most of what I do at work as a systems engineer, albeit not at a crazy high level yet... so it's very, very strange to me when the port-forwarding seems to be causing more syncing issues than it is solving, although not by much. I find myself having to re-add node.chia.net frequently throughout the day as well, although I have admittedly not checked to see what IP that resolves to and if it's in the list after some time. My hope is that the amount of reports I am finding on this and relative dev silence now means that they're working hard on the back end to resolve some of these issues. I only got in in early April, maybe late March and I haven't had this many issues since I first started, but I also feel like I was generally updating the client more often than this as well. I have faith in them. Although at this point I am putting way too much time and effort into something with very risky ROI/very little chance with the limited amount of space I have access to, just to get it working right. No, I am not quitting yet.

Anyway, UDPATE: confirmed, since I started plotting again my client lost sync for the last few hours. I have stopped plotting and changed the port-forwarding and reset and get roughly the same thing every time. I am going to try to get through the next five plots that I need to finish (and it might even be time to dig deeper into how to work with this in PowerShell more) and then just leave it after it syncs and see if it stays that way.

Hoping y'all have a relative solution. Thank you.

Newest logs: debug.log (notice: not a single warning or error still, despite not syncing...) Screenshot of last sync (that's EST and it is currently 12:27am the following day, I started plotting right around 9:52pm): image

berlef80 commented 3 years ago

I am having the same issue since yesterday. I haven't had a problem since started farming in March, but the past two days have been kicked off constantly and having synching issues. I have 40 nodes connected so its not an issue with that. Most of them get stuck too so I kick the ones off so i have a good number with the current height. There are at least 10 other people on Key base with this issue. I really think this is an issue Chia needs to address as many of us are complaining of the same thing...

joelrb commented 3 years ago

@joelrb I know that frustration :/. Just gotta let it finish and keep an eye on it and not queue more. You could also set up a scheduled task if you're using Windows and can roughly estimate when it might be done, give it a couple of hours' headroom, and tell it to restart on its own. Chia won't fully log into the wallet even if you put it in shell:startup, but it'll at least launch and be ready for you to click it.

@dvd101x so this is most of what I do at work as a systems engineer, albeit not at a crazy high level yet... so it's very, very strange to me when the port-forwarding seems to be causing more syncing issues than it is solving, although not by much. I find myself having to re-add node.chia.net frequently throughout the day as well, although I have admittedly not checked to see what IP that resolves to and if it's in the list after some time. My hope is that the amount of reports I am finding on this and relative dev silence now means that they're working hard on the back end to resolve some of these issues. I only got in in early April, maybe late March and I haven't had this many issues since I first started, but I also feel like I was generally updating the client more often than this as well. I have faith in them. Although at this point I am putting way too much time and effort into something with very risky ROI/very little chance with the limited amount of space I have access to, just to get it working right. No, I am not quitting yet.

Anyway, UDPATE: confirmed, since I started plotting again my client lost sync for the last few hours. I have stopped plotting and changed the port-forwarding and reset and get roughly the same thing every time. I am going to try to get through the next five plots that I need to finish (and it might even be time to dig deeper into how to work with this in PowerShell more) and then just leave it after it syncs and see if it stays that way.

Hoping y'all have a relative solution. Thank you.

Newest logs: debug.log (notice: not a single warning or error still, despite not syncing...) Screenshot of last sync (that's EST and it is currently 12:27am the following day, I started plotting right around 9:52pm): image

I moved from windows to macs. I tried windows but per the documentation plotting is slower by 5-10% than mac and linux. I have a mac-mini 2018 and iMac 2013. Indeed it's faster to plot in mac. My iMac is fully sync and the mac mini is the one that has sync problem. Mind you both are on the same network but it's the mac mini got affected by router reboot. I've setup a network of farmer/node and harvesters. Too bad the Mac mini is the farmer and also plotting there.

mariano54 commented 3 years ago

We are looking into this issue, it's important to us to fix the performance and syncing issues. We already have 2 improvements coming in the next release (1.1.6) but we are going to keep looking.

kennylbj commented 3 years ago

@mariano54 when will 1.1.6 be released?

runechronos commented 3 years ago

this is really top priority i can't farm since almost one week with 1.1.5... had no issue before. And I get the same problem with gui or using powershell. Yesterday it was trying to sync but it never catched up with the last block. And today it is just not moving (I opened the 8444->8555 port on my router, and on windows defender) adde peers with "chia show -a" for chiadb.net:8444, node-eu.chia.net:8444 and also the introducer-eu.chia.net:8444 and it seems to be worse. It's just unnaceptable. U have to do something :s

Did someone tried to revert back to 1.1.4 or 1.1.3 ?

BigFactHunt commented 3 years ago

Agree this is a critical bug , since moving to 1.1.5 I have these sync issues, not had previously on 1.1.4 , nothing else changed

zsoca76 commented 3 years ago

I'm on 1.1.5, but have several syncing issues on Win10. I've lost sync at 6:29 CET today. Tried several solutions, ports are open, still no sync... I haven't tried to relaunch the application as I'm plotting continuously, please find a solution asap.

nmmmnu commented 3 years ago

I have two nodes, running on dedicated machines. I am losing sync 2-3 times per day.

One way to fix that is to stop node, remove peer_table_node.sqlite and to start again. Another way is to manually disconnect all connected peers who are not sync (I know I don't help the networks doing so) . After I do these, sometimes it works, sometimes it does not.

when 1.1.6 will be released?

Brittle73 commented 3 years ago

@nmmmnu I can also report that simply shutting down all of Chia and re-launching, including the background processes does seem to work. I find it easier to let the GUI shut it down and then just reboot instead of trying to track down all of them in Task Manager, but either way it comes back up and appears to work fine for a little and then loses sync again. Whether the correlation is relevant or not I do not know, but I seem to do okay, maybe a little up and down, and then get bored and continue plotting and down she goes after about 30 mins to an hour. Sigh.

@mariano54 Thank you for replying, it helps a lot when we know you're working on it. I am assuming you don't have one and/or cannot answer, but for the sake of mirroring the aching question in this thread: do you know when 1.1.6 will be released.

Additionally, I considered attempting to downgrade a version of two, but haven't done it yet. Has anyone had any success trying this?

Brittle73 commented 3 years ago

Also has anyone here tried to plot and harvest on two separate machines or VMs and does that appear to help?

zsoca76 commented 3 years ago

So after 6 hours I got back the sync for an hour, but then it went out of sync again. What I can see in the peer database it's updating/refreshing itself (or at least the date of the file is showing that)since I got back the sync, but in the gui it says it’s not synced...

joelrb commented 3 years ago

Also has anyone here tried to plot and harvest on two separate machines or VMs and does that appear to help?

Yes, just follow the guide (https://github.com/Chia-Network/chia-blockchain/wiki/Farming-on-many-machines).

You have to select a machine as your main Farmer/Full Node, other machines are Harvesters. The Harvester machines do the plotting and no need to copy the plots over to the Farmer machine (you can plot also at the same time in the Farmer machine). The Harvester machine and plots will "appear" in the Farmer machine (in attempted proof as 0/5 if you have 5 plots on that Harvester along with the plots from other machines). It will also appear in Farmer machine as a node in Harvester Network.

You copy the certificate in your Farmer machine and use that to sign the certificate of your Harvester machine. You need the Farmer certificate so that all your plots will be connected to 1 wallet or Farmer node. Otherwise the plots won't be recognize by other machines of your Farmer-Harvester network. By using Farmer certificate to sign Harvester nodes you can actually copy the plots from one machine to another and still be recognized by the Farmer machine or another Harvester machine (if you move plot to this machine) as a valid plot.

I already tried all that and it works.

The guide above is easy to follow.

speeder2005 commented 3 years ago

Has been struggling on this issue for more than a week, when netspace <4EiB. Now I am still in an on/off oscillating . I am exhausted now. I think this is something on developer side to solve. They should make this app less picky on our local environment. Many years ago, with 10Mbit internet I can easily download several GBs file via the bittorrent Bram invented. Never notice a connection issue. Now with 200Mbit speed, we can hardly download the 3GB blockchain file. Make no sense to me.
During my troubleshooting, I have received a lot of suggestions from reddit, github, and youtube. Really appreciate. I think I should contribute something back to this community. If you are in the sync issue and still have some energy to burn, here is a list of tricks I collected as the treat for you. Hope you have a better luck than me!

  1. Update to the latest version.
  2. Disable upnp on plotting pc. Chia wiki has instruction.
  3. Forwarded Port 8444 on router, set windows firewall to allow 8444 inbound. go to portchecker.co to verify the port is open.
  4. Windows defender firewall with advanced security: check settings to ensure not block start_farmer and start_full_node on private networks. ensure your local network connection is set to be recognized as a private network.
  5. In config.yaml change target_outbound_peer_count to 30, target_peer_count to 300, weight_proof_timeout to 400
  6. Make sure Windows clock is auto sync with internet. Check time is the same as your cellphone.
  7. Add introducer (see Chia wiki) and keep adding fresh nodes found on https://chia.keva.app/
  8. Reboot router to see your luck
  9. Reinstall Chia to see your luck, keep the blockchain file. You don't want to resync from bottom for another week.
  10. Remove all "full node" had a height tied to them. And just add introducer-or.chia.net:8444 & chiadb.net:8444
  11. In CLI, run .\chia start introducer
  12. Stop chia node, Turn off windows clock time/zone auto update, turn back on, click sync button. Start chia node.
  13. Run chia GUI as administrator
  14. Stop chia, Delete.chia\mainnet\db\peer_table_node.sqlit , restart chia
  15. In “windows features”, turn on telenet client
  16. Close and uninstall chia on exiting computer. Try it on another computer.
  17. Pray every night and check back the next morning. :)
sparkyyy277 commented 3 years ago

I was having this issue with 1.1.4 so I updated to 1.1.5, still the same issue. Have not been able to farm since Saturday. I tried everything to correct the issue. Last night I finally reached my boiling point and deleted everything and reinstalled. It is now in the syncing process at least, but I have a lot of catching up to do... only at 200604. I would recommend holding out for a fix instead of deleting like me, I may end up stalling in the 300000's anyway.

Brittle73 commented 3 years ago

I have done most of what @speeder2005 said to try, but I think there are enough threads about this to show that this isn't a one off issue.

@sparkyyy277 I started that Friday night out of desperation and I think it barely finished Sunday late, might have been early Monday morning? Sorry dude (>.>)

Also... finally got home and remoted into the VM to find this message... this is a new one. Devs, Any ideas?

chiawut2

runechronos commented 3 years ago

It's just perfect the network will have gone x3-x5 before the problem is solved, and no one will ever win chia except chinese farm. It' s already really hard with 100-200To. So might just be lost money if you have invested in something, We will have pools soon where your money will magically disappear, until at least we see really trusty pools appearing. So it's really really messed up, and we will of course have many new bugs with "pooling plots" appearing for sure.....

nmmmnu commented 3 years ago

If this problem is not addressed in 2-3 days, I will start to believe chia is a scam coin.

No matter what Chinese do, it must not affect synchronization. They don't do DDOS, they just connect some "clients".

on 100 Mbit network, I can sync whole bitcoin chain (~600 GB) for 3 days and then never get out of sync.

bashira1980 commented 3 years ago

i am running chia on windows server 2019 and my plots started to sync and as it was about to finished it changed the status to not synced.

Plotting seems to be running fine and I can complete a plot in under 10hours.

After following some of the recommended suggestions I can not get the syncing going.

I am using gui 1.1.5

Any suggestions on fixing synching issues on windows?

alex-oden commented 3 years ago

Hi, I have installed this version of chia (beta build as I understand) and all working fine and stable with lot of connection from another peers. As I understand and see in logs - if there is some failure in connection it will not stop with sync and beginning with new connection but stays stable with existing peer connection

https://github.com/Chia-Network/chia-blockchain/actions/runs/853387813

Right now by me it working very good without any issues and Full node don't take a lot of power with CPU consumption. Already in Sync about 6 hours stable and FINALLY it sees my another local harvester...

(Also if you have not opened or forward on your router port 8444) You need to do that - here some help https://portforward.com/

jmperezbeth commented 3 years ago

Hi, I have installed this version of chia (beta build as I understand) and all working fine and stable with lot of connection from another peers. As I understand and see in logs - if there is some failure in connection it will not stop with sync and beginning with new connection but stays stable with existing peer connection

https://github.com/Chia-Network/chia-blockchain/actions/runs/853387813

Right now by me it working very good without any issues and Full node don't take a lot of power with CPU consumption. Already in Sync about 6 hours stable and FINALLY it sees my another local harvester...

(Also if you have not opened or forward on your router port 8444) You need to do that - here some help https://portforward.com/

This branch has helped me out... Node synced at last... I'm using a dated CPU and observed that a faster one mantains sync. Syncing is obviously a single threaded process right now. Also limiting number of peers helped, but I can now leave 50 peers as in default config and it catches up nicely...

Thank you very much @alex-oden for the tip!

alex-oden commented 3 years ago

Hi, I have installed this version of chia (beta build as I understand) and all working fine and stable with lot of connection from another peers. As I understand and see in logs - if there is some failure in connection it will not stop with sync and beginning with new connection but stays stable with existing peer connection https://github.com/Chia-Network/chia-blockchain/actions/runs/853387813 Right now by me it working very good without any issues and Full node don't take a lot of power with CPU consumption. Already in Sync about 6 hours stable and FINALLY it sees my another local harvester... (Also if you have not opened or forward on your router port 8444) You need to do that - here some help https://portforward.com/

This branch has helped me out... Node synced at last... I'm using a dated CPU and observed that a faster one mantains sync. Syncing is obviously a single threaded process right now. Also limiting number of peers helped, but I can now leave 50 peers as in default config and it catches up nicely...

Thank you very much @alex-oden for the tip!

You welcome! I hop it will work properly, but no warranty :)

dvd101x commented 3 years ago

@joelrb I know that frustration :/. Just gotta let it finish and keep an eye on it and not queue more. You could also set up a scheduled task if you're using Windows and can roughly estimate when it might be done, give it a couple of hours' headroom, and tell it to restart on its own. Chia won't fully log into the wallet even if you put it in shell:startup, but it'll at least launch and be ready for you to click it.

@dvd101x so this is most of what I do at work as a systems engineer, albeit not at a crazy high level yet... so it's very, very strange to me when the port-forwarding seems to be causing more syncing issues than it is solving, although not by much. I find myself having to re-add node.chia.net frequently throughout the day as well, although I have admittedly not checked to see what IP that resolves to and if it's in the list after some time. My hope is that the amount of reports I am finding on this and relative dev silence now means that they're working hard on the back end to resolve some of these issues. I only got in in early April, maybe late March and I haven't had this many issues since I first started, but I also feel like I was generally updating the client more often than this as well. I have faith in them. Although at this point I am putting way too much time and effort into something with very risky ROI/very little chance with the limited amount of space I have access to, just to get it working right. No, I am not quitting yet.

Anyway, UDPATE: confirmed, since I started plotting again my client lost sync for the last few hours. I have stopped plotting and changed the port-forwarding and reset and get roughly the same thing every time. I am going to try to get through the next five plots that I need to finish (and it might even be time to dig deeper into how to work with this in PowerShell more) and then just leave it after it syncs and see if it stays that way.

Hoping y'all have a relative solution. Thank you.

Newest logs: debug.log (notice: not a single warning or error still, despite not syncing...) Screenshot of last sync (that's EST and it is currently 12:27am the following day, I started plotting right around 9:52pm): image My issue is better now. I made a script in Power Shell that adds the introducers every hour, it didn’t disconnect overnight or during the day the few times I checked. My point about port forwarding is that I configured the router but it doesn’t seem to work. I will try the new version once a few plots are done.

kennylbj commented 3 years ago

Hey guys, since 1.1.6 is released, why not try it first to see if the issues above are fixed?

fabioyy commented 3 years ago

1.1.6 fixed for me

runechronos commented 3 years ago

1.1.6 did not change a thing for me. The node just don't move... syncing is still as slow as it was, so i never catch up to the actual block... still at 306910, i'm running behind since a bit more than 1 week and 1.1.5

alex-oden commented 3 years ago

1.1.6 did not change a thing for me. The node just don't move... syncing is still as slow as it was, so i never catch up to the actual block... still at 306910, i'm running behind since a bit more than 1 week and 1.1.5

I think the best solution for you is:

  1. Open port on your router 8444 for the connection of full nodes. Before that you need to change your pc DCHP connection to static ip.
  2. Delete DB (database) of Peer to peer connections - its must be here: C:\Users\Username.chia\mainnet\db\peer_table_node

How to change DCHP to static ip on computer Windows 10: https://www.youtube.com/watch?v=5iRp1Nug0PU How to open port on router: https://portforward.com/

and at last if nothing helps - remove completely Chia and begin with new installation... but hope it will work for you

github-actions[bot] commented 3 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

nb1980 commented 3 years ago

having same issue on my Mac mini...none of the above has resolve the sync error. have tried multiple old versions and new and still no luck being able to synch

github-actions[bot] commented 3 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

Brittle73 commented 3 years ago

Oops, sorry for the late reply, I meant to sooner.

I am happy to report that the next update did fix the issue and as of right now I have not experienced the same issue. Resolved, at least on my end. Thanks devs.

The issue right now is lack of XCH in my wallet and the expected crash being a little harder than I think most of expected. C'est la vie, nothing any of us can do about that for now.

github-actions[bot] commented 3 years ago

This issue has been flagged as stale as there has been no activity on it in 14 days. If this issue is still affecting you and in need of review, please update it to keep it open.

emlowe commented 3 years ago

Closing issue per previous comment