search

Qortal / qortal

Qortal Project Core - decentralize the world - Data storage, communications, web hosting, decentralized trading, complete infrastructure for the future blockchain based Internet
https://qortal.org
71 stars 60 forks source link

Excess Network Usage #34

Open TRM13 opened 3 years ago

TRM13 commented 3 years ago

Kubuntu 20.04, core is qortal-1.4.5-b9a0d48 (minting mode) and using vnstat 2.6 measuring network usage.

I am logging 300 MB per hour (200+ GB) per month. That seems quite excessive. Is that normal? Do we have any baseline to go by?

Thanks (MeadMan)

archived-2 commented 3 years ago

Hey, thanks for raising this. I imagine it's a symptom of the constant syncing / re-orgs that are going on right now, causing far more data to move between nodes than necessary. I'm currently working on a new approach to syncing which I hope will fix a lot of these issues.

Please can we revisit this in a couple of weeks when some new updates are out, and check that it has improved?

TRM13 commented 3 years ago

Sure I'd love to. Whenever you want. Thanks

TRM13 commented 3 years ago

Just a quick update. I've also noticed daily transmit of 2-4 GB in one hour, twice daily for several days. The times are 08:00 and 18:00 to start. Anything in the core set to go off twice a day? I have nothing else running on this system, it is dedicated to Qortal and no system side stuff is happening at those times.

No reply required. Just putting it in here so if someone notices they don't open another ticket as I think it is related.

The huge bursts of transmit traffic seemed to have stopped. I re-sync'd yesterday and I'm still looking at 250-300 GB a month but no bursts (yet lol).

archived-2 commented 3 years ago

Hey @TRM13, just wondering if you have any recent data on network usage since 1.5.2? Would be interesting to know if it has reduced at all, now that the re-orgs are much smaller.

TRM13 commented 3 years ago

I've switched to ntopng as it gives a finer resolution of traffic. I'll run a 12 hour & 24 hour log and get back to you tomorrow.

archived-2 commented 3 years ago

Thanks - much appreciated! I'm hoping it will be a lot less then the 300MB per hour we used to use...

TRM13 commented 3 years ago

Ran ntopng to gather stats for 24 hours: Sent: 4.4 GB Recd: 3.2 GB Which works out to 228 GB/Month.

I've verified with the data my ISP provides and I went from about 130 GB a month to 330 GB after I started my Qortal node. So it seems to be 200+ GB per month is about correct. This may just be inherent in the design. Low CPU but high bandwidth requirements.

archived-2 commented 3 years ago

Thanks for checking it. You're right that high bandwidth is inherent in the design (every block has a large number of candidates, so nodes must sync several "versions" of each block before finding the best one). In the long run I do expect bandwidth to decrease from where we are now, as we continue to tighten up the sync process, but clearly we're not there yet!