search

sentora / sentora-core

Sentora is a web hosting control panel written in PHP for *NIX
GNU General Public License v3.0
648 stars 446 forks source link

Error BW calculation break daemon #121

Closed MBlagui closed 9 years ago

MBlagui commented 9 years ago

Hi,

Generating bandwidth.. PHP Fatal error: Allowed memory size of 304087040 bytes exhausted (tried to allocate 645695949 bytes) in /etc/zpanel/panel/dryden/sys/bandwidth.class.php on line 24

Notice I increased memory to 290M and still doing more. This is inflating too much I think.

I have a high traffic website with much logs and seem current way of calculating BW trigger the issues.

M B

jd1pinoy commented 9 years ago

I have also noticed this in Ubuntu with bunch of domains and logs on it.. it bells my Putty on large logs. I'll test and see Daemon in CentOS..

MBlagui commented 9 years ago

Config Centos 6.5 here so PHP 5.3. I started increasing ram but noticed it's requirement more and more and not fixing it. seem BW calculation is stored in php process and big bang once it had a lot of data to process.

allebb commented 9 years ago

Previously we had added a separate Apache log file purely for bandwidth calculation of which once read by the daemon would then get truncated to ensure that it never grows too large!

So I'm wondering what's happening here - maybe someone has now removed this feature which will be very frustrating!

On 12 February 2015 at 14:32, MBlagui notifications@github.com wrote:

Config Centos 6.5 here so PHP 5.3. I started increasing ram but noticed it's requirement more and more and not fixing it. seem BW calculation is stored in php process and big bang once it had a lot of data to process.

— Reply to this email directly or view it on GitHub https://github.com/sentora/sentora-core/issues/121#issuecomment-74079639 .

5050 commented 9 years ago

The most probably cause is because the routine CalculateFromApacheLog() loads the full file to extract bandwidth data.

The whole bandwidth computation could be a lot of improved by redefining the log format used in apache config files to write only the bandwidth value instead of the whole request, and then to adapt the computation routine to the new format generated.

Until such change is done, it is possible to do a small change to the routine CalculateFromApacheLog() to read the file line by line instead of to read it fully and then strip and compute each line. As I already rewritten completely this routine a long time ago (july 2013, but conserved the historical readfile), I just pushed this change, checked on my own server.

MBlagui commented 9 years ago

Patched an old zpanel install too & that was AWESOME Daemon faster.

Even lowered my ram to 190 MB and got it... Will keep it under watch

allebb commented 9 years ago

My point was is that the bandwidth log file should be read every 5 minutes, extracted the bandwidth and then the file is 'reset' so I'm surprised that such a large file exists and can be created in only a few minutes. (if your running a site with that much traffic I wouldn't recommend using Sentora anyway :))

On 12 February 2015 at 15:40, MBlagui notifications@github.com wrote:

Patched an old zpanel install too & that was AWESOME Daemon faster.

— Reply to this email directly or view it on GitHub https://github.com/sentora/sentora-core/issues/121#issuecomment-74091709 .

MBlagui commented 9 years ago

Why bobby not running sentora? As apache & mysql handle the load using a dedicated server.

Your point here bobby is just like you say centos won't hold that load. As on web side we almost use the default stack and only add some security sandboxing. PHP & Apache packages are the OS default same like the config. I have some time spikes over 400-500 pages views/min and that could kill the BW & then daemon will be stuck and unable to run.

Also it's a WP site having internal cache that is very effective and I was able to handle over 2000 live users at 50% ressources and while some crappy WP plugins enabled ( using 80% of the ressources!).

5050 commented 9 years ago

I also have a website with huge bandwidth served. a few monthes ago it goes over 9 000 GB/week (35 000 GB/month). Because clients get files by 10 MB chunks, it creates very large bandwidth files, even per 5 min, which have their size uselessly increased by logging IP, time, request and http result that are not required to compute bandwidth. It explains why I though to reduce them. Happily, the bandwidth is lower now.

But whatever it is, I do not see any reason to not use Sentora on this server !

allebb commented 9 years ago

What I'm trying to say guys is that just because the bandwidth amount is large (Pascal I assume your site serves that much data due to flight simulator ZIP file downloads etc?) it doesn't mean the bandwidth file will be large... If the bandwidth file reaches 10 MB in a 5 minute window then that isn't really a problem, as long as the PHP daemon has greater than 10 MB allocated to it of which we force set it at 256MB I believe by default. (after checking that, I see that has also been removed... simply just need to increase the amount of memory that the daemon.php script can use!)

So my point being Me.B is that is to generate a text file that is greater than 256MB (a total guess but lets say 1,000,000 resource hits?) in a window less than 5 minutes is pretty heavy for any modest website - I'm not saying the TOTAL download bandwidth in that time just the bandwidth log file size! So with that in mind personally if I've got a site generating that much traffic (hits wise) I'd run it on Nginx and have it 'dedicated' for serving that site, I wouldn't have it on a shared hosting server.

If you guys don't see what I'm saying then I guess it's the language barrier and I'll therefore have to give up and let you guys do what you think you think and prove me wrong ;)

On 12 February 2015 at 16:21, Pascal Peyremorte notifications@github.com wrote:

I also have a website with huge bandwidth served. a few monthes ago it goes over 9 000 GB/week (35 000 GB/month). Because clients get files by 10 MB chunks, it create very large bandwidth files, even per 5 min, which have their size uselessly increased by logging IP, time, request and http result that are not required to compute bandwidth. It explain why i planned to reduce them. Happily, the bandwidth is lower now, but whatever it is, I do not see any reason to not use Sentora on this server !

— Reply to this email directly or view it on GitHub https://github.com/sentora/sentora-core/issues/121#issuecomment-74100201 .

MBlagui commented 9 years ago

@bobsta63 website running on dedicated with only few corporate websites aside. It's not running a real shared configuration. I have 2 dedicated servers running like that one with only a website and another a big website and half dozen corporate websites that never use resources. The hits happen so quickly in spikes, as they are offer news & love boosting them on facebook and that can be a kill. Most of the time resource use don't spike above 5% but I need to sustain spikes that can reach above 50% cpu use(Mysql big problem here). My issue had been more with MySQL and WP plugins rather than apache it self & I have 32 GB ram available so would not be an issue here over nginx ram footprint. And I'm talking here about only those special cases as I have other "normal" shared hosting server with even less resources and running fine.