search

causefx / Organizr

HTPC/Homelab Services Organizer - Written in PHP
GNU General Public License v3.0
5.08k stars 289 forks source link

PHP memory limit of 128MB is insufficient #990

Closed rcdailey closed 5 years ago

rcdailey commented 5 years ago

Using the organizrtools/organizr-v2 Docker image, I got the following error repeatedly in the /config/log/nginx/error.log file:

FastCGI sent in stderr: "PHP message: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 22595624 bytes) in /config/www/Dashboard/api/functions/log-functions.php on line 30" while reading response header from upstream, client: 172.19.0.7, server: _, request: "POST /api/?v1/login HTTP/1.0", upstream: "fastcgi://127.0.0.1:9000", host: "domain.com", referrer: "https://domain.com/organizr/"

This was occurring pretty much any time a request came in through Organizr's built-in nginx server. This caused issues like "Login Error - API Connection Failed" error messages when attempting to log in. Restarting and even recreating the container didn't help. This error occurs immediately on start up.

Bumping memory_limit to 256M in /config/php/php-local.ini fixed the issue.

Whether or not this is a bug versus just a normal memory requirement for Organizr, I'd like some advice on how to analyze memory usage of PHP. Something similar to top that would allow me to see what is consuming the most of that default 128M of memory that PHP uses. If normal expected operating memory usage is supposed to be below that number, maybe there's an issue, but without any way to run diagnostics to get some more insights, I don't know how we can prove that. Any advice is greatly appreciated.

causefx commented 5 years ago

i will look into it. try switching to the ‘php-tom’ tag

causefx commented 5 years ago

php-fpm

autocorrect :)

rcdailey commented 5 years ago

If that means I have to do the fastcgi configuration myself I'll pass. Too much configuration overhead for me, and too invasive for the linuxserver letsencrypt container since I'd have to mount the www directory as a volume.

On Wed, Nov 28, 2018, 11:01 PM causefx notifications@github.com wrote:

php-fpm

autocorrect :)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/causefx/Organizr/issues/990#issuecomment-442707302, or mute the thread https://github.com/notifications/unsubscribe-auth/ABr6dvuRV-1CkCExhijSu2LCU98XBiMCks5uz2pCgaJpZM4Y476A .

causefx commented 5 years ago

the tag does everything.

Roxedus commented 5 years ago

how big is organizrs own log? its stored in the db location

rcdailey commented 5 years ago

@si0972

Result of ls -l in www/db:

   98304 Oct 25 19:40 'organizrdb[2018-10-26_00-40-57][2.0.0-beta-800].bak.db'
  139264 Nov 28 22:23  organizrdb.db
    5637 Nov 28 22:23  organizrLoginLog.json
 1760943 Nov 29 08:39  organizrLog.json

@causefx: Sorry, can you clarify please? What do you mean "does everything"? How is php-fpm tag different from develop-v2. Is it also version 2? How do you feel this will address the issue? I'd like your help learning a bit more about this. Thanks!!

causefx commented 5 years ago

sorry. the tag i was speaking about is the docker container tag. instead of using latest, use php-fpm. that should fix it as that contains custom values for php setup.

Sent from my iPhone

On Nov 29, 2018, at 7:41 AM, Robert Dailey notifications@github.com wrote:

@si0972

Result of ls -l in www/db:

98304 Oct 25 19:40 'organizrdb[2018-10-26_00-40-57][2.0.0-beta-800].bak.db' 139264 Nov 28 22:23 organizrdb.db 5637 Nov 28 22:23 organizrLoginLog.json 1760943 Nov 29 08:39 organizrLog.json @causefx: Sorry, can you clarify please? What do you mean "does everything"? How is php-fpm tag different from develop-v2. Is it also version 2? How do you feel this will address the issue? I'd like your help learning a bit more about this. Thanks!!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rcdailey commented 5 years ago

Sorry I understand that they are tags, I communicated what I mean to say wrong. You're basically suggesting that instead of organizrtools/organizr-v2, I use organizrtools/organizr-v2:php-fpm, correct? If so, then my question actually is about the difference between those two images. What does php-fpm tag do differently than the latest tag? What configuration changes, if any, were made? What difference is there that addresses the issue I mentioned about memory limits? Also, would it still be beneficial to go over some diagnostic tools I can use to investigate this issue now or in the future?

rcdailey commented 5 years ago

I tried organizrtools/organizr-v2:php-fpm along with removing the explicit memory_limit = 256M and I'm seeing no errors so far in /config/log/nginx/error.log. Assuming that log is still the correct place to look for errors. Why did this new image fix the issue? Would still like to understand that...

causefx commented 5 years ago

The reason you dont need the limit anymore is because we allow unlimited requests and spawn new children when needed as they are now called on-demand

We also switch to running the process on a socket now.

https://github.com/organizrTools/docker-organizr-v2/blob/php-fpm/root/defaults/www.conf

Roxedus commented 5 years ago

I would wipe the login log, it's quite big

Sendt fra min iPhone

  1. nov. 2018 kl. 16:58 skrev causefx notifications@github.com<mailto:notifications@github.com>:

The reason you dont need the limit anymore is because we allow unlimited requests and spawn new children when needed as they are now called on-demand

We also switch to running the process on a socket now.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/causefx/Organizr/issues/990#issuecomment-442886836, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AGx-MtJRhtEhorKK_QGMHzsVDo3-094Nks5u0AQlgaJpZM4Y476A.

rcdailey commented 5 years ago

Perfect. Thank you for explaining. Will this eventually make it to the latest tag? If so, maybe keep this open until then? I'll need to know when to switch the tag again in my compose file.

rcdailey commented 5 years ago

@causefx Any labels/milestones you can attach to this so you know when to close it? When will it make it to latest? Thanks!

causefx commented 5 years ago

I will speak with tron about maybe moving this to latest.

rcdailey commented 5 years ago

@causefx Wanted to drop in and check on the status of the v2 image. Should I continue to use the php-fpm tag? Or has it been combined with latest yet?

Reason I ask is that I am still using php-fpm and I still get this multiple times a day:

"PHP message: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 23072768 bytes) in /config/www/Dashboard/api/functions/log-functions.php on line 30" while reading response header from upstream

causefx commented 5 years ago

how big is both organizr log files ? the ones located in same directory as db

rcdailey commented 5 years ago 
    98304 Oct 25 19:40 'organizrdb[2018-10-26_00-40-57][2.0.0-beta-800].bak.db'
   147456 Dec 26 22:55  organizrdb.db
     1134 Dec 26 22:55  organizrLoginLog.json
 23765265 Dec 26 22:55  organizrLog.json

Things had been working fine after you suggested to use the new php-fpm tag, but recently that old issue has come back. Does it have something to do with these log files?

causefx commented 5 years ago

delete the organizrlog file, then test again plz

rcdailey commented 5 years ago

I think I recall that this was the stopgap solution before. But it seems the log files continue to be a problem. Why do they grow in size to the point where a 23MB allocation occurs? Are these logs not managed/maintained by Organizr to keep their size down? Is this something I need to be doing? What's the process?

causefx commented 5 years ago

can u check and see what the log message is that is being repeated. i thought i wrote a function to check the size before a read. i will check after the holiday

Sent from my iPhone

On Dec 26, 2018, at 10:04 PM, Robert Dailey notifications@github.com wrote:

I think I recall that this was the stopgap solution before. But it seems the log files continue to be a problem. Why do they grow in size to the point where a 23MB allocation occurs? Are these logs not managed/maintained by Organizr to keep their size down? Is this something I need to be doing? What's the process?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rcdailey commented 5 years ago

Well unfortunately I just deleted the log file, per your instructions. Unless you mean the log files under log/nginx? I'll keep an eye on it and see if it grows to a considerable size again. Enjoy your holiday, and let me know what you find! Thanks!

causefx commented 5 years ago

no worries. was just wondering. i’ll get back with ya later. thanks.

Sent from my iPhone

On Dec 26, 2018, at 10:16 PM, Robert Dailey notifications@github.com wrote:

Well unfortunately I just deleted the log file, per your instructions. Unless you mean the log files under log/nginx? I'll keep an eye on it and see if it grows to a considerable size again. Enjoy your holiday, and let me know what you find! Thanks!

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

causefx commented 5 years ago

I will open this back up if someone else has this problem again.

rcdailey commented 5 years ago

@causefx I'm happy to reword the description of the issue, but I think we should leave it open to address the growing logs issue, which I'm still seeing. This seems to be the root cause of the memory issues. This time, I didn't delete the file. Most of the file is this:

date":"2019-01-21T14:58:34Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:58:40","utc_
date":"2019-01-21T14:58:40Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:58:45","utc_
date":"2019-01-21T14:58:45Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:58:51","utc_
date":"2019-01-21T14:58:51Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:58:56","utc_
date":"2019-01-21T14:58:56Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:02","utc_
date":"2019-01-21T14:59:02Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:07","utc_
date":"2019-01-21T14:59:07Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:13","utc_
date":"2019-01-21T14:59:13Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:18","utc_
date":"2019-01-21T14:59:18Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:24","utc_
date":"2019-01-21T14:59:24Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:29","utc_
date":"2019-01-21T14:59:29Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:35","utc_
date":"2019-01-21T14:59:35Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:40","utc_
date":"2019-01-21T14:59:40Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:46","utc_
date":"2019-01-21T14:59:46Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:51","utc_
date":"2019-01-21T14:59:51Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 14:59:57","utc_
date":"2019-01-21T14:59:57Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 15:00:02","utc_
date":"2019-01-21T15:00:02Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 15:00:08","utc_
date":"2019-01-21T15:00:08Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 15:00:13","utc_
date":"2019-01-21T15:00:13Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 15:00:19","utc_
date":"2019-01-21T15:00:19Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
date":"2019-01-21 15:00:24","utc_
date":"2019-01-21T15:00:24Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number"},{"
rcdailey commented 5 years ago

And one log that appears before all of that is from Tautulli:

date":"2019-01-24T03:56:31Z","type":"error","username":"myusername","ip":"1.2.3.4","message":"Tautulli Token Function - Error: [http:\/\/nami.media\/tautulli]cURL error 18: transfer closed with outstanding read data remaining"},{"
causefx commented 5 years ago

i will rework this area but i would suggest using local ip’s for your homepage items.

Sent from my iPhone

On Jan 24, 2019, at 4:43 PM, Robert Dailey notifications@github.com wrote:

And one log that appears before all of that is from Tautulli:

date":"2019-01-24T03:56:31Z","type":"error","username":"myusername","ip":"1.2.3.4","message":"Tautulli Token Function - Error: [http:\/\/nami.media\/tautulli]cURL error 18: transfer closed with outstanding read data remaining"},{" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

rcdailey commented 5 years ago

@causefx Has this been reworked at all? I think we should reopen this. I'm getting this issue still. My log.json is 22MB and attempting to log in gives me errors. Here is a little snippet of what I'm seeing today:

date":"2019-03-29T01:25:45Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"
date":"2019-03-29 01:25:50","utc_
date":"2019-03-29T01:25:50Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"
date":"2019-03-29 01:25:55","utc_
date":"2019-03-29T01:25:55Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"
date":"2019-03-29 01:26:00","utc_
date":"2019-03-29T01:26:00Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"
date":"2019-03-29 01:26:05","utc_
date":"2019-03-29T01:26:05Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"
date":"2019-03-29 01:26:10","utc_
date":"2019-03-29T01:26:10Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"
date":"2019-03-29 01:26:15","utc_
date":"2019-03-29T01:26:15Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"
date":"2019-03-29 01:26:20","utc_
date":"2019-03-29T01:26:20Z","type":"error","username":"SYSTEM","ip":"1.2.3.4","message":"Transmission Connect Function - Error: cURL error 35: error:1408F10B:SSL routines:ssl3_get_record:wrong version number"},{"

This repeats over hundreds of lines.