Closed bramus closed 5 years ago
andris-sevcenko
commented
5 years ago I'm unable to reproduce this locally with a 22MB image. To be fair, my thumbnails gets generated. Can you clear out your log folder, reproduce this and see if there's anything in the logs?
The reason I'm asking is that the file should be removed from that folder as soon as it's downloaded and moved to storage/runtime/assets/sources folder, so I'm wondering at which point is the script dying exactly.
bramus
commented
5 years ago It's the memory limit of the server that I'm running into:
[28-Sep-2018 16:36:21 Europe/Brussels] PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 16000001 bytes)
in /home/account/vendor/pixelandtonic/imagine/lib/Imagine/Gd/Image.php on line 617The memory_limit on the server is set to 128M.
andris-sevcenko
commented
5 years ago Ah, I think I see what's going on here.
There's a Craft setting for caching cloud images for generating transforms and thumbnails. (https://docs.craftcms.com/v3/config/config-settings.html#maxcachedcloudimagesize)
So, as you request a thumbnail for an image, the image is pulled down to the server from the remote source and cached locally according to the setting. It defaults to a source image of 2000x2000, so, if you have that setting left at default, Craft tries to resize the source image to that size. At this point, since we're talking about an 11MB image, Craft is unable to process that using GD library with only 128MB of RAM.
There are multiple ways to approach this: 1) You can set that setting to false. Potential drawbacks - no local source is stored, so every new transform or new thumbnail size will warrant a new download. 2) You can increase the memory limit. Potential drawbacks - GD still is slow and the quality can be a bit of a lack-luster. 3) You can switch to Imagick. Potential drawbacks - if you're not increasing the memory limit, Imagick will use a swap file when it runs out of memory which can eat up server space a bit and make Imagick slower.
I would recommend a combination of the last two options.
bramus
commented
5 years ago Thanks for further investigating this, @andris-sevcenko.
I was wondering if you could re-open this issue. Not to try to work around the currently set limitations (which won't be possible), but to have a failsafe in place which prevents the ~/storage/runtime/temp folder from filling up with multiple copies of one and the same file.
A deletion of files that have the same filename(prefix) before the source image is being pulled from the remove source would be nice addition: That way only one copy of a file can be left in ~/storage/runtime/temp, should things go wrong.
andris-sevcenko
commented
5 years ago There we go.
cballenar
commented
2 years ago I'm running into this issue currently 🤔 I'm running latest CraftCMS 3.7.18.2 (latest as of this post) on PHP 7.4.25. I tried to set my Temp folder to be in Google Cloud Storage but that doesn't seem to be working. Instead, my main server is running out of space due to multiple copies of the same images getting stored in the Temp folder.
andris-sevcenko
commented
2 years ago @cballenar can you list a few of the filenames that are multiple copies? Also is there anything relevant in Craft log files?
cballenar
commented
2 years ago @andris-sevcenko thanks for follow up!
Below is the format these files use where the first portion seems to be the actual filename and is then followed by an automatically generated string.
I found that one of these files was created 465 times :|
-rw-r--r-- 1 www-data www-data 0 Oct 21 16:28 local-pronaa-16171951695d682.43174614.jpg
-rw-r--r-- 1 www-data www-data 0 Oct 21 17:39 local-pronaa-16171a5b5cdada4.22679641.jpg
-rw-r--r-- 1 www-data www-data 0 Oct 21 18:06 local-pronaa-16171ac21c05687.95788916.jpg
[...]
-rw-r--r-- 1 www-data www-data 0 Oct 26 17:39 minam-equipos-limpieza-publica61783d35c02301.22211731.jpg
-rw-r--r-- 1 www-data www-data 0 Oct 26 17:39 minam-equipos-limpieza-publica61783d38950579.94627616.jpg
-rw-r--r-- 1 www-data www-data 0 Oct 26 17:39 minam-equipos-limpieza-publica61783d3b426603.80293889.jpg
[...]
-rw-r--r-- 1 www-data www-data 124606 Jan 28 2021 'lurifico desague 0601251381ca870.40018390.jfif'
-rw-r--r-- 1 www-data www-data 124606 Jan 28 2021 'lurifico desague 06012604cf3e1e5.36797728.jfif'
-rw-r--r-- 1 www-data www-data 120317 Jan 28 2021 'lurifico desague 1601255884ec9a3.84250880.jfif'
[...]
-rw-r--r-- 1 www-data www-data 39241 Jul 24 2020 'minagri 15f1afafe27d399.69181317.jfif'
-rw-r--r-- 1 www-data www-data 40687 Jul 24 2020 'minagri 25f1afb030a21d8.06103496.jfif'
-rw-r--r-- 1 www-data www-data 40540 Jul 24 2020 'minagri 35f1afb05233dc7.44841691.jfif'
[...]I've looked in phperrors.log but didn't find anything associated to this. I had a harder time navigating web.log but I think this one is it:
2021-11-01 00:44:33 [-][-][-][error][yii\web\HttpException:500] yii\web\ServerErrorHttpException: Image transform cannot be created. in /var/www/mywebsite.com/vendor/craftcms/cms/$
Stack trace:
#0 [internal function]: craft\controllers\AssetsController->actionGenerateTransform(50646)
#1 /var/www/mywebsite.com/vendor/yiisoft/yii2/base/InlineAction.php(57): call_user_func_array(Array, Array)
#2 /var/www/mywebsite.com/vendor/yiisoft/yii2/base/Controller.php(180): yii\base\InlineAction->runWithParams(Array)
#3 /var/www/mywebsite.com/vendor/craftcms/cms/src/web/Controller.php(190): yii\base\Controller->runAction('generate-transf...', Array)
#4 /var/www/mywebsite.com/vendor/yiisoft/yii2/base/Module.php(528): craft\web\Controller->runAction('generate-transf...', Array)
#5 /var/www/mywebsite.com/vendor/craftcms/cms/src/web/Application.php(274): yii\base\Module->runAction('assets/generate...', Array)
#6 /var/www/mywebsite.com/vendor/craftcms/cms/src/web/Application.php(577): craft\web\Application->runAction('assets/generate...', Array)
#7 /var/www/mywebsite.com/vendor/craftcms/cms/src/web/Application.php(253): craft\web\Application->_processActionRequest(Object(craft\web\Request))
#8 /var/www/mywebsite.com/vendor/yiisoft/yii2/base/Application.php(386): craft\web\Application->handleRequest(Object(craft\web\Request))
#9 /var/www/mywebsite.com/web/index.php(21): yii\base\Application->run()
#10 {main}Does this indicate an issue with permissions? I've seen something funky happen in recent updates where I used to keep all files and directories under myuser:www-data but now some directories are under www-data:www-data. I attempted to change this on a copy of the server and had to kill it cuz it broke Craft pretty bad. I ended up with 500 errors and weird project config files (..0xa) :|
Thanks!
andris-sevcenko
commented
2 years ago @cballenar it's not directly related to the original issue, although it is quite similar. I'll see if we can maybe have garbage collector also prune the temp folder and get rid of any files that are older than, for example, a week or so.
cballenar
commented
2 years ago Thank you! Heads up, these are generating quite rapidly. During my troubleshooting I cleared these files and just a couple days later the server storage was maxed out, which meant ~3Gb were generated in that time. Should I consider bumping up this storage space nonetheless?
andris-sevcenko
commented
2 years ago Do you have a frontend upload form or something like that? Perhaps you can send over your entire log folder to support@craftcms.com and reference this issue?
cballenar
commented
2 years ago Nothing like that :\ In reality only one admin adds content. I will do that! Thank you very much!
cballenar
commented
2 years ago Quick follow up in case anyone stumbles on this in the future. In my case this issue was caused due to a misconfiguration in the permissions of the folders. Some were owned by a different user which led to some things working correctly and others not so much.
Thanks!
Description
For a Craft project of mine I have defined the Asset Volumes to be S3 Volumes. This way there are no huge disk space requirements on the server hosting the Craft project, as the files are stored on S3. (For reference: there are already 5GB worth of photos uploaded into the bucket)
Combined with imgix for resizing the original images (which are huge) I've moved away some more load from the server as the CPU-expensive image transforms + caching needs are now imgix's thing to tackle.
To my surprise however I was getting disk usage warnings from my server: the project was using a whopping 7.6GB of disk space.
Looking onto the server I noticed that the culprit is the
storage/runtime/temp/folder, which takes up 5.7GB:Digging deeper into that big
storage/runtime/temp/folder, I notice that it's filled with some – but not all – of the images the client has uploaded. Above that those files appear in there a lot of times. Here's a list of just one of the files:Hmmz … debug modus on:
ZWIJNDRECHT-06b.jpgon there, which also is about 11MB.storage/runtime/temp/, they indeed are copies of the one stored in the S3 bucket.ZWIJNDRECHT-06b.jpgappear.881.Swinging back to the disk contents and it's confirmed, there's no thumbnail there for said asset:
In short:
The admin needs to generate a thumbnail for the image. Since it it has no knowledge of imgix being available (on the frontend), it performs the resize action itself.
Therefore it downloads the original asset from S3 into
storage/runtime/temp/and starts resizing it. That action however fails (file too big, action taking too much time, …) and the temp downloaded file is left instorage/runtime/temp/.Next time a thumbnail needs to be shown this thing starts all over again.
Rinse. Repeat.
Steps to reproduce
storage/runtime/temp/after generation has failed, you'll see a temp file left in there.Additional info