Closed martinhellwagner closed 2 years ago
aelvan
commented
2 years ago Hi,
Yes, it's suppose to only be a temp folder that is used when transcoding to modern file formats (webp, avif, jxl), and the temp file is suppose to be deleted afterwards. But there might be a bug, I'll have a look.
In the meantime, can you provide some more details on what formats you're transcoding to and how.
martinhellwagner
commented
2 years ago Thanks for the quick answer! This is what our Imager-X config looks like:
'*' => [
'transformer' => 'craft',
'imagerSystemPath' => '@webroot/assets/imager/',
'imagerUrl' => '/assets/imager/',
'cacheDuration' => 31536000, // 1 Year
'cacheDurationRemoteFiles' => 31536000, // 1 Year
'cacheDurationExternalStorage' => 31536000, // 1 Year
'jpegQuality' => 80,
'fallbackImage' => getenv('FALLBACK_IMAGE') ?: null,
'mockImage' => getenv('MOCK_IMAGE') ?: null,
'optimizers' => [
'jpegoptim',
'jpegtran',
'mozjpeg',
'optipng',
'gifsicle',
],
'pngCompressionLevel' => 0,
'smartResizeEnabled' => true,
'useCwebp' => true,
'webpQuality' => 80,
],In the temp folder, I'm seeing only .png and .jpeg files. Is this expected behaviour?
aelvan
commented
2 years ago Yes, that's to be expected. When using cwebp to convert to webp, Imager will create a transformed file in the original format with full quality in the temp folder, and transcode that to webp using cwebp.
But, I checked and Imager is suppose to delete the file after transcoding, but if an error occurs it doesn't (I'll fix that). Can you check your logs to see if you get an error when trying to convert to webp, it should say something like Custom encoder failed. Output was....
martinhellwagner
commented
2 years ago Indeed we are getting quite a lot of these errors (329 times in the web.log file). I've copied one for reference:
The executed command was "/usr/bin/cwebp -q 80 -m \{effort\} /var/www/ocilion/htdocs/releases/930deb084305736924112988f10ac28ea260526a/storage/runtime/imager/temp/18a747fd6bde15a97ad653ea6ab4ad8d.png -o /var/www/ocilion/htdocs/current/web/assets/imager/images/Video-on-Demand-Poster/Netzbetreiber/59254/godzilla_vs_kong_10b268ca6ba9325119055562eaa5fb9c.webp"
2021-11-17 12:28:45 [-][-][-][error][spacecatninja\imagerx\transformers\CraftTransformer::saveWithCustomEncoder] Custom encoder failed. Output was:
Usage:
cwebp [-preset <...>] [options] in_file [-o out_file]
If input size (-s) for an image is not specified, it is
assumed to be a PNG, JPEG, TIFF or WebP file.
Options:
-h / -help ............. short help
-H / -longhelp ......... long help
-q <float> ............. quality factor (0:small..100:big), default=75
-alpha_q <int> ......... transparency-compression quality (0..100),
default=100
-preset <string> ....... preset setting, one of:
default, photo, picture,
drawing, icon, text
-preset must come first, as it overwrites other parameters
-z <int> ............... activates lossless preset with given
level in [0:fast, ..., 9:slowest]
-m <int> ............... compression method (0=fast, 6=slowest), default=4
-segments <int> ........ number of segments to use (1..4), default=4
-size <int> ............ target size (in bytes)
-psnr <float> .......... target PSNR (in dB. typically: 42)
-s <int> <int> ......... input size (width x height) for YUV
-sns <int> ............. spatial noise shaping (0:off, 100:max), default=50
-f <int> ............... filter strength (0=off..100), default=60
-sharpness <int> ....... filter sharpness (0:most .. 7:least sharp), default=0
-strong ................ use strong filter instead of simple (default)
-nostrong .............. use simple filter instead of strong
-sharp_yuv ............. use sharper (and slower) RGB->YUV conversion
-partition_limit <int> . limit quality to fit the 512k limit on
the first partition (0=no degradation ... 100=full)
-pass <int> ............ analysis pass number (1..10)
-crop <x> <y> <w> <h> .. crop picture with the given rectangle
-resize <w> <h> ........ resize picture (after any cropping)
-mt .................... use multi-threading if available
-low_memory ............ reduce memory usage (slower encoding)
-map <int> ............. print map of extra info
-print_psnr ............ prints averaged PSNR distortion
-print_ssim ............ prints averaged SSIM distortion
-print_lsim ............ prints local-similarity distortion
-d <file.pgm> .......... dump the compressed output (PGM file)
-alpha_method <int> .... transparency-compression method (0..1), default=1
-alpha_filter <string> . predictive filtering for alpha plane,
one of: none, fast (default) or best
-exact ................. preserve RGB values in transparent area, default=off
-blend_alpha <hex> ..... blend colors against background color
expressed as RGB values written in
hexadecimal, e.g. 0xc0e0d0 for red=0xc0
green=0xe0 and blue=0xd0
-noalpha ............... discard any transparency information
-lossless .............. encode image losslessly, default=off
-near_lossless <int> ... use near-lossless image
preprocessing (0..100=off), default=100
-hint <string> ......... specify image characteristics hint,
one of: photo, picture or graph
-metadata <string> ..... comma separated list of metadata to
copy from the input to the output if present.
Valid values: all, none (default), exif, icc, xmp
-short ................. condense printed message
-quiet ................. don't print anything
-version ............... print version number and exit
-noasm ................. disable all assembly optimizations
-v ..................... verbose, e.g. print encoding/decoding times
-progress .............. report encoding progress
Experimental Options:
-jpeg_like ............. roughly match expected JPEG size
-af .................... auto-adjust filter strength
-pre <int> ............. pre-processing filterAh, dang, I introduced a bug in 3.5.0 it seems, which probably explains everything. Sorry, I'll push a bugfix shortly!
martinhellwagner
commented
2 years ago Thanks, much appreciated! 🙏🏼
aelvan
commented
2 years ago Both of these issues should be fixed in 3.5.4. You can safely delete the temp folder. Let me know if files still keep building up there, then there's something else that's wrong.
Worth noting that as of 3.5.0, useCwebp has been deprecated and replaced with an new concept called custom encoders. You should remove that config setting, and add the a customEncoder config for webp. The one in the example should work, just make sure you get the path right.
martinhellwagner
commented
2 years ago Thanks so much! Will update that in all of our projects that previously had 3.5.0+ installed.
So would that be an acceptable configuration in the imager-x.php config file?
return [
'*' => [
'transformer' => 'craft',
'imagerSystemPath' => '@webroot/assets/imager/',
'imagerUrl' => '/assets/imager/',
'cacheDuration' => 31536000, // 1 Year
'cacheDurationRemoteFiles' => 31536000, // 1 Year
'cacheDurationExternalStorage' => 31536000, // 1 Year
'jpegQuality' => 80,
'fallbackImage' => getenv('FALLBACK_IMAGE') ?: null,
'mockImage' => getenv('MOCK_IMAGE') ?: null,
'optimizers' => [
'jpegoptim',
'jpegtran',
'mozjpeg',
'optipng',
'gifsicle',
],
'pngCompressionLevel' => 0,
'smartResizeEnabled' => true,
'customEncoders' => [
'webp' => [
'path' => '/usr/local/bin/cwebp',
'options' => [
'quality' => 80,
'effort' => 4,
],
'paramsString' => '-q {quality} -m {effort} {src} -o {dest}'
],
],
],
// Live Env
'live' => [
],
// Stage Env
'stage' => [
'optimizers' => null,
],
// Local Env
'local' => [
'optimizers' => null,
],
];Yes, just make sure that the path to the cwebp runtime (ie /usr/local/bin/cwebp) is correct. You didn't have that configured in the settings you posted earlier, and the default value used to be /usr/bin/cwebp. Based on the error message you posted, it looks like the path should be /usr/bin/cwebp, but you might have it both places. :)
martinhellwagner
commented
2 years ago How can I find out what is the runtime path of cwebp? I've tried whereis cwebp on the server – seems like it's located at /usr/bin/cwebp:
aelvan
commented
2 years ago whereis is the way. So your config should be:
'customEncoders' => [
'webp' => [
'path' => '/usr/bin/cwebp',
'options' => [
'quality' => 80,
'effort' => 4,
],
'paramsString' => '-q {quality} -m {effort} {src} -o {dest}'
],
],Yes, adding that everywhere now. Thanks again for the help! 🙂 💯
martinhellwagner
commented
2 years ago @aelvan
Since we had some server troubles recently and want to fully understand what caused them (and if it's possible that the Imager-X bug caused it), can I ask a couple of questions about the plugin?
Thanks so much for your answers!
aelvan
commented
2 years ago Hi,
Are failed transforming / encoding jobs retried periodically? If so, how many times?
Depends on what fails. But in general, if a transform is not created for some reason, Imager will try to create it on the next request. So in the case of the issue above, the WebP transforms would be retried on every request. Which means the temporary files would be created on every request. So yes, it could most definitely have impacted performance in a serious way (again, sorry about that!).
Does Imager-X access the database in any way? If so, is it possible that the failed transforming / encoding jobs have caused a heavy load on MySQL?
No, Imager does not use the database at all.
Not sure if you're aware, but there's been several issues with Craft lately that has resulted in heavy database load, we've had our fair share of this happening across our sites. Looks like the last changes done in 3.7.19 have mostly fixed this (although introduced some new challenges), so if you're still on 3.7.12, I'd recommend updating.
Are a lot of CPU / RAM resources used when transforming / encoding images?
Yes, most definitely, transforming images is a very resource intensive process.
Is it possible that the transformation / encoding of images in one project causes a shortage of resources, leading to large First Byte Times in other projects hosted on the same server?
It could, if the server resources are shared and strained. One php-fpm process will generally try to take 100% of the cpu if it needs it (it will for image transforms), so if you're server has four cpu's and more than four processes are battling over those, something has to give (this is a simplification, there're usually more factors that come into play, but...).
martinhellwagner
commented
2 years ago Thanks for the great insights @aelvan! I assume it's pretty likely that the constant re-tries caused a lot of strain on the CPU. All our important projects run on 3.7.21 already, but I'll keep an eye out for any projects running on < 3.7.19.
I'm submitting a...
Steps to reproduce
Description
It seems like the files in "storage/runtime/imager/temp" never get deleted. Shouldn't this just be a temp folder? On some sites, we have up to 50 GB assets pushed into this folder. Is this normal?
Additional info