freephile / meza

Setup an enterprise MediaWiki server with simple commands
MIT License
0 stars 0 forks source link

Thumbnails are not rendering #62

Closed freephile closed 8 months ago

freephile commented 8 months ago

Environment

Issue details

Thumbnails are not rendering on File pages, nor on https://localhost/demo/Special:ListFiles, although there are no errors in /opt/data-meza/logs/mw-debug.log, nor on-screen. I also directly checked with https://localhost/demo/Special:ListFiles?requestDebug=true

In the debug log it says:

TransformationalImageHandler::doTransform: creating 120x96 thumbnail at mwstore://local-backend/local-thumb/2/28/Helix.jpg/120px-Helix.jpg using scaler im
TransformationalImageHandler::doTransform: Transforming later per flags.

So, is the problem is with the job queue? Thumbnails were working previously (recently).

The only configuration in LocalSettings.php for thumbnails is:

$wgUploadDirectory = "{{ m_uploads_dir }}/$wikiId";

$wgEnableUploads = true;
$wgMaxUploadSize = 1024*1024*100; // 100 MB
$wgGenerateThumbnailOnParse = false;

$wgHashedUploadDirectory = true;

The Release Notes for 1.39 mention that wgLocalFileRepo did change, but I believe that only involves useJsonMetadata.

Ref:

  1. https://www.mediawiki.org/wiki/Manual:$wgUploadDirectory
  2. https://www.mediawiki.org/wiki/Manual:$wgHashedUploadDirectory
  3. https://www.mediawiki.org/wiki/Manual:$wgLocalFileRepo
freephile commented 8 months ago

By changing the value of $wgGenerateThumbnailOnParse from false to true, I was able to get immediate thumbnail generation.*

In the debug log, instead of the 'transforming later' message, you see stuff like this:

[memcached] MemCache: got WANCache:global:filerepo-file:wiki_demo:e4949c80faeddc9031b745d965b4c811010a1ab0|#|v
File::transform: Doing stat for mwstore://local-backend/local-thumb/2/28/Helix.jpg/180px-Helix.jpg
[FileOperation] FileBackendStore::ingestFreshFileStats: File mwstore://local-backend/local-thumb/2/28/Helix.jpg/180px-Helix.jpg does not exist
TransformationalImageHandler::doTransform: creating 180x144 thumbnail at /tmp/transform_d054bc9fed12.jpg using scaler im
TransformationalImageHandler::doTransform: called wfMkdirParents(/tmp)
BitmapHandler::transformImageMagick: running ImageMagick: '/usr/bin/convert' '-quality' '80' '-background' 'white' '-define' 'jpeg:size=180x144' '/opt/data-meza/uploads/demo/2/28/Helix.jpg' '-thumbnail' '180x144!' '-set' 'comment' 'File source: https://localhost/demo/File:Helix.jpg' '+set' 'Thumb::URI' '-depth' '8' '-sharpen' '0x0.4' '-rotate' '-0' '-sampling-factor' '2x2,1x1,1x1' '/tmp/transform_d054bc9fed12.jpg'
[exec] Executing: /bin/bash '/opt/htdocs/mediawiki/vendor/wikimedia/shellbox/src/Command/limit.sh' ''\''/usr/bin/convert'\'' '\''-quality'\'' '\''80'\'' '\''-background'\'' '\''white'\'' '\''-define'\'' '\''jpeg:size=180x144'\'' '\''/opt/data-meza/uploads/demo/2/28/Helix.jpg'\'' '\''-thumbnail'\'' '\''180x144!'\'' '\''-set'\'' '\''comment'\'' '\''File source: https://localhost/demo/File:Helix.jpg'\'' '\''+set'\'' '\''Thumb::URI'\'' '\''-depth'\'' '\''8'\'' '\''-sharpen'\'' '\''0x0.4'\'' '\''-rotate'\'' '\''-0'\'' '\''-sampling-factor'\'' '\''2x2,1x1,1x1'\'' '\''/tmp/transform_d054bc9fed12.jpg'\''' 'SB_INCLUDE_STDERR=1;SB_CPU_LIMIT=180; SB_CGROUP='\'''\''; SB_MEM_LIMIT=314572800; SB_FILE_SIZE_LIMIT=104857600; SB_WALL_CLOCK_LIMIT=180; SB_USE_LOG_PIPE=yes' 2>&1

**NOTE: I needed to forcefully reload PHP-FPM with each change to PHP (LocalSettings.php) in order for those changes to be seen by the webserver. sudo systemctl reload php-fpm is a pain and a 'gotcha' so this should be understood better.

freephile commented 8 months ago

According to the $wgGenerateThumbnailOnParse documentation

If this is false, a valid thumbnail URL is still outputted, but no file will be created at the target location. This may save some time if you have a thumb.php or 404 handler set up which is faster than the regular webserver(s)

This led us to setting Thumb.php properly and image thumbnailing now works as expected.

It turns out that it was the 'short urls' feature that confused MediaWiki about thumbnailing.

find /opt/data-meza/uploads/ -name '*Helix.jpg'
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/180px-Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/120px-Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/640px-Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/300px-Helix.jpg
/opt/data-meza/uploads/demo/2/28/Helix.jpg
[userx@localhost meza]$ find /opt/data-meza/
freephile commented 8 months ago

Thumbs are currently generated on demand, meaning the full complement of 'common sizes' as defined in $wgThumbLimits and $wgImageLimits (Meza is silent on these settings, so we inherit the defaults) are not generated at file upload time. This is working as intended and emphasizes performance and storage efficiency.

IF thumbnails were generated on parse (ie, on file upload), then you'd have to wait for all the following images to be returned:

$wgThumbLimits = [
    120,
    150,
    180,
    200,
    250,
    300
];
$wgImageLimits = [
    [ 320, 240 ],
    [ 640, 480 ],
    [ 800, 600 ],
    [ 1024, 768 ],
    [ 1280, 1024 ],
    [ 2560, 2048 ],
];

To be clear, ImageLimits is responsible for producing the links on File namespace pages as shown in the image below: image

You can see using find that we do not have all the ThumbLimits images generated for each upload. They only get generated when 'viewed' in some page context.

find /opt/data-meza/uploads/ -name '*Helix*'
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/180px-Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/120px-Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/640px-Helix.jpg
/opt/data-meza/uploads/demo/thumb/2/28/Helix.jpg/300px-Helix.jpg
/opt/data-meza/uploads/demo/thumb/8/82/Helix2-basket.webp
/opt/data-meza/uploads/demo/thumb/8/82/Helix2-basket.webp/120px-Helix2-basket.webp.png
/opt/data-meza/uploads/demo/thumb/8/82/Helix2-basket.webp/640px-Helix2-basket.webp.png
/opt/data-meza/uploads/demo/thumb/8/82/Helix2-basket.webp/180px-Helix2-basket.webp.png
/opt/data-meza/uploads/demo/thumb/5/50/Helix3.jpeg
/opt/data-meza/uploads/demo/thumb/5/50/Helix3.jpeg/80px-Helix3.jpeg
/opt/data-meza/uploads/demo/thumb/5/50/Helix3.jpeg/400px-Helix3.jpeg
/opt/data-meza/uploads/demo/thumb/5/50/Helix3.jpeg/321px-Helix3.jpeg
/opt/data-meza/uploads/demo/thumb/5/50/Helix3.jpeg/513px-Helix3.jpeg
/opt/data-meza/uploads/demo/2/28/Helix.jpg
/opt/data-meza/uploads/demo/8/82/Helix2-basket.webp
/opt/data-meza/uploads/demo/5/50/Helix3.jpeg

It's not clear, however we may also want to add a thumb_handler.php in Apache's .htaccess