Missing thumbnails with large image galleries in the wiki

[HikeAndMap]

Dear Sir, Madam,

I talked a lot in the Telegram groups about this issue to make sure it's not just me. But a lot of people report the wiki.openstreetmap.org is loading extremely slow and as I discovered not only so slow that it's annoying it's even generating bugs because of the poor performance of the server.

https://en.wikipedia.org/wiki/Road_signs_in_the_Philippines has no problem rendering the thumbnails within a second and showing it to the user.

Porting it to wiki.openstreetmap.org and changing it into an easy lookup table for the mapper plus adding the official government ID to each sign, there are many issues 1) loading takes ages sometimes 2) half of the thumbnails aren't loaded

A possible reason why wiki.openstreetmap.org is creating errors - opposed to wikipedia.org which doesn't have any problems - I found an entry at dreamhost guide, source: https://help.dreamhost.com/hc/en-us/articles/217292577-MediaWiki-Installing-and-more

MediaWiki can occasionally have problems generating image thumbnails, such as returning corrupted images or the error message "Error creating thumbnail". A common cause of this is not enough memory.

If you upload a large image and see "Error creating thumbnail" where a thumbnail should be, try adding the following to your LocalSettings.php file in your site’s primary directory:

$wgMaxShellMemory = 524288;

Maybe this won't solve the issue at hand - who knows - but I'm quite sure it's a server performance issue.

When looking at the "page source" what the server returns..

up to the point it loads the embedded thumbnails properly I see:

</p>
            </div>
        </div></li>
        <li class="gallerybox" style="width: 45px"><div style="width: 45px">
            <div class="thumb" style="width: 40px;"><div style="margin:0px auto;"><a href="/wiki/File:Philippines_road_sign_R2-3_R.svg" class="image"><img alt="" src="https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Philippines_road_sign_R2-3_R.svg/40px-Philippines_road_sign_R2-3_R.svg.png" decoding="async" width="40" height="40" srcset="https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Philippines_road_sign_R2-3_R.svg/60px-Philippines_road_sign_R2-3_R.svg.png 1.5x, https://upload.wikimedia.org/wikipedia/commons/thumb/b/bb/Philippines_road_sign_R2-3_R.svg/80px-Philippines_road_sign_R2-3_R.svg.png 2x" /></a></div></div>
            <div class="gallerytext">
<p><b>R2-3 (R)</b>

But from the moment the error arises it doesn't return to the user inside the generated html the links:

</p>
            </div>
        </div></li>
        <li class="gallerybox" style="width: 45px"><div style="width: 45px">
            <div class="thumb" style="height: 40px;"></div>
            <div class="gallerytext">
<p><b>W8-9</b>

See it's empty!

[tomhughes]

Well if you told us what page you were having a problem with maybe we would be able to diagnose the problem instead of you having to try and guess?

[tomhughes]

To address your actual point, the wiki server has plenty of resources - it is using less than a quarter of it's CPU and memory even during peaks:

https://munin.openstreetmap.org/openstreetmap.org/tabaluga.openstreetmap.org/index.html#system

[HikeAndMap]

ou told us what page you were having a problem with maybe we would be able to diagnose the problem instead of you having to try and guess?

Oops, I thought I included it

https://wiki.openstreetmap.org/wiki/Road_signs_in_the_Philippines/sign_lookup_table

I should mention: occasionally it generates the html complete.. then it works..

And thank you for looking into this - really appreciated

[tomhughes]

Thanks, but I'm not sure I can help.

I'm not sure why you think there is an error creating the thumbnails because I don't see any sign of that - what I see is that for whatever reason the gallery tag isn't even putting in image references to load the thumbnails on those entries but I can't see any logical reason why.

None of it really makes sense - some of the missing ones actually have pregenerated thumbnails on the serve and some of the ones which are there don't have any thumbnails on the server!

[tomhughes]

We have $wgGenerateThumbnailOnParse set to false which as I understand things from https://www.mediawiki.org/wiki/Manual:$wgGenerateThumbnailOnParse means that when the page is generated it is just supposed to output the URLs and then the images are generated on the fly.

But that isn't happening - it is just outputting text with no URL so no thumbnail is requested and none gets generated.

[HikeAndMap]

Well I was hoping the $wgMaxShellMemory = 524288;

Would do the trick - but if you think it has nothing to do with it..

For me nothing makes sense.. I just know the following: creating a page on wikipedia

just start a new page: https://en.wikipedia.org/w/index.php?title=Philippines_road_signs&action=submit

and then I click preview (I don't actually create that page, it's just exemplary)

the wikipedia.org loads it instantly and shows the page exactly as it should be.

Obviously a specific issue like a lookup table to get the proper government ID for the traffic_signal=PH:xxxx tagging belongs on the openstreetmap wiki and not the general wiki :)

Anyway - again thanks for looking into this

[HikeAndMap]

We have $wgGenerateThumbnailOnParse set to false which as I understand things from https://www.mediawiki.org/wiki/Manual:$wgGenerateThumbnailOnParse means that when the page is generated it is just supposed to output the URLs and then the images are generated on the fly.

But that isn't happening - it is just outputting text with no URL so no thumbnail is requested and none gets generated.

Ohh, when will this be reflected .. server restart ?

[tomhughes]

Saving that page does take a very long time which suggests it may be trying to generate the thumbnails at that point but eventually timing out. Our resource limits are far higher than you're suggesting already though:

$wgGenerateThumbnailOnParse = false;
$wgMaxImageArea = 125000000;
$wgMaxShellMemory = 5524000;
$wgMaxShellFileSize = 819200;
$wgMaxShellTime = 360;
$wgMaxShellWallClockTime = 360;

[tomhughes]

We have $wgGenerateThumbnailOnParse set to false which as I understand things from https://www.mediawiki.org/wiki/Manual:$wgGenerateThumbnailOnParse means that when the page is generated it is just supposed to output the URLs and then the images are generated on the fly. But that isn't happening - it is just outputting text with no URL so no thumbnail is requested and none gets generated.

Ohh, when will this be reflected .. server restart ?

When will what be reflected? I haven't changed anything, I was just reporting what the current configuration is...

[HikeAndMap]

000

yup, so that can't be it.. it's maybe a bug inside the wikimedia itself.. although weird it does work properly on wikipedia.org

[HikeAndMap]

So right now it works.. Around 1.5 hours after you changed the $wgGenerateThumbnailOnParse Since it's been 1.5 hour not sure if that setting changed the behavior, or is there a delay between changing that setting and when it's applied?

And this is the data from loading the page:

<!-- 
NewPP limit report
Cached time: 20211026194709
Cache expiry: 86400
Dynamic content: false
Complications: []
CPU time usage: 0.650 seconds
Real time usage: 0.701 seconds
Preprocessor visited node count: 313/1000000
Post‐expand include size: 266/2097152 bytes
Template argument size: 0/2097152 bytes
Highest expansion depth: 2/40
Expensive parser function count: 0/500
Unstrip recursion depth: 0/20
Unstrip post‐expand size: 212417/5000000 bytes
Number of Wikibase entities loaded: 0/250
-->
<!--
Transclusion expansion time report (%,ms,calls,template)
100.00%    3.717      1 WikiProject_Philippines/header
100.00%    3.717      1 -total
-->

Which is interesting if it works - the page is loaded in less than a second..

[HikeAndMap]

Changing px of the gallery:

<!-- 
NewPP limit report
Cached time: 20211026195727
Cache expiry: 86400
Dynamic content: false
Complications: []
CPU time usage: 0.801 seconds
Real time usage: 2.551 seconds
Preprocessor visited node count: 313/1000000
Post‐expand include size: 266/2097152 bytes
Template argument size: 0/2097152 bytes
Highest expansion depth: 2/40
Expensive parser function count: 0/500
Unstrip recursion depth: 0/20
Unstrip post‐expand size: 212845/5000000 bytes
Number of Wikibase entities loaded: 0/250
-->
<!--
Transclusion expansion time report (%,ms,calls,template)
100.00%    4.082      1 WikiProject_Philippines/header
100.00%    4.082      1 -total
-->

Takes a bit longer to load - but still loading all thumbs now..

I noticed already - sometimes there's moments like half an hour during that period everything is loaded just fine..

and then after half an hour - it again has this problem..

let's hope changing the $wgGenerateThumbnailOnParse made a persistent effect that it now may always be working

[tomhughes]

As I keep telling you I didn't change anything!

[HikeAndMap]

As I keep telling you I didn't change anything!

Ah I misunderstood I thought you mentioned you changed that setting..

my mistake.

it's weird then sometimes it's loading the page properly and fast and other times not - again I'm not the only one having this issue - just most folks on Telegram don't take the time to report issues in the proper channels

[HikeAndMap]

https://commons.wikimedia.org/wiki/Category:Images

This page shows 200 images in what I think is also a gallery?

So it should be working regardless of size right?

[HikeAndMap]

So today I got a hold on some MediaWiki experts and they had a look at it and came up with the following:

• the server seems to have slow loads on a lot of wiki.openstreetmap.org pages
• There's a timing out issue between the host/server of wiki.openstreetmap.org and the https://commons.wikimedia.org
• The InstantCommons are then not/partially loaded - and you're left with just showing the filename text instead of the thumbnail
• what exactly is causing the timeout is hard to tell
• looks a lot like potentially server load

Those are the 4 points I was informed about considering this issue but obviously something between wiki.openstreetmap.org and commons.wikimedia.org is going wrong, whatever the cause is.

[HikeAndMap]

https://www.mediawiki.org/wiki/Manual:$wgForeignFileRepos

That's from the manual:

Performance

You may need to configure the $wgMainCacheType as well. Default is often CACHE_NONE, meaning it will load the image from the remote host on each page load which will be very slow. Similarly, you need to set apiThumbCacheExpiry to zero if you prefer to use the foreign thumbnails which will be faster.

Setting $wgResponsiveImages = false; may improve performance of instant commons at the cost of making images be lower quality on high resolution displays.

Currently MediaWiki does not support pipelining foreign api requests. A high performance site may want to look at setting up a local proxy (like nginx) that can coalesce multiple requests into a single pipelined request to reduce round trip times from TCP & TLS handshakes. 

and

Likewise, avoid sharing a database between the commons wiki and other content wikis, especially if you must use ForeignAPIRepo instead of ForeignDBRepo due to limitations imposed by your service provider or administrator. This generates a large number of potentially long-lived database connections that can result in impaired performance, or can exceed DB connection limits even on wikis with very little traffic or activity. 

As I understood, https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:$wgUseInstantCommons is basically a wrapper around wgForeignFileRepos

but has the option:

For example, if you have 100 InstantCommons images on a page, your wiki will make 400 API requests to the Commons API at page load time and the first byte of the page won't be returned for about 30 seconds. This can be somewhat improved by disabling $wgResponsiveImages You may prefer to override the default descriptionCacheExpiry by creating your own $wgForeignFileRepos entry, or maybe not use foreign file repositories at all and instead copy images from Commons to your own wiki using importImages.php

[1ec5]

In case you haven’t come across it yet, there’s some overlap with #466: sometimes the page itself times out, and rerequesting it sometimes has missing thumbnails as you’ve experienced. Should we close one of these issues as a duplicate?

[tomhughes]

How is that related? There is no timeout fetching anything here - what happens here is that the page renderer never even generates the link to the image.

[tomhughes]

Also what does commons have to do with this? The images here are all from our wiki...

[1ec5]

How is that related? There is no timeout fetching anything here - what happens here is that the page renderer never even generates the link to the image.

That’s why I only described it as overlap: different symptoms, but both the timeout and the missing images occur on different requests for the same page, whether it’s “Road signs in the Philippines/sign lookup table” or one of the pages mentioned in #466. (The whole genre of road sign lookup table pages is affected, including many of the pages in this category.)

You aren’t seeing links for the missing images because the images are in a <gallery> tag, which doesn’t link the alt text that appears when the image fails to load.

Also what does commons have to do with this? The images here are all from our wiki...

Not for “Road signs in the Philippines/sign lookup table”: those images all come from Commons.

[tomhughes]

So I was unaware that the images on that page were actually from commons, because I didn't even know that was possible!

It does explain why I was having so much trouble finding the images on our system though...

There is no problem with our link to commons - our wiki server is in Amsterdam as is one of the wikimedia clusters and the two networks are directly connected with a 1.25ms RTT between them.

We've been using the simple $wgUseInstantCommons setting for a long time now though apparently that used to enable thumbnail caching but it no longer does - supposedly because you get better performance without it.

Switching to the verbose method and enabling caching in https://github.com/openstreetmap/chef/commit/8e9c9e21a5fc964dde220cfa0e99d98bd3739c0a does seem to have improved things though even though it actually still seems to be generating hotlinks to commons (though it does also generated cached thumbs on our system) but it's possible that if all the thumbs expire together that the next person to load the page may find it slow.

Fundamentally having to make hundreds of API calls to a third party system to render a page is always going to cause slowdown...

[HikeAndMap]

'''An issue that is under investigation might mean that not all thumbnails are rendered. Click the following action to load additional thumbnails'''

My compliments, beautiful wording!

btw the page does seem to render now perfectly fine and fast!

[HikeAndMap]

Dear Tom,

thank you very much for your help. The page in question now consistently loads properly.

if not yet cached loading can take 4 or 5 seconds (still better than the 30 seconds waiting before) but most importantly it now loads all images properly.

once cached it shows all images within an eye-blink, again all images loading properly.

I tested this - while always clearing the whole browser cache - afternoon yesterday, evening, at night and now this morning..

it now never fails and always responses exactly what the user would expect.

I'm very pleased and thank for very much for dealing with annoying people like me dwelling on issues .

I'd buy you a drink - but we most likely live on opposite sides of the planet so what I can offer is wiring you something for the drink over paypal.

Let me know if that's an option.

Cheers and thanks again for the effort

[HikeAndMap]

Sad, the issue came back now

it worked well for 24+ hours

but now (evening time Europe 29 October 2021) it came back

[HikeAndMap]

Hi Tom, the "File:" can be omitted within a gallery..

Says the manual of the MediaWiki

I saw you put back the "File:"

it doesn't make a difference - tried that already

[HikeAndMap]

Hello Tom,

https://www.mediawiki.org/wiki/Extension:QuickInstantCommons

Might this be something to try to resolve this issue?

[happy5214]

Hello Tom,

https://www.mediawiki.org/wiki/Extension:QuickInstantCommons

Might this be something to try to resolve this issue?

QuickInstantCommons is now listed as stable on the MediaWiki project wiki (it was in beta as of the above comment). Perhaps this should be further investigated?

[matkoniecz]

It was investigated and turned out to be not stable - see https://github.com/openstreetmap/chef/pull/491 and https://github.com/wikimedia/mediawiki-extensions-QuickInstantCommons/issues/1

[HikeAndMap]

It was investigated and turned out to be not stable - see openstreetmap/chef#491 and wikimedia/mediawiki-extensions-QuickInstantCommons#1

Yes I read a few other comments elsewhere from other wiki page admins stating it's not yet stable.. there's still some work to do.

I totally agree stability first! At my home private server stability isn't much of an issue - usually I can deal with it.

But a public available space you make the right call to go for stability first. IMHO

[tomhughes]

Closing as duplicate of #466.

[Firefishy]

I have now enabled QuickInstantCommons but I am not 100% happy as we are using the master branch instead of a REL1_37 branch.

[HikeAndMap]

I tested this extensively over a few months - because I hate to report something has been fixed - or not, without doing a long-term observation if it's really the case what I'm saying.

So now I can say with 100% confidence:"all problems I had have been resolved in this matter"

Thank you very much to the team taking care of us mappers :)