Image clarity

[desb42]

As an example I compared the enwiki and xowa http-server version The page I picked is a random example to try to make my point en.wikipedia.org/wiki/Argentine_Declaration_of_Independence

You will see (possibly) that the enwiki version is sharper. On tracking down this difference I note that enwiki uses the relatively new 'srcset' feature of the \ tag to pick an 'appropriate' image for my screen. In my case that is the thumb image that is 1.5 larger than the defined width and height. To try to put this another way the thumbnail that is 1.5 times bigger is squeezed into the defined width and height

I realise that trying to keep all versions of these thumbnails offline is defeating the purpose.

I am not suggesting any change here

What I would like to do is experiment with generating the 1.5 sized versions in place of the 'real' version in my own builds (to see the overall impact on size of the image database(s)) I am hoping you can give me some pointers into your code as to how to do this. for example is it part of the file.page_regy step and/or part of file.fsdb_make step?

[gnosygnu]

Yeah, this could be difficult. First, I believe that the srcset attribute is lumped on to every [[File:]] image. So supporting srcset would probably affect the vast majority of images in a wiki

As a rough guess, I say that 1.5 versions would scale out the image databases to 2x (100 GB -> 200 GB). This is because a 10 x 10 image of 100 pixels would become a 15 x 15 of 225 pixels. There are a few other variables (compression quality; originals not affected), but the end result would probably be around there.

I have a poor eye with regards to these details. How significant do you feel the difference is from a user-experience? I usually focus on text, and the images just offer me some approximate data. I also tend to use XOWA mostly on Android nowadays, and the image quality there feels sufficient to the mobile viewport.

Regarding your questions, there is some fragmented documentation in the comments of http://xowa.org/home/wiki/Dev/Command-line/Dumps . I list more details below.

But I don't think this would be an easy fix. The main problem is that XOWA uses a composite-key in the HTML to fetch the data from the file-databases. Any change to include srcset would involve changing some of that composite-key logic which has a good deal of complexity to it.

At a high-level, this is what happens.

• XOWA gets the HTML for a page
• XOWA parses the HTML
• When it sees an image tag, it createsa composite-key of FileTitle + FileIsThumbnail + FileWidth + FileHeight + FileUpright + FileMovieTime + FilePdfPage.
  • A typical key of [[File:A.png|thumb]] would look something like this: "A.png|1|-1|-1|-1|-1|-1"
• It then takes the FileTitle and retrieves the original dimensions from the file-core database and orig_reg table
• It then calculates the exact image width and height using the composite key and the original dimensions to find the corresponding entry in fsdb_thm
• It then uses the thm_id and thm_bin_db_id to find the actual binary image in any of the database files.

So, the "easiest" thing would be

• Change the lnki_temp table to multiple all thumb widths by 1.5. This would eventually force XOWA to download 1.5 size images.
• Change the composite-key generation to look for these 1.5 images. The entry-point would be here: https://github.com/gnosygnu/xowa/blob/master/400_xowa/src/gplx/xowa/files/Xof_file_wkr.java#L117

However, this is just pie-in-the sky design. I'm sure there's a lot more complexity when the details are done.

---

* add     ('simple.wikipedia.org' , 'wiki.mass_parse.exec'): 

• Loads every page and generates the HTML
• While generating the HTML, it logs every image (with title, width, height, etc) into a table called lnki_temp

  // aggregate the lnkis
  add     ('simple.wikipedia.org' , 'file.lnki_regy');

  Takes all the rows in lnki_temp and aggregates them into lnki_regy. For example, A.png with width 300 may be used by 5 pages in link_temp; we only want 1 row of A.png with width 300 here

// generate orig metadata for files in the current wiki (for example, for pages in en.wikipedia.org/wiki/File:*)
  add     ('simple.wikipedia.org' , 'file.page_regy')        {build_commons = 'n';}

Generates a complete list of images. Primarily for redirects.

• It does this by adding entries with repo_tid = 1 to a table called page_regy which sits in commons.wikimedia.org and a database called xowa.file.page_regy.sqlite3.
• Basically, this table defines a complete list of possible images available to the wiki (everything in commons.wikimedia.org/wiki/File + those images found in simple.wikipedia.org/wiki/File).

// generate all orig metadata for all lnkis
add     ('simple.wikipedia.org' , 'file.orig_regy');

• Creates a complete meta-table using a combination of lnki_regy (all images with their known usages of height and width) and orig_regy (a list of all known images and redirects).
• This meta-table contains the attributes of the orginal image only
• Basically, we want info like "A.png has an original width of 1000 and an original height of 1200" for every image found in lnki_regy.

// generate list of files to download based on "orig_regy" and XOWA image code
add     ('simple.wikipedia.org' , 'file.xfer_temp.thumb');

• Creates a "download table". This basically determines the actual file to download using lnki_regy, orig_regy, and XOWA code.
• This step is necessary b/c MediaWiki has different ways of specifying a file with a given width. For example, thumb, 300px, upright=1.5, 400x200px (which may not actually yield a 400x200px image depending on the original dimensions)

// aggregate list one more time
add     ('simple.wikipedia.org' , 'file.xfer_regy');

• Aggregate the results again. This is because an image of 300px could have been specified different ways in lnki_regy; thumb, 300px, upright=1.5, 400x200px

// identify images that have already been downloaded
add     ('simple.wikipedia.org' , 'file.xfer_regy_update');

• Find those images that are already present in XOWA file databases and mark them as do not download

// download images. This step may also take a long time, depending on how many images are needed
add     ('simple.wikipedia.org' , 'file.fsdb_make') {

• Download images based on xfer_regy and xfer_status = 0

[desb42]

Thanks for the extensive answer

From a general user perspective, I would guess this to be very minor issue. This is intended to be an offline system where space is a consideration (unlike wikicommons) I would not suggest that you put anymore effort into this. I might have a play though