Add Download button if url= attribute in XML library

[kelson42]

Following #344, we want a download icon in the kiwix-serve taskbar if kiwix-serve is used with an XML file having a url= attribute.

[veloman-yunkan]

If the url attribute in the library XML file points to a metalink file (like the case is for all entries in http://download.kiwix.org/library/library_zim.xml), should the download button simply point to that same file and let the browser or user take further actions after downloading it? Or kiwix-serve should pick from the list of mirrors a direct http/https link to the ZIM file and use if for the download button?

Note that as of today neither firefox nor chrome (without any special extensions/add-ons installed) handle metalinks. For example, you can click the following link and see if the download of the ZIM file starts.

[kelson42]

@rgaudin @mgautierfr Currently we have .meta4 link in the library.xml served on https://library.kiwix.org. The remark of @veloman-yunkan is pertinent. We need here a link pointing to the ZIM file directly on http://download.kiwix.org. We can not remove the meta4 link because this is used by clients like Kiwix Desktop. What is the best solution to your opinion to deal with that?

To me the only one proper solution is to create a list of subnoses <url> OR to have attributes like metalink_url to put the metalink and keep url attribute for the file itself.

[rgaudin]

Where is this URL attribute coming from? Do we generate this? How generic is this gonna be? My question being, are we adding a feature that's gonna be useful only for library.kiwix.org?

One the question, if you have a list, how do you pick which element to use?

Shouldn't we be serving an URL on download.kiwix.org that lists the available download methods: metalink, torrent, individual HTTP links on all mirror and checksums ? Kinda like the ubuntu releases page.

This would be dynamic as all our options are the same for every file. It would have the added benefit of being usable outside library.kiwix.org and could simplify the wiki monster a bit.

[kelson42]

@rgaudin I quite agree with this last comment, but actually we don't really need a list because the list is given in the HTTP header already, see for example:

$ curl -I https://download.kiwix.org/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim
HTTP/2 302 
server: nginx/1.17.6
date: Mon, 18 Jan 2021 10:05:37 GMT
content-type: text/html; charset=iso-8859-1
x-mirrorbrain-mirror: nluug.nl
x-mirrorbrain-realm: region
link: <http://download.kiwix.org/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim.meta4>; rel=describedby; type="application/metalink4+xml"
link: <http://download.kiwix.org/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim.torrent>; rel=describedby; type="application/x-bittorrent"
link: <https://ftp.nluug.nl/pub/kiwix/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim>; rel=duplicate; pri=1; geo=nl
link: <https://mirrors.dotsrc.org/kiwix/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim>; rel=duplicate; pri=2; geo=dk
link: <https://ftp.fau.de/kiwix/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim>; rel=duplicate; pri=3; geo=de
link: <https://ftp.acc.umu.se/mirror/kiwix.org/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim>; rel=duplicate; pri=4; geo=se
link: <https://mirror.download.kiwix.org/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim>; rel=duplicate; pri=5; geo=fr
digest: MD5=L6nZiDXAKkHnBl/3PRrQAA==
digest: SHA=ihu81ow8sojO1ZtsYA3eIuQnDWg=
digest: SHA-256=r7csztDhhsZfKaNq3R5FqnM4Jep2b4NVuccf+01ht7E=
location: https://ftp.nluug.nl/pub/kiwix/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim

Maybe aria2c is already able to deal properly with these headers out of the box?!

[kelson42]

Actually, aria2c can deal out-of-the-box with this, see:

$ aria2c https://download.kiwix.org/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim

01/18 11:09:30 [NOTICE] Downloading 1 item(s)

01/18 11:09:30 [NOTICE] Download complete: /tmp/test/vikidia_fr_all_maxi_2021-01.zim.meta4

01/18 11:09:30 [NOTICE] CUID#10 - Redirecting to https://laotzu.ftp.acc.umu.se/mirror/kiwix.org/zim/vikidia/vikidia_fr_all_maxi_2021-01.zim
[#0b78b3 786MiB/787MiB(99%) CN:1 DL:61MiB]                                                                                                                                                                                                                                       
01/18 11:09:42 [NOTICE] Download complete: /tmp/test/vikidia_fr_all_maxi_2021-01.zim

Download Results:
gid   |stat|avg speed  |path/URI
======+====+===========+=======================================================
330084|OK  |   128KiB/s|/tmp/test/vikidia_fr_all_maxi_2021-01.zim.meta4
0b78b3|OK  |    66MiB/s|/tmp/test/vikidia_fr_all_maxi_2021-01.zim

Status Legend:
(OK):download completed.
$ ls
vikidia_fr_all_maxi_2021-01.zim  vikidia_fr_all_maxi_2021-01.zim.meta4

Not sure how how kiwix-ios whould react if the OPDS stream had just a .zim URL. I guess it would just work. @automactic would you be able to confirm?

Not sure how kiwix-android would work... I believe the current code check the check hashes but not sure.

[rgaudin]

I'm not sure what's left to discuss. Ticket is: if there's a url attribute on the library for that ZIM, display a download button pointing to it. How this URL is generated/appended to the library or where it actually points to is not relevant here.

Now from @veloman-yunkan and @kelson42 comments, we know we can't use url as this is already in place and used by kiwix-desktop and that would break. So maybe using a different download_url attribute instead?

[mgautierfr]

There is already a url in the library. It is used by the kiwix-lib opds code to generate the download url in the opds stream (and allow client to download zim files)