Add ability to open ZIM file from URL

[nixcamic]

I have a few ZIM files hosted on the local network and it would be nice to be able to open them straight over HTTP (Which seems to be possible in the backend) instead of only being able to browse local files.

[mossroy]

It's probably technically possible, but does not look like a priority to me.

@sharun-s did something similar in his (unmerged) branch. See #275. His code was intended to allow reading a local file through XHR (only working if the file is inside the same folder as the source code). As it's based on XHR, I suppose it would be possible to fetch an http resource in the same way.

But, to me, it looks like a very unusual way to use kiwix-js. If your ZIM files are available through HTTP in your local network, it might be better to use http://wiki.kiwix.org/wiki/Kiwix-serve. It would allow you to directly read the content of the ZIM files through any browser.

[nixcamic]

The problem is we can only serve static content. I noticed that the test suite will load zim files over http however it seems to be limited to small files.

Also I noticed the following text in @sharun-s branch of kiwix-html5, specifically the bold bit, I can't seem to find any more information about what is meant by that though.

This branch is based on the Kiwix-JS Project. Code is available here. The javascript code has many pieces that allow it to be used in 5 different "modes" detailed below. Find the right repo on the KIWIX github page to suite your needs. This version is focused on Standalone mode.

Standalone - distribute zims and code folder on a usb key/sdcard with instructions to click the index.html file to run. It requires least instructions to anyone with low computer literacy i.e. most people who are offline. Local server based which would make duplications of massive zims across many clients unnecessary for local classroom type scenarios WebExtension - kiwix-js WebApp (phone gap/corova based) on app store NativeApp eg the Firefox OS app

[nixcamic]

So I actually have it working using @sharun-s branch, serving the entire Spanish Wikipedia (16GB as a single .zim file!) over HTTP, and it isn't too horribly slow. The only issue I'm having at the moment is that search doesn't work properly in Chrome.

[Jaifroid]

That's excellent news @nixcamic ! Could you let us know which browsers you've tested this on? Last I knew there was a bug in Firefox that prevented using this technique on files larger than a few gigs.

[nixcamic]

I've tested in Firefox 59.0.2 and Firefox 60 (both 64 bit) and Chrome 65. It was by no means a comprehensive test but I navigated to a half dozen pages in each, and images and everything loaded fine. The only issue I have ATM is that the search in @sharun-s branch has some sort of incompatibility with Chrome, it will complete the search but you cant actually navigate to any of the found pages.

[sharun-s]

@nixcamic open devtools in chrome and see if it shows and error. Off the top of my head, you probably need to enable the --allow-file-access-from-files flag when starting a Chrome session.

offtopic - @mossroy something odd going on with my email notifications. I am getting an email on all kinds of things. Before I would just get when my name was mentioned. Any idea how I can revert to that state?

[Jaifroid]

@nixcamic According to @sharun-s's https://github.com/kiwix/kiwix-js/issues/292#issuecomment-356799553 there is or was a bug with Firefox using XHR range requests via the file:// protocol. I understand that you are accessing the ZIM, at least, over a network (presumably same domain or no cross-origin issues), so this may be why the range requests are working for you, which is fortunate. Are you accessing the app's index.html with localhost, with a URL in the same overall domain as the ZIM, or with file://? Note that Chrome does not work out-of-the-box with file:// access (though you can set the flag @sharun-s mentions), but other browsers do.

@sharun-s , is there any advance on using your method to create packaged apps for Kiwix JS? (I should probably ask this in #292, but answer may pertain to this issue as well.)

[sharun-s]

@Jaifroid I don't know about packaged apps but standalone/url mode appears to be working on Firefox. I think they fixed one of the bugs. It's been a while since I looked at the code so a bit hazy on the details. Plan to do some work on the code this month and will update you guys if there is something I notice.

[Jaifroid]

Thanks @sharun-s . It'd be very exciting if we've got to the point where we could provide a Wikivoyage or WikiMed version as a standalone, which doesn't require picking the ZIM or downloading from an app store. E.g. run WikiMed from a USB stick without having to install anything, at least in Firefox and Edge if not in Chrome.

[sharun-s]

@Jaifroid it's pretty simple to do. You just need to create/modify a desktop browser shortcut with the url to open. Just have to mention the page and zim in the url parameters. I have a couple shortcuts on my desktop to stackoverflow articles and some wiki articles.

[nixcamic]

@sharun-s I don't think --allow-file-access-from-files will help since I'm accessing it over HTTP not from file://. On a somewhat unrelated note, I can't get my .zim file to show up in the list of known files on the load page, even though I added it to the list in zimArchiveLoader.js. Is there somewhere else it needs to be also?

@Jaifroid I'm accessing via localhost (And over the network via my computers hostname).

[sharun-s]

@nixcamic Ah sorry misread your comment. Didn't realize you were trying my branch with a local server setup. Either way let me know what error you are seeing on your server and in devtools when you search.

That load page is just a hacked up proof of concept :( Basically there are two lists. One is the list of archive objects local to your system (knownArchives in zimArchiveLoader.js). And the list of all known available archives/zims (catalogue in library.js). In URL mode (which means you don't use the fileSelector and create a File object pointing at the zim) a pseudo archive/file object needs to be created. The list (knownArchives) in zimArchiveLoader.js is basically a short list of those objects (really the zim's on my machine). That list in future will/should be dynamically generated as files are added and deleted. Right now it is hardcoded just to keep things simple. The (detected archives on disk)UI is generated by comparing the names in the two lists. Basically add/modify an entry in catalogue array in library.js and you should see it show up. Rather than hijacking this thread further I have created an issue against my repo here where you can post anyother questions related to that branch.

[mossroy]

@nixcamic : I still don't understand why kiwix-serve would not be a better solution for you. If your content is already hosted on a server, you might run kiwix-serve on it, instead of exposing the ZIM files through an HTTP server. Kiwix-serve would handle reading the ZIM files (on the server side), and expose their content to any browser on your local network.

@sharun-s : the email notifications you get are probably due to the fact that you are a member of the kiwix group, and are by default a watcher of new repositories. You can adjust the notifications of github, by unwatching some repositories you are not interested in, on https://github.com/watching. See https://help.github.com/articles/watching-and-unwatching-repositories/

[sharun-s]

@mossroy oh I see now. Thanks!

And yes agree, for local serving via HTTP my branch isn't built for that. Just for standalone/url mode. It is using HTTP range requests, so theoretically possible, but I haven't looked at the code in a while and can't really say how much work would be required. Ideally it happens on a separate branch.

[nixcamic]

@mossroy I'm serving the site from a basic router (32mb ram) that can only host static HTTP, I guess I could try and cross compile kiwix-serve and see if it can run on the very limited hardware available.

[mossroy]

@nixcamic : before compiling yourself, try to see if the pre-compiled nightly builds might run on it : https://download.kiwix.org/nightly/ . If you're lucky, your router uses a supported architecture, like armhf.

[nixcamic]

@mossroy Nope, its a MIPS.

[kelson42]

@Jaifroid This is basically a duplicate of #659 you have closed, so I wonder if we should keep that one?