Broken rendering, mixing content beetween different (TED) ZIM files

[kelson42]

Using the last nightly build of Kiwix for Android I have achieved to catch a new bug.

To reproduce the bug, you need this file: https://download.kiwix.org/zim/ted/tedxgeneva-2014_fr_all_2015-03.zim with the correct following welcome page:

... And the last TED-Business ZIM file: https://download.kiwix.org/zim/ted/ted_en_business_2016-06.zim with the correct following welcome page:

The problem appears if I open the tedx file and then the ted file, by doing so, the welcome page is completely broken with element from tedx AND ted welcome pages. Here it is how it looks like:

If I do the other way around, similar result (broken):

If I close Kiwix and then open a file, everything is OK. I strongly suspect a cache which is not cleared beetween load of different ZIM files within the same tab... and for that reason the json files necessary to the display of the welcome page for both ZIM files is still the other one (same for pictures).

[ghost]

Hello, I am new to the project but I was wondering if I can work on this to get started?

[brijeshshah13]

@mhutti1 @paologarcia002 Sure. Go ahead.

[kelson42]

@paologarcia002 That would be really welcome, as this bug clearly show a flaw in the pages caching system.

[ghost]

What version did this happen on or was this a general issue

[kelson42]

@paologarcia002 It was a nightly. Just try to reproduce it.

[ghost]

Just out of curiosity, but how does a particular item like the ted business item from get rendered within the app?

[mhutti1]

WebView taking HTML page from ZimContentProvider

[ghost]

Thanks

[kelson42]

FYI we have json content loading here. The whole is then rendered in javascript.

[ghost]

Ah sorry about not following up had exams, but I just tried to recreate the problem and for me it doesn’t change at all. For instance I go to tedx then ted business, all the the tedx stuff is there despite me changing to ted business

[ghost]

Also do the jsons/js get sent and rendered from the webview like the html, or is that from jni?

[mhutti1]

Everything goes from the JNI to the Webview. I'm confused about your first comment. It seems to imply you can reproduce the issue.

[ghost]

My bad but for not being clear but for me, I can’t even render ted business after visiting tedx, as if I never left tedx, not even the content, but Kelson’s screenshots show a mix where ted business content is visable, just the wrong header.

[mhutti1]

@paologarcia002 Create a new issue with reproduction steps if that is the case.

[kelson42]

This has not been fixed with #691. I have also remarked that time to time pictures are not rendered properly, I suspect the root cause is the same.

[mhutti1]

@kelson42 I can't reproduce on my own device is it still not working?

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[macgills]

I cannot replicate to echo @mhutti1 's sentiment. @kelson42 please verify that you can not reproduce this issue and if that is the case please close it

[kelson42]

@macgills Trust you, will reopen if it reappears.

[kelson42]

The problem is still there with newly created ZIM files and Kiwix Android 3.0.4. It should be easy to reproduce for you.

See video.

Tested files are:

• https://download.kiwix.org/zim/other/mali-pour-les-nuls_fr_all_2019-09.zim
• https://download.kiwix.org/zim/other/vsauce-language_en_all_2019-10.zim

[kelson42]

@rgaudin Are you also impacted?

[rgaudin]

I'll test but I've had mixed content in the past and it was not related to video ZIMs. I couldn't systematically reproduces so I didn't report it.

[kelson42]

@rgaudin Just try the same steps like I did, always fails for me.

[macgills]

Do you want me to hop off modularisation for a bit to investigate this?

[kelson42]

@macgills no non... or maybe you can just confirm that you have a reprocase? But this is something we will have to fix in 3.1.

[macgills]

I confirm the bug exists using these 2 files. The root cause is a real puzzler because we have set the Zim to read from before any load occurs, webview or kiwixlib being in error seems most likely at this early stage. Or my current usage of kiwixlib is grossly abusive

[macgills]

Looking through the api of JNIKiwixReader I see the method dispose. I have until now been relying on GC to remove these objects from memory. Is dispose necessary?

[kelson42]

@macgills I don't know, but if the kiwix-lib JNI is able to deal only with on ZIM file at a time (which I guess is the case). I would better call it before opening a new/different ZIM file.

[macgills]

I create a new instance of JNIKiwixReader each time I open a zim file (also when scanning files). This issue presents itself just having opened either of these files at any time, I was opening them and interspersing it with opening another zim and only their contents were mangled which to me points to a kiwixlib issue. Whenever I am free from modularisation I will give the dispose call a chance. Additionally I do not particularly understand the reason for JNIKiwix aside from setting the icu file and how that object interacts with the rest of the objects is a mystery to me

[macgills]

the dispose call had no effect, not sure what could be happening

[macgills]

Unnsuccessful load example:

I/kiwix: Attempting to create reader with: /storage/emulated/0/Android/data/org.kiwix.kiwixmobile/files/Kiwix/mindfield_en_all_2019-11.zim
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/A/home.html = text/html
D/ZimFileReader: reading  A/home.html(mime: text/html, size: 2826) finished.
I/chromium: [INFO:CONSOLE(1)] "Uncaught TypeError: Cannot read property 'start' of undefined", source:  (1)
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/A/home.html = text/html
D/kiwix: invalidUrl = false
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/chosen/chosen.min.css = text/css
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/videojs/video-js.min.css = text/css
D/ZimFileReader: reading  I/profile.jpg(mime: image/jpeg, size: 13023) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/profile.jpg = image/jpeg
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/home.css = text/css
D/ZimFileReader: reading  -/assets/videojs/video-js.min.css(mime: text/css, size: 40028) finished.
D/ZimFileReader: reading  -/assets/chosen/chosen.min.css(mime: text/css, size: 10370) finished.
D/ZimFileReader: reading  -/assets/home.css(mime: text/css, size: 570908) finished.
I/chromium: [INFO:CONSOLE(111)] "Uncaught TypeError: Cannot read property 'slice' of undefined", source: content://org.kiwix.kiwixmobile.zim.base/-/assets/db.js (111)
D/kiwix: invalidUrl = false
D/ZimFileReader: reading  I/profile.jpg(mime: image/jpeg, size: 13023) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/profile.jpg = image/jpeg
D/kiwix: Loaded URL: content://org.kiwix.kiwixmobile.zim.base/A/home.html#1
D/kiwix: Loaded URL: content://org.kiwix.kiwixmobile.zim.base/A/home.html#1
D/kiwix: invalidUrl = false

I/chromium: [INFO:CONSOLE(111)] "Uncaught TypeError: Cannot read property 'slice' of undefined", source: content://org.kiwix.kiwixmobile.zim.base/-/assets/db.js (111) this line looks particularly suspicious

as it is notably absent from a successful load

D/ZimFileReader: reading  A/home.html(mime: text/html, size: 2810) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/A/home.html = text/html
I/TextToSpeech: Connected to ComponentInfo{com.google.android.tts/com.google.android.tts.service.GoogleTTSService}
I/Choreographer: Skipped 63 frames!  The application may be doing too much work on its main thread.
I/TextToSpeech: Set up connection to ComponentInfo{com.google.android.tts/com.google.android.tts.service.GoogleTTSService}
D/ZimFileReader: reading  -/assets/data.js(mime: application/javascript, size: 4689) finished.
D/ZimFileReader: reading  -/assets/zim_prefix.js(mime: application/javascript, size: 1828) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/chosen/chosen.min.css = text/css
D/ZimFileReader: reading  -/assets/db.js(mime: application/javascript, size: 2818) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/home.css = text/css
D/ZimFileReader: reading  I/profile.jpg(mime: image/jpeg, size: 7462) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/videojs/video-js.min.css = text/css
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/ogvjs/ogv-support.js = application/javascript
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/zim_prefix.js = application/javascript
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/videojs/video.min.js = application/javascript
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/ogvjs/ogv.js = application/javascript
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/videojs-ogvjs.js = application/javascript
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/profile.jpg = image/jpeg
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/chosen/chosen.jquery.js = application/javascript
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/data.js = application/javascript
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/jquery.min.js = application/javascript
I/OpenGLRenderer: Initialized EGL, version 1.4
D/OpenGLRenderer: Swap behavior 2
D/kiwix: invalidUrl = false
D/ZimFileReader: reading  -/assets/chosen/chosen.min.css(mime: text/css, size: 10370) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/db.js = application/javascript
D/kiwix: Loaded URL: content://org.kiwix.kiwixmobile.zim.base/A/home.html
D/kiwix: TextToSpeech was initialized successfully.
D/ZimFileReader: reading  -/assets/videojs-ogvjs.js(mime: application/javascript, size: 17645) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/-/assets/app.js = application/javascript
D/ZimFileReader: reading  -/assets/jquery.min.js(mime: application/javascript, size: 97163) finished.
D/ZimFileReader: reading  -/assets/home.css(mime: text/css, size: 570908) finished.
D/ZimFileReader: reading  -/assets/videojs/video.min.js(mime: application/javascript, size: 480271) finished.
D/ZimFileReader: reading  -/assets/chosen/chosen.jquery.js(mime: application/javascript, size: 47491) finished.
D/ZimFileReader: reading  -/assets/app.js(mime: application/javascript, size: 5037) finished.
D/ZimFileReader: reading  -/assets/videojs/video-js.min.css(mime: text/css, size: 40028) finished.
D/ZimFileReader: reading  -/assets/ogvjs/ogv-support.js(mime: application/javascript, size: 4142) finished.
D/vndksupport: Loading /vendor/lib64/hw/android.hardware.graphics.mapper@2.0-impl.so from current namespace instead of sphal namespace.
D/vndksupport: Loading /vendor/lib64/hw/gralloc.msm8992.so from current namespace instead of sphal namespace.
D/ZimFileReader: reading  -/assets/ogvjs/ogv.js(mime: application/javascript, size: 148482) finished.
I/zygote64: Compiler allocated 7MB to compile void android.widget.TextView.<init>(android.content.Context, android.util.AttributeSet, int, int)
I/zygote64: Do full code cache collection, code=247KB, data=187KB
    After code cache collection, code=223KB, data=136KB
D/ZimFileReader: reading  I/banner.jpg(mime: image/jpeg, size: 45120) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/banner.jpg = image/jpeg
D/ZimFileReader: reading  I/videos/BupG57U82NI/video.jpg(mime: image/jpeg, size: 51051) finished.
D/ZimFileReader: reading  I/videos/BupG57U82NI/video.jpg(mime: image/jpeg, size: 51051) finished.
D/ZimFileReader: reading  I/videos/vkPj7y7KacY/video.jpg(mime: image/jpeg, size: 56970) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/vkPj7y7KacY/video.jpg = image/jpeg
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/BupG57U82NI/video.jpg = image/jpeg
D/ZimFileReader: reading  I/videos/rIrF9TSbPis/video.jpg(mime: image/jpeg, size: 58381) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/BupG57U82NI/video.jpg = image/jpeg
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/rIrF9TSbPis/video.jpg = image/jpeg
D/ZimFileReader: reading  I/videos/BupG57U82NI/video.jpg(mime: image/jpeg, size: 51051) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/p1TXZDTtOOE/video.jpg = image/jpeg
D/ZimFileReader: reading  I/videos/Y1W67I0Gp8Y/video.jpg(mime: image/jpeg, size: 50752) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/Y1W67I0Gp8Y/video.jpg = image/jpeg
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/_f3DnpvpsUM/video.jpg = image/jpeg
D/ZimFileReader: reading  I/videos/p1TXZDTtOOE/video.jpg(mime: image/jpeg, size: 51700) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/BupG57U82NI/video.jpg = image/jpeg
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/qbO7_ivMldc/video.jpg = image/jpeg
D/ZimFileReader: reading  I/videos/_f3DnpvpsUM/video.jpg(mime: image/jpeg, size: 56619) finished.
D/ZimFileReader: reading  I/videos/qbO7_ivMldc/video.jpg(mime: image/jpeg, size: 53210) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/BupG57U82NI/video.webm = video/webm
V/View: dispatchProvideAutofillStructure(): not laid out, ignoring 2 children of 1073741833
I/AssistStructure: Flattened final assist data: 6468 bytes, containing 1 windows, 25 views
V/View: dispatchProvideAutofillStructure(): not laid out, ignoring 2 children of 1073741833
I/zygote64: Do partial code cache collection, code=249KB, data=156KB
I/zygote64: After code cache collection, code=245KB, data=155KB
    Increasing code cache capacity to 1024KB
I/AssistStructure: Flattened final assist data: 6468 bytes, containing 1 windows, 25 views
D/kiwix: invalidUrl = false
D/ZimFileReader: reading  I/videos/BupG57U82NI/video.jpg(mime: image/jpeg, size: 51051) finished.
D/ZimFileReader: reading  I/videos/BupG57U82NI/video.jpg(mime: image/jpeg, size: 51051) finished.
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/BupG57U82NI/video.jpg = image/jpeg
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/videos/BupG57U82NI/video.jpg = image/jpeg
D/kiwix: Loaded URL: content://org.kiwix.kiwixmobile.zim.base/A/home.html#1
D/kiwix: Loaded URL: content://org.kiwix.kiwixmobile.zim.base/A/home.html#1
D/ZimFileReader: getting mimetype for content://org.kiwix.kiwixmobile.zim.base/I/profile.jpg = image/jpeg
D/ZimFileReader: reading  I/profile.jpg(mime: image/jpeg, size: 7462) finished.
D/kiwix: Loaded URL: content://org.kiwix.kiwixmobile.zim.base/A/home.html#1
D/kiwix: invalidUrl = false

@rgaudin @kelson42 any ideas?

[kelson42]

@macgills I think you should create an automated test for that specific case so we can work on something easier. At the low level.

[macgills]

@kelson42 you are going to have to clarify as I don't know what you mean

[kelson42]

@macgills This bad behaviour should be narrowed down in a "unit test". Without a simplier reproduction case, it will be difficult to understand it. I also want to secure that one time it will be fixed (probably in the libkiwix) then it will never occur anymore.

[macgills]

Any test I write will absolutely not be a unit test seeing as our repro case is

1) is download these 2 specific files 2) open both of them 3) observe a visual error in a webview

There is no lower level for this from the android app perspective

[kelson42]

@macgills forget about download, just assume the files are there and forget the webview, just mockup the content requests which are wrong. Just open the first make one-two content requests and then open the second and do the same (with similar URLs, I guess the problem is around this). If we have somehow a problem, this should be enough to reproduce it. If this is the case, I will build two smaller ZIM files easier to deal with in the futures automated test.

What is sure is that its almost impossible to help you with such a complex case, the problem has to be narrowed down. Then it will be easy to understand it.

[macgills]

The act of reducing the content of these zim files down to their minimum reproducable state is valuable but I don't know how to do that so it is up to you.

I am also not sure how easy it is unless we get it down to literally 1 request as I believe it is the webview that is responsible for successive content requests from the ContentProvider

[kelson42]

@mhutti1 Here are two small ZIM files to test with:

• http://tmp.kiwix.org/youtubetest1_en_all.zim
• http://tmp.kiwix.org/youtubetest2_en_all.zim