Scrambled/missing Wikifundi data after shrink

[scoutg85]

Hi, I recently made an image that includes Wikifundi. Upon clicking on Wikifundi, I get the following error page:

[a3ce2503712086e0dcd3692b] 2022-06-05 04:43:49: Fatal exception of type Wikimedia\Rdbms\DBQueryError

Is there a way to fix this ? I am going to re-order the image and try flashing it on an SD again.

Side notes: The image is less than 100gb and was burned on a 256gb mini SD.

[kelson42]

@scoutg85 That would be my favourite scenario: corruped SD card or data on SD cars.

[scoutg85]

Made another image with only Wikifundi and the same error occurred.

[a3ce2503712086e0dcd3692b] 2022-06-05 04:43:49: Fatal exception of type Wikimedia\Rdbms\DBQueryError

Any one else with the same issue?

[kelson42]

@scoutg85 What is the http request URL you made to get such an error?

[scoutg85]

Wikifundi, I have tried making an image with wiififundi selected as an option and it's just not working.

On Wed, Jun 8, 2022, 12:10 AM Kelson @.***> wrote:

@scoutg85 https://github.com/scoutg85 What is the http request URL you made to get such an error?

— Reply to this email directly, view it on GitHub https://github.com/offspot/kiwix-hotspot/issues/651#issuecomment-1149436652, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO4FYKAXVTLFVSD2RKRV343VOAMNDANCNFSM5YDZMQUA . You are receiving this because you were mentioned.Message ID: @.***>

[kelson42]

Can you post please your config file here?

[scoutg85]

  Not sure if this is what you are asking for: Image Creation Task        629fd5553c58c8001ef4fd19Scheduler ID       629fd5533c58c8001ef4fcf3Order L1082R629F Config Display:                {    "admin_account": {        "login": "admin",        "password": "obiwan2007"    },    "branding": {        "css": null,        "favicon": null,        "logo": {            "fname": "logo.gif"        }    },    "content": {        "aflatoun": false,        "africatik": false,        "edupi": true,        "edupi_resources": null,        "kalite": [],        "mathews": false,        "nomad": false,        "wikifundi": [            "en"        ],        "zims": ""    },    "language": "en",    "name": "Civil Affairs CIM Cloud",    "project_name": "Civil Affairs CIM Cloud",    "size": "16 GB",    "timezone": "US/Eastern",    "wifi_password": null} Sent from Mail for Windows From: KelsonSent: Wednesday, June 8, 2022 7:18 AMTo: offspot/kiwix-hotspotCc: scoutg85; MentionSubject: Re: [offspot/kiwix-hotspot] Wikifundi Fatal Exception of Type Wikimedia (Issue #651) Can you post please your config file here?—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***> 

[rgaudin]

@scoutg85 thank you for your report. I'll try to reproduce. Was this a Cardshop run?

[scoutg85]

**Update: Tried another image and Wikifundi is still suffering with the same issue: [4bc29c4bf490ff854199c545] 2022-06-09 00:29:57: Fatal exception of type Wikimedia\Rdbms\DBQueryError.

In addition, Edupi is not loading...it gets stuck at "Loading Edupi."

I am going to make another image without Wikifundi just to test Edupi separately.

[Popolechien]

the scraper crashed at 94%

This is unrelated to Wikifundi / the cardshop but relates to this zimfarm job: https://farm.openzim.org/recipes/armypubs_en_all

[Popolechien]

@scoutg85 Correct. Specifically this job here.

[scoutg85]

@Popolechien I removed my previous posts to avoid confusion.

Any idea why the Wikfundi is still not working?

[rgaudin]

@scoutg85 I am working on it and will let you know once we have a fix. I can tell you that your image is indeed not working and that wikifundi data and database are missing/truncated.

[scoutg85]

@rgaudin I appreciate the response. I thought I was doing something wrong.

[rgaudin]

Additional information:

• When flashing your image to card, I get a warning that DATA partition is corrupt.
• Ran the installer locally and didn't get the corrupt warning and WikiFundi EN works as expected.

Logs indicate that the file was properly downloaded (4GB) and was extracted but I noticed that extracting it to the mounted data partition is the last operation before unmounting said partition (given that configuration has no ZIM). My feeling is that the lack of sync() here have the unmount happen before data is actually wired to disk and that's how we end-up with the wikifundi structure but a 0b database and no images.

I am going to add a sync() and test that build on the cardshop worker with the same configuration to validate this hypothesis.

[rgaudin]

Just a head's up: it did not fix the problem. I'll keep you posted. Sorry for the delay.

[rgaudin]

Unfortunately, this is still not resolved. Just a summary of general findings:

• works fine if launched from the host machine of the worker (ie. not inside the Docker container)
• works fine on my laptop both locally and inside a docker container (using that same image, but not launched via the worker code)
• works fine if launched inside the container on worker, but launched manually:

docker run --privileged --entrypoint "" ghcr.io/offspot/cardshop-worker:latest kiwix-hotspot cli --config /output/CACC.json --root

It's still progress but I'm not much happy with it as the worker code hasn't changed in a very long time…

Only difference I noted is that manual tests were not requesting a 256GB card while task was. Testing that now.

[rgaudin]

Only difference I noted is that manual tests were not requesting a 256GB card while task was. Testing that now.

Failed. I now have a relatively small surface to investigate.

[scoutg85]

Only difference I noted is that manual tests were not requesting a 256GB card while task was. Testing that now.

Just to clarify, it will work if the media size is smaller than 256?

[scoutg85]

I just generated an image with wikifundi and it seems to be working again (size 256gb)

[rgaudin]

Thanks for confirming. Findings lead to the culprit being in the shrinkage of non-full images that would be shrunk a bit too much.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be now be reviewed manually. Thank you for your contributions.

[kelson42]

@rgaudin Considering the edge case and the fact that Kiwix-Hotspot 3.0 won't suffer of this problem, I wonder if this ticket will see a real fix coming at some point?

[rgaudin]

Probably not. Closing as per d25db9c