search

openzim / zim-requests

Want a new ZIM file? Propose ZIM content improvements or fixes? Here you are!
https://farm.openzim.org
37 stars 2 forks source link

mdwiki is failing #1117

Open tim-moody opened 1 month ago

tim-moody commented 1 month ago

Recipe URL

https://farm.openzim.org/recipes?name=mdwiki

Last log lines

"method": "get",
                "url": "https://mdwiki.wmcloud.org/w/api.php?action=query&format=json&prop=redirects%7Crevisions%7Cpageimages&rdlimit=max&rdnamespace=0%7C3000%7C3002&redirects=true&titles=African_histoplasmosis%7CAfrican_tick_bite_fever%7CAfrican_trypanosomiasis%7CAge_and_female_fertility%7CAge_spot%7CAgenesis_of_the_corpus_callosum%7CAggregatibacter_actinomycetemcomitans%7CAggressive_NK-cell_leukemia%7CAggressive_digital_papillary_adenocarcinoma%7CAggressive_fibromatosis%7CAgomelatine%7CAgoraphobia%7CAgranulocytosis%7CAgustin_Walfredo_Castellanos%7CAicardi_syndrome%7CAicardi%E2%80%93Gouti%C3%A8res_syndrome%7CAinhum%7CAir_pollution%7CAir_pollution_forecasting%7CAir_pollution_measurement%7CAir_quality_index%7CAir_quality_law%7CAirborne_transmission%7CAirway_remodelling%7CAkathisia%7CAlagille_syndrome%7CAlastrim%7CAlbendazole%7CAlbinism%7CAlbinism_in_humans%7CAlbright's_hereditary_osteodystrophy%7CAlbuminuria%7CAlcaftadine%7CAlclometasone%7CAlcohol-related_brain_damage%7CAlcohol-related_traffic_crashes_in_the_United_States%7CAlcohol_Dependence_Data_Questionnaire%7CAlcohol_Use_Disorders_Identification_Test%7CAlcohol_abuse%7CAlcohol_and_breast_cancer%7CAlcohol_and_cancer%7CAlcohol_and_cardiovascular_disease%7CAlcohol_and_health%7CAlcohol_dependence%7CAlcohol_detoxification%7CAlcohol_education%7CAlcohol_flush_reaction%7CAlcohol_intoxication%7CAlcohol_law%7CAlcohol_packaging_warning_messages"
        },
        "code": "ERR_BAD_RESPONSE",
        "status": 502
}
[error] [2024-07-20T16:44:43.052Z]

**********

Request failed with status code 502

How many times the recipe failed in a row?

Once

How many ZIM have been produced before failure?

Many

Which action did you undertake so far?

None, I do not have sufficient rights in Zimfarm but I recommend to disable the recipe for now

What's next?

This has to be reported to website admin (upstream website problem)

More details

This was not run on the farm, but I am seeing new 502 errors that I am pursuing.

kelson42 commented 1 month ago

@tim-moody Why we deal with https://mdwiki.wmcloud.org/ in place of mdwiki.org? Can you solve the bad upstream HTTP errors?

tim-moody commented 1 month ago

I made some fixes and reran:

T:12135; finish
[log] [2024-07-21T00:18:18.535Z] Summary of scrape actions: {
        "files": {
                "success": 101952,
                "fail": 132
        },
        "articles": {
                "success": 67992,
                "fail": 734
        },
        "redirects": {
                "written": 267373
        }
}
[log] [2024-07-21T00:18:18.540Z] Finished dump
[log] [2024-07-21T00:18:18.540Z] Closing HTTP agents...
[log] [2024-07-21T00:18:18.540Z] All dumping(s) finished with success.
[log] [2024-07-21T00:18:18.540Z] Flushing Redis DBs
[info] [2024-07-21T00:18:18.540Z] Finished running mwoffliner after [22782s]
[log] [2024-07-21T00:18:18.541Z] Exiting with code [0]
[log] [2024-07-21T00:18:18.541Z] Deleting temporary directory [/dev/shm/mwoffliner-1721498317879]

Looks good to me.

tim-moody commented 1 month ago

Summary of changes:

Jaifroid commented 3 weeks ago

Would you be able to help with this @benoit74? See https://github.com/openzim/mwoffliner/issues/2069#issuecomment-2282540744. In sum, all that's needed is:

It's been a couple of months now since we've been able to release an updated WikiMed app (desktop), due to the maxi recipe using mwOffliner dev instead of 1.13.0 (seemingly).

benoit74 commented 3 weeks ago

I have the power to do it, but with great power comes great responsibilities ^^

Unfortunately for now @kelson42 requested to disable all mwoffliner recipes ... this needs to be discussed with him.

From my PoV, we should reenable all mwoffliner recipes with 1.13 since it looks like 1.14 is unfortunately not yet sufficiently ready and/or tests are not progressing fast enough.

I will bring the discussion to Slack to see if we can change current situation, thank you for insisting a bit.

Jaifroid commented 3 weeks ago

@tim-moody I'm not sure you're on our Slack, or at least on the mwOffliner channel, but the outcome of the discussion there is that we can no longer use 1.13.0 due to a promise made to the Wikimedia people, and although I thought that a recent Wikivoyage ZIM was 1.13.0, it in fact was 1.14-dev, just its metadata were incorrect.

The bottom line is that if mwOffliner is failing to make an mdwiki with the dev branch, then we can't currently fix that, and we'll have to wait to produce any new WikiMed apps till we have a new ZIM. I don't know how important you feel it is to release apps based on the latest MDWiki info. It might be possible to use ZIMs produced by you if we can upload them to download.kiwix.org, because to build the app both ZIMs needs to be there (they are built via GitHub Actions).

I don't know if that's a good idea in practice... Maybe we should wait till the issues with dev have been ironed out, noting that it will need to use the new mobile HTML REST API, as the mobile-sections API is now deprecated.

tim-moody commented 3 weeks ago

@kelson42 thanks very much for your clear explanation. I was not aware of discussions with WMF, and it all makes sense now. I had also noticed that dev reports its version as 1.13.0, which adds confusion.

We are of course working to make mdwiki work with dev, but are not there yet, and my travels have also slowed things down.

I will try building the mdwiki-app recipe to see if that is an option. I would need credentials to upload to download.kiwix.org. I have spoke with James, and worst case he is OK with using a previous ZIM for the app.

Thanks for your ongoing help and support.

Jaifroid commented 3 weeks ago

@tim-moody Although the Android app(s) use mdwiki-app, for the desktop app, I only use it for an old UWP version that cannot handle full-text search. For the Electron app, I use the normal maxi with ft index, so I don't think having just the -app version would help for my side, and I'm not sure how often the Android people release an updated mdwiki app.

In terms of your building with 1.13.0 (the real 1.13, not the dev masquerading as such), personally I don't think that would matter if the API is still working and hasn't been disabled, as one build is nothing compared to the massive hit on the mobile-sections API that running the full ZIM farm on it would represent. But that's your personal decision of course đŸ˜‰.

Regarding credentials, I have these and could upload (so long as we have both versions: app_maxi and normal maxi), but I think we'd need agreement that this is OK as an emergency workaround before proceeding. Would library.kiwix.org pick up a ZIM uploaded to download.kiwix.org automatically? @benoit74?

I'm not sure of the value in releasing a new app with the last available (old) ZIM, if the old app with the same old ZIM is still working fine (which it is). We use a permalink to it in any case: https://kiwix.github.io/kiwix-js-pwa/wikimed/ . It could annoy people and waste their bandwidth!

tim-moody commented 3 weeks ago

@kelson42 @Jaifroid I will take everyone's advice and stop producing ZIMs until we can use dev. In the mean time we can use existing ZIMs and the app.

benoit74 commented 3 weeks ago

Regarding credentials, I have these and could upload (so long as we have both versions: app_maxi and normal maxi), but I think we'd need agreement that this is OK as an emergency workaround before proceeding. Would library.kiwix.org pick up a ZIM uploaded to download.kiwix.org automatically? @benoit74?

Yes, it is fully automated (currently refreshed once per hour at xx:30), also directly usable in our imager service, and all our readers using the OPDS catalog, e.g. Android, Apple, ... so handle with care. Especially since once published, someone might also start to download it, or use it to create a custom hotspot image with our imager service, ... lot's of frustration ahead if not done correctly.