search

common-voice / cv-sentence-extractor

Scraping Wikipedia for fair use sentences
52 stars 52 forks source link

Bad behavior in wikiextractor #200

Open HarikalarKutusu opened 1 year ago

HarikalarKutusu commented 1 year ago

Here is the original from this article:

Antik Yunanca Grekçe: matesis kelimesi matematik kelimesinin köküdür ve bilirim anlamına gelmektedir.

This is the related source:

Antik Yunanca ''{{dil|grc|matesis}}'' kelimesi matematik kelimesinin köküdür ve ''bilirim'' anlamına gelmektedir.

And this is what is extracted (from text/AA/wiki_00 file):

Antik Yunanca ' kelimesi matematik kelimesinin köküdür ve \"bilirim\" anlamına gelmektedir.

Somehow a ' is introduced and the Greek word is dropped. So the sentence has no meaning but except for the ' character, it is OK. As the Greek word is also removed, we also cannot blacklist it.

I'm not sure how many such occurrences would drop into the random 3 selection, but a solution might be good.

PS: I'm aware this is NOT a cv-sentence-extractor issue, but the workflow includes wikiextractor, so...

MichaelKohler commented 1 year ago

Previously filed issue, also with a link to an issue on the WikiExtractor repo: https://github.com/common-voice/cv-sentence-extractor/issues/72

HarikalarKutusu commented 1 year ago

@MichaelKohler, what is the reason for git checkout e4abb4cbd019b0257824ee47c23dd163919b731b for wikiextractor?

There has been a rather recent commit for template handling in that repo, I'm not sure if it is relevant, but as far as I can understand the issue is also related to templates...

(I scanned closed issues but only from titles, I should have looked deeper)

MichaelKohler commented 1 year ago

Sorry for the delay here. Here is the history of the Workflow file before it became wiki.sh: https://github.com/common-voice/cv-sentence-extractor/commits/1106b6851e4f725b3cbea3fc9128ae0af6dcb388/scripts/wiki-extraction.sh . From what I can see, this pinned commit got introduced in https://github.com/common-voice/cv-sentence-extractor/commit/4f120239c79f4e6b20abbef0f81503c4c93ed88a.

Before that, we had https://github.com/common-voice/cv-sentence-extractor/commit/ef4c47f4011550b3f0d2d7515ac66918a76cc391 which tried to fix an issue we ran into because of a new version. This didn't seem to work out nicely and therefore I pinned the version to at least keep the pipeline going as needed.

There are potentially new improvements there with newer versions, very well possible indeed. But I'd guess this would need some work to get it going again.

Alternatively, it seems there are also Cirrus dumps, which claim to include already expanded templates. That might be worth a try as well, though I can't say if there are other issues with that of course.