tatuylonen / wiktextract

Wiktionary dump file parser and multilingual data extractor
Other
800 stars 82 forks source link

Broken Sense parsing for Translations #47

Closed qixils closed 3 years ago

qixils commented 3 years ago

Using the kaikki.org all-language dump dated 2021-05-08, over 2,700 senses for translations say only "Translations" instead of an actual sense/definition or a null value.

Code used to create a table of all instances:

import json

print("<details>\n<summary>All Instances</summary>\n\n<table>\n<tr>\n<th>Word\n<th>Language\n<th>Translated Word\n<th>Translation Language\n")

with open("kaikki.org-dictionary-all.json", "r") as f:
    for line in f:
        data = json.loads(line)
        if 'translations' in data:
            for trans in data['translations']:
                if 'sense' in trans and trans['sense'] == 'Translations':
                    print("<tr>")
                    for line in [f"<a target=\"_blank\" rel=\"noopener noreferrer\" href=\"https://en.wiktionary.org/wiki/{data['word']}\">{data['word']}</a>", data['lang'], trans['word'], trans['lang']]:
                        print("<td>" + line)

print("</table>\n</details>")
Truncated table of various examples
Word Language Translated Word Translation Language
dag English გოიმი Georgian
original English origina Esperanto
original English öz Crimean Tatar
! Translingual Aiton
Rieti English Rieti French
turnaround English täyskäännös Finnish
roost English ajocar-se Catalan
Laurentian English laurentien French
Laurentian English Laurentien French
souvlaki English souvlaki Polish
third-person plural English 第三人称复数 Chinese
third-person singular English 第三人称单数 Chinese
adiaphoron English adiafora Polish
to English toe Dutch
A-sharp major English ais-duuri Finnish
ko English Mandarin
esoteric English 神秘 Mandarin
major English 專攻 Mandarin
polar coordinates English napakoordinaatit Finnish
nice English حلو Gulf Arabic
offspring English وَلَد Arabic
behold English هَا Arabic
sysop English 系統操作員 Mandarin
quillon English pareerstang Dutch
scarce English نَادِر Arabic
semantic memory English semanttinen muisti Finnish
episodic memory English episodinen muisti Finnish
epi- English մակ- Armenian
chock English carrément French
chock English carrément French
dor English gewone mestkever Dutch
seraglio English serajlo Esperanto
responsible English odpovědný Czech
fugal English fuga Esperanto
cacography English cacographie French
hydration English hydratace Czech
incitation English incitation French
irresolute English tvivlrådig Danish
orthonormal English ortonormální Czech
pomace English bërsi Albanian
kingfish English މުޑަ ހަނދި Dhivehi
intrapreneurship English sisäinen yrittäjyys Finnish
insuperable English непреодолим Bulgarian
knapsack English раница Bulgarian
bad blood English mala sangre Spanish
mediumwave English ona mitjana Catalan
occlusal English oclusal Catalan
extrication English extricación Spanish
orthotropic English orthotropique French
cartilage English kraakbeen Afrikaans
azo- English asó- Irish
sportsmanship English رُوح رِيَاضِيَّة Arabic
Metis English Métis French
pugilistic English pugilistique French
hydrated English hiodráitithe Irish
screensaver English حَافِظ شَاشَة Arabic
unbend English redresser French
mana English 吗那, 魔力, 超自然力量 Mandarin
eye candy English oogpoeding Afrikaans
wariness English prudence French
keelhaul English kielhaal Afrikaans
half volley English cop de sobrebot Catalan
premium English إصدار مُكَلف Arabic
insufflation English innblástur Icelandic
ontic English onties Afrikaans
penicillin English بِنِسِلِين Arabic
jump seat English сгъваема седалка Bulgarian
waterlogged English läpimärkä Finnish
gravitas English powaga Polish
notochord English хорда Bulgarian
workman English عَامِل Arabic
transferee English aistrí Irish
Miami English Myaamia Miami
non-negative English nezáporný Czech
bell-end English gland French
Chin English ချင်းလူမျိုး Burmese
victimization English victimització Catalan
ne plus ultra English nec plus ultra French
be there English er zijn Dutch
polytropic English polytrooppinen Finnish
blazing English ярък Bulgarian
ulceration English ulcerace Czech
codimension English codimension French
recreational drug English 娛樂性藥物 Mandarin
tarsal English nilkkaluu Finnish
transverse English pae Maori
Frenchify English فَرْنَسَ Arabic
maze English обърквам Bulgarian
morpho-syntax English Morphosyntax German
mahasattva English 摩訶薩埵 Mandarin
bodhicitta English ဗောဓိစိတ္တ Burmese
upstream English stroomopwaarts Dutch
kevlar English kevlar French
huntsman English monteiro Galician
monosemic English monosémický Czech
welch English luistaa Finnish
Frankenstein's monster English monstre de Frankenstein French
goop English paćka Polish
goop English paćka Polish
ménage à trois English sesso a tre Italian
escheat English kaduk Polish
race to the bottom English 比爛 Mandarin
seaway English laivareitti Finnish
cochlea English koglea Afrikaans
sorted English sorterad Swedish
getting English obtention French
reeve English graaf Dutch
zygo- English zigo- Catalan
Mollusca Translingual Molluska English
toponymic English toponímic Catalan
synaeresis English synereesi Finnish
synovia English synovie French
welcher English maksamatta jättäjä Finnish
surcingle English surfaix French
sashay English schwingen German
substantiality English substanțialitate Romanian
Hon. English hon. French
epistolary English épistolaire French
spreading English התפשטות Hebrew
Drammen English Drammen Norwegian Bokmål
R & B English არენბი Georgian
double-barrelled English બેનાળી Gujarati
runner bean English 荷包豆 Mandarin
sound bite English clip sonore French
perceptron English percettrone Italian
glop English paćka Polish
glop English paćka Polish
meteoric English meteòric Catalan
nativism English nativismi Finnish
ferrite English ферит Bulgarian
tetrad English tetradi Finnish
oxo- English oxo- Danish
pleading English защита Bulgarian
definitional English definitorisk Danish
muscularity English gespierdheid Dutch
three-cornered English τρίγωνος Ancient Greek
musclebound English мускулест Bulgarian
starburst English flambée d'étoiles French
gamma globulin English gammaglobuliini Finnish
playoff English плейоф Bulgarian
uncomment English odkomentovat Czech
stillness English calma Catalan
wordplay English ahuillahtolli Classical Nahuatl
angry fruit salad English Klickibunti German
main sequence English 主序帶 Mandarin
discontinuity English barney Manx
backroom English Hinterzimmer- German
parallax English paral·laxi Catalan
Oompa Loompa English Umppa-Lumppa Finnish
thixotropy English thixotropie French
lanterloo English 盧牌戲 Mandarin
oka English okka Turkish
vitrification English vitrificació Catalan
intercalation English intercalation French
monobasic English monobasique French
heteroclitic English hétéroclite French
thank-you English bedankje Dutch
Rogers English Rodrigues Portuguese
honed English geslepen Dutch
honk English պիբիբ Armenian
melancholia English меланхолия Bulgarian
lactide English lattide Italian
groaning English valittava Finnish
federate English fédéré French
typed English maschinengeschrieben German
ice dam English presa de hielo Spanish
labellum English label Catalan
polychromy English polychromie French
polychrome English многоцветен Bulgarian
polyembryony English polyembryonie French
date rape English violación por acompañante Spanish
Gautama English ဂေါတမ Burmese
sulfonium English solfonio Italian
abiding-place English woonplaats Dutch
take a hike English stick Swedish
bargaining power English pozycja przetargowa Polish
electrolytic English elektrolytický Czech
beat one's meat English walić konia Polish
sulfurated English solforato Italian
Catawba English Katapa Unami
inviolate English կոյս Old Armenian
monomaniacal English monoman Swedish
bentonite English bentonita Catalan
distributive English წილობითი რიცხვითი სახელი Georgian
diegetic English diegètic Catalan
steely English acerado Spanish
gumball English kauwgombal Dutch
Sirius English Sirius Afrikaans
homestay English 寄宿家庭 Mandarin
reciprocation English réciprocité French
grosgrain English canneté Italian
iconographic English iconografisch Dutch
emptying English vuotamento Italian
Vance English 萬斯 Mandarin
toddy English grog Dutch
locking English lukitus Finnish
out of whack English a la virulé Spanish
furnishing English мебелировка Bulgarian
streaked English durchzogen German
low-maintenance English pflegeleicht German
infatuated English vernarrt German
fireside English lar Galician
toasted English paahdettu Finnish
bolometric English bolometrico Italian
borderline personality disorder English borderline-personlighedsforstyrrelse Danish
pantagruelian English pantagruélique French
raphe English rafe Catalan
kiteboarding English tablaje cometa Spanish
tafferel English peräparras Finnish
measurable function English funcție măsurabilă Romanian
kitesurfer English kiteboarder Spanish
exquisitely English exquisément French
Miao English 苗語 Mandarin
periclinal English periclinale Italian
mastering English Bewältigung German
decarbonize English décarboniser French
grilled English pariloitu Finnish
draining English Austrocknung German
horse blanket English རྟ་ཁེབས Tibetan
steam shovel English dampskovl Danish
patagium English patagio Italian
Pallas English بالاس Arabic
polyketide English polyketidi Finnish
path length English パス長 Japanese
calcitonin English Calcitonin German
missal English missal Catalan
cladogram English cladogramme French
shriveled English ひからびた Japanese
rotisserie English asador Galician
seersucker English gestreept cloqué Dutch
Champlain English Champlain French
radiolocation English radiolokacja Polish
spearfish English paea Maori
graven English gegraveerd Dutch
death throe English agonio Esperanto
parsing English разбор Bulgarian
indefinable English nedefinovatelný Czech
hematocrit English hematokrito Basque
quasicrystalline English quasicristallino Italian
unsorted English osorterad Swedish
pump and dump English 風説の流布 Japanese
fractured English frânt Romanian
marshalling English Marshalling German
heat sink English dissipador tèrmic Catalan
comportment English comportement French
unreality English neamhréaltacht Irish
traversal English Traversal German
quasistatic English quasistatico Italian
bellyful English lán boilg Irish
unexcited English heyecansız Turkish
canon law English droit canon French
homophony English homofonia Catalan
unmitigated English sin paliativos Spanish
submersion English Submersion German
high-definition television English teräväpiirtotelevisio Finnish
indeterminism English indeterminismus Czech
eruptive English eruttivo Italian
orphan drug English weesgeneesmiddel Dutch
inapplicability English inapplicabilité French
disintermediation English dezintermediacja Polish
waste product English afvalstof Dutch
ultrasonography English ultrasonografie Czech
symptomatology English عِلْم اَلْأَعْرَاض Arabic
fuck it English γαμώ το Greek
catch phrase English powiedzonko Polish
clean out English puszczać z torbami Polish
prehistorically English esihistoriallisesti Finnish
longsome English langsome Scots
autoformat English autoformater French
hurdling English překážkový běh Czech
trilateral English եռակողմ Armenian
textually English textlich German
malleolus English mal·lèol Catalan
fixism English fixisme French
mind-bending English оказващ влияние Bulgarian
puerpera English puerpère French
Moab English 摩押 Cantonese
monetarism English 貨幣主義 Mandarin
oidium English oïdi Catalan
Turkish Van English Türkisch Van German
play hardball English iść po trupach Polish
pomology English 果樹學 Mandarin
fancy dress English маскараден костюм Bulgarian
tropicalization English tropicalisation French
haplo- English haplo- Catalan
forty-ninth parallel English 49^e parallèle French
physio- English fyzio- Czech
runcinate English roncinato Italian
Washingtonian English вашингтонски Macedonian
without a doubt English 毫無疑問 Mandarin
crypto- English kripto- Afrikaans
unsearchable English undurchsuchbar German
positronic English positronový Czech
unpin English désépingler French
redistillation English ridistillazione Italian
grand jury English 大陪審團 Cantonese
inhomogeneity English neamh-aonchineálacht Irish
Swiss German English schweizertysk Danish
anionically English anionicamente Portuguese
echo sounding English ecoscandaglio Italian
Uusimaa English Uusimaa Finnish
landgraviate English landgrevskab Danish
lord protector English rigsforstander Danish
aminophenol English amminofenolo Italian
coronene English Mandarin
introrse English introrso Italian
pappus English pappus French
keratinize English sarveistua Finnish
Lancastrian English lancastriano Italian
Lancastrian English lancastriano Italian
interpleader English peticionario Spanish
wet work English мо́крое де́ло Russian
chiro- English chiro- Italian
near miss English per un pelo Italian
might as well English równie dobrze móc Polish
planimetric English επιπεδομετρικός Greek
ryegrass English որոմ Armenian
virtual machine English 虛擬機 Mandarin
Doggerland English Agderia Norwegian
five-second rule English 3秒ルール Japanese
inferiorly English inferiormente Italian
tricuspid English tricuspide Italian
mycorrhization English micorizació Catalan
globin English globina Catalan
otherness English различност Bulgarian
plural noun English ainmear iolra Scottish Gaelic
editability English modifiabilité French
off one's game English no dar pie con bola Spanish
levitical English levitico Italian
otherworld English alén Galician
gigaton English gigatuna Czech
man child English zangolotino Spanish
by definition English par définition French
swap out English échanger French
swap in English einlagern German
twizzle English твизл Russian
mythicize English mythifier French
e-grade English e-aste Finnish
freemium English فريميوم Arabic
tea dance English thé dansant French
tridactyl English kolmisorminen Finnish
microanatomy English microanatomia Italian
traversable English traversable French
heatshield English Hitzeschild German
unifacial English unifacciale Italian
morra English morra Italian
stratiform English stratiforme Italian
optimizer English ოპტიმიზატორი Georgian
unmatchable English inigualable Spanish
hypodermic needle English kanyle Danish
megastructure English mégastructure French
poloidal English poloïdal French
Shina English शीना Hindi
miliary English miliar Catalan
constitutionalize English constitutionaliser French
ecchi English eĉio Esperanto
Italianism English italianismus Czech
Suren English Σουρην Ancient Greek
gesturality English gestualità Italian
throwable English lançable French
toothlessness English tişsizlik Crimean Tatar
tyrosinemia English tyrosinemia Finnish
Maple Leaf English Unifolié French
converb English 副动词 Mandarin
Araucanian English araucano Spanish
headhouse English stacidomo Esperanto
Kleene star English Kleenen tähti Finnish
in trouble English in Schwierigkeiten German
interrex English interrex Dutch
MVC English Modell-Präsentation-Steuerung German
Asturia English Asturie Czech
ultrafilter English ultrafiltre French
cursorial English juoksu- Finnish
salad oil English 沙拉油 Mandarin
verbalist English verbalista Spanish
multiton English Multiton German
viscounty English beskonted Breton
Saint Charles English Saint Charles French
cheki English چكی Ottoman Turkish
neogothic English neogòtic Catalan
Open Game English otevřené zahájení Czech
back of one's mind English w tyle głowy Polish
polycentrism English ポリセントリズム Japanese
aminosalicylic acid English aminosalisyylihappo Finnish
fuckery English tyfuszooi Dutch
sabotiere English sorbetière French
capitulary English capitulario Spanish
gayish English гейский Russian
work wonders English wonderen doen Dutch
amylin English amilina Portuguese
scansorial English grimpeur French
go great guns English ir viento en popa Spanish
mombin English mombin French
walk away English yuragool wat Nyunga
constitutionalization English constitutionnalisation French
fecundability English fécondabilité French
turkeycock English indiot Catalan
exteriorization English extériorisation French
fooster English fústar Irish
make one's mark English jättää jälkensä Finnish
contrarotation English controrotazione Italian
kiss goodbye English heittää hyvästit Finnish
biopolitics English biopolitiikka Finnish
mud bath English кална баня Bulgarian
snaggletooth English dentatura irregolare Italian
have a mind of one's own English olla oma tahto Finnish
I beg your pardon English undskyld hvad? Danish
decarbonization English décarbonation French
mechanotherapeutic English mécanothérapique French
d100 English d100 Finnish
weak verb English svagt verbum Danish
in the game English dalam permainan Indonesian
white sauce English valkokastike Finnish
up above English dans l'au-delà French
magnetic pole English magneettinapa Finnish
Pontic English pontisch German
Conservative Party English 保守黨 Cantonese
rowen English äpäre Finnish
hygrometric English hygrométrique French
auriculo- English auriculo- Spanish
kick over the traces English niskuroida Finnish
kinetoscope English kinétoscope French
padauk English ပိတောက် Burmese
mystacial English mystaciale French
heavy lifting English raskas työ Finnish
muffle furnace English mufalfhoirnéis Irish
Jin English Japanese
antipolitics English antipolítica Portuguese
molybdic English molybdique French
grammaticalization English grammaticalisation French
Legion of Honor English Légion d'honneur French
alimonious English alimentacyjny Polish
sideline cut English poc sleasa Irish
proportional analogy English analogie proportionnelle French
Lamech English לֶמֶךְ Hebrew
Orcus English Orko Esperanto
talcose English talcoso Italian
physicomathematical English fisicomatematico Italian
Sanskritization English sanscritizzazione Italian
sebiferous English sébifère French
Panslavic English panslavisk Swedish
mammee apple English abricó-do-pará Portuguese
gayby English bebé gay Spanish
winter squash English calabaza de invierno Spanish
hormone therapy English endocrinoterapia Italian
chemically imbalanced English químicamente desbalanciado Spanish
chemical imbalance English desbalance químico Spanish
castellany English castellanie French
performativity English перформативност Bulgarian
monochord English monocorde French
Isan English Iszán Hungarian
mashrabiyya English moucharabieh French
saltatorial English sauteur French
Alawite English alaviitti Finnish
first-class object English First-Class-Objekt German
chirograph English cyrograf Polish
Crown English Couronne French
antitoxoplasmal English antitoxoplasmose Portuguese
by any means English de toute façon French
keep at arm's length English distantsi hoidma Estonian
Quaoar English Kvaoaro Esperanto
mother-hive English Leibbiene German
dhokla English ઢોકળું Gujarati
brood sow English мегҗ Kalmyk
cotillage English kyuar Old Welsh
pub. English sumpt. Latin
contraventional English contravvenzionale Italian
put one's house in order English pôr a casa em ordem Portuguese
Black Turtle English 玄武 Mandarin
Three Rivers English Trois-Rivières French
bill and coo English hacer arrumacos Spanish
Rahim Yar Khan English રહીમ યાર ખાન Gujarati
two sides of the same coin English وَجْهَانِ لِعُمْلَةٍ وَاحِدَةٍ Arabic
Borussia English Borussia German
in the same breath English samaan hengenvetoon Finnish
phytonym English fitomínio Portuguese
suckling pig English خنزير الرضاعة Arabic
shield fern English Schildfarn German
Waswanipi English Wâswânipî Cree
Toutatis English Teutates German
free and clear English libre de cargas Spanish
long line English recta larga Spanish
antitaurine English antitaurin French
Hongapore English 香加坡 Cantonese
hitbox English osumalaatikko Finnish
-tropy English -tropie French
Airdrie English An t-Àrd Ruigh Scottish Gaelic
Rhenish Palatinate English Rheinpfalz German
Montrose English Monadh Rois Scottish Gaelic
para-athlete English para-athlète French
commission agent English agent Polish
-otic English -oza Esperanto
Bala English Y Bala Welsh
southernize English méridionaliser French
without a hitch English bez zarzutu Polish
it's a long story English 說來話長 Mandarin
Central German English Mitteldeutsch German
Lojbanistan English Loĵbanujo Esperanto
West Central German English Westmitteldeutsch German
East Central German English Ostmitteldeutsch German
chartered accountant English comptable agréé French
pour in English ἐγχέω Ancient Greek
Alexandrianism English alessandrinismo Italian
fiscalism English Fiskalismus German
Poo English Po Asturian
Montaigne English Montaigne French
macro language English Makrosprache German
distance function English Distanzfunktion German
Kachin English ကချင် Burmese
gastrorrhaphy English ศัลยกรรมซ่อมกระเพาะ Thai
necrophagic English necrofagico Italian
optimeter English ottimetro Italian
one-handedness English Einhändigkeit German
possibilist English possibiliste French
radiopharmacy English radiopharmacie French
whistle for English kysellä turhaan Finnish
monotonize English monotoniser French
parabellum English parabellum French
cold brew coffee English cold brew Polish
bussy English cuceta Portuguese
say no more English škoda slov Czech
dirty kitchen English dalikan Kapampangan
Zygentoma Translingual silberfisch German
pear wine English vin de poire French
pro memoria English pro memoria Latin
prepositus English praepositus Latin
Nottinghamian English nottinghamien French
Costa English Costa Catalan
third country English třetí země Czech
state aid English valtionapu Finnish
namepiece English objet éponyme French
Gagnon English Gagnon French
post-irony English postironi Swedish
break one's back English urakoida Finnish
land agent English aibhéardaí Irish
slice of the pie English pala kakusta Finnish
LTS English LTS Portuguese
rub someone's nose in English wypominać Polish
Taxila English 𐨟𐨐𐨿‫𐨮𐨭𐨁𐨫 Gandhari
have it out English διαπληκτίζομαι Greek
retropropulsion English retropropulsión Spanish
revagination English revaginazione Italian
Vachon English Vachon French
epilanguage English épilangue French
Theodosian English theodosianisch German
Poundmaker English Pitikwahanapiwiyin Plains Cree
triangulene English trianguleno Spanish
fire triangle English palokolmio Finnish
Blouin English Blouin French
seigneuresse English seigneuresse French
Laguardia English Laguardia Italian
caterpillar fungus English 冬蟲夏草 Mandarin
Trumpistan English تْرَامْبِسْتَان Arabic
mosaiculture English mosaïculture French
new wine in an old bottle English neuer Wein in alten Schläuchen German
table hockey English pöytäjääkiekko Finnish
Babelic English babélique French
defense attorney English puolustusasianajaja Finnish
speedskiing English ski de vitesse French
common currency English yleinen tapa Finnish
banked slalom English banked slalom Danish
iced capp English capp glacé French
compile-time English bei der Kompilierung German
dark kitchen English spookkeuken Dutch
pterygo- English pterigo- Spanish
push over English whakahinga Maori
ball hockey English hockey-balle French
smoked beef English boeuf fumé French
mystery guest English mysteryguest Dutch
fuck this English baise ça French
spiral worm English piri ngongo Maori
Chinese macaroni English macaroni chinois French
Ometepec English Ndyöve San Pedro Amuzgos Amuzgo
pagoda flower English dagtong Cebuano
know how to pick 'em English weet ze wel uit te kiezen Dutch
Korolev cross English Крест Королёва Russian
South African Republic English Zuid-Afrikaansche Republiek Afrikaans
Ingen English Ingen Dutch
ribsteak English entrecôte / bifteck de côte / entrecôte sur os / bifteck de côte sur os French
Grey Cup English Coupe Grey French
free electron English vapaa elektroni Finnish
porter's lodge English 門房 Mandarin
Tetreaultville English Tétreaultville French
gender role English 性別角色 Mandarin
Liberal Party English Parti libéral French
shed blood English przelewać krew Polish
Qaidam English 柴達木 Cantonese
caninoid English caninoïde French
caninoid English caninoïde French
make an entrance English faire son entrée French
Lacaille English Lacaille French
avianize English avianiser French
take someone at their word English ottaa todesta Finnish
bind hand and foot English kahlita Finnish
Chinese virus English 中國病毒 Mandarin
Boisbriand English Boisbriand French
liner lock English liner lock -veitsi Finnish
frame lock English frame lock -veitsi Finnish
from top to bottom English läpikotaisin Finnish
cosom hockey English hockey cosom French
stone bow English arbalète à jalet French
Frum English Fromm German
BNO English 英國國民(海外) Mandarin
Okinotorishima English 沖ノ鳥島 Japanese
Mahia English Māhia Maori
Nowy Targ English Nowy Targ Polish
Tempelhof English Tempelhof German
Rosemount English Rosemont French
primary tense English päätempus Finnish
secondary tense English sivutempus Finnish
heavy artillery English ծանր հրետանի Armenian
airgasm English airgasme French
24 Sussex English 24 French
tatuylonen commented 3 years ago

Thanks for the detailed report, code and table that made it easy to analyze!

This is happening because the translation table (e.g., in "dag") has "Translations" in the header where the sense usually goes. I think the simplest solution is to check if the sense taken from the table header is "Translations", and if so, then just make it empty, as there is really no sense description available in those cases.

I already implemented this fix and it seems to work, at least for "dag". The pre-extracted data should have this fixed tomorrow (unless something goes wrong).

BTW, there will also be a new download link for the actual raw output of wiktextract (wiktwords) at the bottom of https://kaikki.org/dictionary/ starting tomorrow.

qixils commented 3 years ago

it's no problem at all, thanks for the quick fix :)