Closed Ingerid closed 1 year ago
Feilene i oppgavelisten er fra valideringsrapporten etter å ha kjørt UniversalDependencies/tools/validate.py:
python tools/validate.py --max-err 0 --lang no --level 3 --no-tree-text --no-space-after grew_ndt2ud/data/grew_output_train.conllu 2>&1 | tee validation-report_ndt2ud.txt
Jeg skrev et lite skript for å oppsummere rapporten:
rows = Path("validation-report_ndt2ud.txt").read_text(encoding="utf-8").split("\n")
error_info_regx = re.compile(r"^\[Line (\d+)(?: Node )?(\d+)?\]\: \[(L.*)\] (.*)$", flags=re.DOTALL)
errors = []
for row in rows:
m = error_info_regx.fullmatch(row)
if m is None:
print(row)
continue
errors.append(m.groups())
df = pd.DataFrame(errors, columns=["line", "node", "errortype", "message"])
print("Different types of errors:")
type_counts = df.errortype.value_counts()
print(type_counts)
Oppdatert feilmeldingsliste uten å spesifisere valideringsnivå:
python ../tools/validate.py --max-err 0 --lang no --no-space-after data/grew_output_train.conllu 2>&1 | tee validation-report_ndt2ud.txt
L3 Syntax right-to-left-appos 1328 L4 Morpho feature-upos-not-permitted 449 L3 Syntax rel-upos-advmod 359 L3 Syntax upos-rel-punct 156 L3 Syntax punct-is-nonproj 120 L3 Syntax rel-upos-punct 88 L3 Syntax punct-causes-nonproj 86 L2 Syntax 0-is-not-root 52 L3 Syntax leaf-aux-cop 7 L3 Syntax too-many-subjects 2
En ny språkspesifikk feiltype:
Udapi har funksjonalitet som fikser ganske mange av feilene:
right-to-left-appos
-feilene)
cat data/ndt_nb_train_udmorph.conllu | udapy -s \
ud.SetSpaceAfterFromText \
ud.FixPunct \
ud.FixLeaf \
ud.FixRightheaded \
> out.conllu && mv out.conllu data/ndt_nb_train_udmorph.conllu
Krav:
Lag nye issues for nødvendige endringer: