I have tried to obtain bitext from the JW300 corpus in plain text format. The webpage http://opus.nlpl.eu/JW300-v1.php gives the instruction to use opus-tools to extract bitext from the alignment XML files.
For example, for the language pair English (en) - Burmese (my) I used the following command:
opus_read -d JW300 -s en -t my -wm moses -w jw300.en jw300.my
While the resulting text files have the same number of lines, the alignment seems to be off.
The resulting files look like this:
$ head jw300.??
==> jw300.en <==
Can You Get By for Less ? PRICES keep going in one direction — up !
The soaring cost of living today threatens to wipe out what little savings some have managed to scrape together .
Especially hard hit are people on fixed incomes . Is there anything that you can do to neutralize the impact of rising prices ?
Let us consider approaches to the problem that certain persons have found practical .
Multiple English sentences are aligned to a single Burmese sentence, and some English lines are empty.
If I look at the result of Google Translate on the Burmese part, it looks like all the information is there in principle, but the alignment is off:
Can you get it for a small fee?
Commodity prices are skyrocketing!
Today's rising cost of living threatens to deplete some of their scrap savings.
Particularly disadvantaged are those with a fixed income.
Is there anything you can do to prevent the effects of rising commodity prices?
Let us consider some of the solutions that some have found practical.
Do you have to have it?
When you are thinking of buying something, you will find it helpful to ask, "Is this really what I need?"
For example
The usefulness of a car is not limited to the loss of a car, but to the fact that it can be bought or sold.
Is there anything that can be done to fix this? It looks like even when only considering 1:1 alignments, there is an offset that causes the wrong sentence pairs to align.
jorgtied
commented
1 year ago
issue reported in OpusTools: https://github.com/Helsinki-NLP/OpusTools/issues/18
I have tried to obtain bitext from the JW300 corpus in plain text format. The webpage http://opus.nlpl.eu/JW300-v1.php gives the instruction to use opus-tools to extract bitext from the alignment XML files.
For example, for the language pair English (en) - Burmese (my) I used the following command:
opus_read -d JW300 -s en -t my -wm moses -w jw300.en jw300.my
While the resulting text files have the same number of lines, the alignment seems to be off.
The resulting files look like this:
$ head jw300.?? ==> jw300.en <== Can You Get By for Less ? PRICES keep going in one direction — up ! The soaring cost of living today threatens to wipe out what little savings some have managed to scrape together .
Especially hard hit are people on fixed incomes . Is there anything that you can do to neutralize the impact of rising prices ?
Let us consider approaches to the problem that certain persons have found practical .
Must You Have It ?
==> jw300.my <== အကုန်အကျ နည်း နည်း ဖြင့် သင် ရ နိုင် ပါ သလော ကုန်ဈန်း သည် လား ရာ တစ်ဖက် တည်း ဖြစ် သော အထက်သို့ သာ တရိပ်ရိပ် တက်နေ သည် ! ယနေ့ လူ နေ ှု စရိတ် မြင့် တက်နေ ခြင်း က အခ သူများ ခြစ် ခြစ် ခြုတ် ခြုတ် စုဆောင်း ထား သည့် စု ငွေ လေး ကုန် သွား စေ ရန် ခြိမ်းခြောက် လျက် ှိ သည် ။ အထူးသဖြင့် ပို ၍ အခက်အခဲ ကြုံ ရ သူများ မှာ ပုံသေ ဝင်ငွေ ရ သူများ ဖြစ်သည် ။ ကုန်ဈန်း မြင့် တက် ခြင်း ၏ ဂယက်ရိုက် ှု ကို တားဆီး ရန် သင် လုပ်ဆောင် နိုင် သည့် အရာ တစ်စုံတစ်ရာ ှိ ပါ သလော ။ လက်တွေ့ ကျသည် ဟု အခ သူများ တွေ့ ှိ ခဲ့ ကြ သည့် ပြဿနာ ဖြေ ရှင်း နည်း များ ကို သုံးသပ် ကြည့် ကြ စို့ ။ ယင်း သည် သင့် တွင် ှိ ဖို့ လို သလော တစ်ခုခု ကို ဝယ် မည် ဟု သင် စဉ်းစား သည့် အခါ “ ဤ အရာ သည် ကပ် အတွက် အမှန် လိုအပ် သလော ” ဟု မေး ခြင်း သည် အကဖြစ်ထွန်း ကြောင်း သင် တွေ့ မြင် ရ ပါ မည် ။ ဥပမာ ၊ ကား မှ ရှိ သော အသုံးတည့် ှုသည် ကား ဈေး ကျ သွား ၍ ဆ ခြင်း ကို မ ဆို ဘဲ ယင်း ကို ဝယ် ခြင်း ၊
Multiple English sentences are aligned to a single Burmese sentence, and some English lines are empty.
If I look at the result of Google Translate on the Burmese part, it looks like all the information is there in principle, but the alignment is off:
Can you get it for a small fee? Commodity prices are skyrocketing! Today's rising cost of living threatens to deplete some of their scrap savings. Particularly disadvantaged are those with a fixed income. Is there anything you can do to prevent the effects of rising commodity prices? Let us consider some of the solutions that some have found practical. Do you have to have it? When you are thinking of buying something, you will find it helpful to ask, "Is this really what I need?" For example The usefulness of a car is not limited to the loss of a car, but to the fact that it can be bought or sold.
Is there anything that can be done to fix this? It looks like even when only considering 1:1 alignments, there is an offset that causes the wrong sentence pairs to align.