Closed acul3 closed 1 year ago
/test dataset=kopi_nllb subset_id=kopi_nllb_ace_Latn-neardup
Check test log here: https://github.com/IndoNLP/nusa-crowd/actions/runs/3051388439
Not quite familiar with the dataset, but find this sample from test data weird, doesn't look like Acehnese language
{'id': '0', 'text': 'diff geurende lelietje-van-dalen ilan 150ml, ivoor'}
yeah probably some of sample got incorrect language due language identification using LASER model or original source dataset
see 5.2 section on NLLB for detail data gathering
btw for dedup and neardup,, i only take LASER score higher than 0.9
Year, kinda agree with @acul3, I think it is relatively common for data from LASER to be rather noisy especially for such low resource languages. I think we can approve this dataset.
245
accepted config name format:
example: say if you want to load aceh language with neardup format and nusantara source ,the code will look like
test unit
Checkbox
nusantara/nusa_datasets/my_dataset/my_dataset.py
(please use only lowercase and underscore for dataset naming)._CITATION
,_DATASETNAME
,_DESCRIPTION
,_HOMEPAGE
,_LICENSE
,_URLs
,_SUPPORTED_TASKS
,_SOURCE_VERSION
, and_NUSANTARA_VERSION
variables._info()
,_split_generators()
and_generate_examples()
in dataloader script.BUILDER_CONFIGS
class attribute is a list with at least oneNusantaraConfig
for the source schema and one for a nusantara schema.datasets.load_dataset
function.python -m tests.test_nusantara --path=nusantara/nusa_datasets/my_dataset/my_dataset.py
.