Closed fozziethebeat closed 1 year ago
I merged the loaders.
Can you give advice on the preferred config names?
With the merge, I used:
code_mixed_jv_id_source
-> All the datacode_mixed_jv_id_jv_nusantara_text
-> jv sentiment datacode_mixed_jv_id_id_nusantara_text
-> id sentiment datacode_mixed_jv_id_nusantara_t2t
-> mt dataThe loaders all work but the unittest library doesn't like this since there's no config name substring that includes the two tasks + source so any subset_id
value ends up with some constructed config name being missing.
/test dataset=code_mixed_jv_id subset_id=code_mixed_jv_id_id
Check test log here: https://github.com/IndoNLP/nusa-crowd/actions/runs/3160630960
Checkbox
nusantara/nusa_datasets/my_dataset/my_dataset.py
(please use only lowercase and underscore for dataset naming)._CITATION
,_DATASETNAME
,_DESCRIPTION
,_HOMEPAGE
,_LICENSE
,_URLs
,_SUPPORTED_TASKS
,_SOURCE_VERSION
, and_NUSANTARA_VERSION
variables._info()
,_split_generators()
and_generate_examples()
in dataloader script.BUILDER_CONFIGS
class attribute is a list with at least oneNusantaraConfig
for the source schema and one for a nusantara schema.datasets.load_dataset
function.python -m tests.test_nusantara --path=nusantara/nusa_datasets/my_dataset/my_dataset.py
.Tested with