Closed djanibekov closed 5 months ago
@djanibekov thank you for your work. Checked, LGTM! The only comment: not sure if we need a tokenized version of schema
vlsp2016_sa_tokenized_seacrowd_text
, for me they are identical. I understand the reason for adding this schema since there are two files: tokenized and not tokenized. But in the end, they look similar. We can delete this tokenized schema or leave it as it is. What do you think @holylovenia?
Let's have both for the sake of completion and clarity.
Closes #344
Checkbox
seacrowd/sea_datasets/{my_dataset}/{my_dataset}.py
(please use only lowercase and underscore for dataset folder naming, as mentioned in dataset issue) and its__init__.py
within{my_dataset}
folder._CITATION
,_DATASETNAME
,_DESCRIPTION
,_HOMEPAGE
,_LICENSE
,_LOCAL
,_URLs
,_SUPPORTED_TASKS
,_SOURCE_VERSION
, and_SEACROWD_VERSION
variables._info()
,_split_generators()
and_generate_examples()
in dataloader script.BUILDER_CONFIGS
class attribute is a list with at least oneSEACrowdConfig
for the source schema and one for a seacrowd schema.datasets.load_dataset
function.python -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py
orpython -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py --subset_id {subset_name_without_source_or_seacrowd_suffix}
.