Closed acul3 closed 2 years ago
Approving this. Thanks @acul3 for your contribution!
hi @holylovenia yes. i'm just following the step from oscar tool paper...you can read it here at section 1 and 3
tldr this just to keep "line by line-oriented" text but also not destroy document boundary
hi @holylovenia yes. i'm just following the step from oscar tool paper...you can read it here at section 1 and 3
tldr this just to keep "line by line-oriented" text but also not destroy document boundary
Noted, @acul3! Thanks for explaining.
Close #223
ccepted config name format:
list availablle year = ["2016",2017","2018","2019","2020","2021","2022","all"]
if you use
all
, it will load all available yearexample: say if you want to load news for year 2016 only and nusantara source ,the code will look like:
test unit
Checkbox
nusantara/nusa_datasets/my_dataset/my_dataset.py
(please use only lowercase and underscore for dataset naming)._CITATION
,_DATASETNAME
,_DESCRIPTION
,_HOMEPAGE
,_LICENSE
,_URLs
,_SUPPORTED_TASKS
,_SOURCE_VERSION
, and_NUSANTARA_VERSION
variables._info()
,_split_generators()
and_generate_examples()
in dataloader script.BUILDER_CONFIGS
class attribute is a list with at least oneNusantaraConfig
for the source schema and one for a nusantara schema.datasets.load_dataset
function.python -m tests.test_nusantara --path=nusantara/nusa_datasets/my_dataset/my_dataset.py
.