Open madenindya opened 1 year ago
Hi, I found many difficulties with this dataset and would like to have some advice:
I mark my line of codes as TODO
for the things that need further discussion
Additional Questions:
Hi, I found many difficulties with this dataset and would like to have some advice:
- The data source is from Kaggle, how do I download it here? The original .zip size is huge. In local, I only download Indonesian data and use it for test reference.
- When creating entities offset, I found it hard to get as the original data didn't show that. When I tried to make it on my own, it was also hard to infer as there are some exact words in a sentence.
- And because I couldn't get the offset, I don't know whether it's better to have a different EntID for the same word in the same sentence. As it might come from the same/different position word, but I don't have that info.
I mark my line of codes as
TODO
for the things that need further discussionAdditional Questions:
- Why the entites.text is List & not just a single string? (context: schema's KB)
Same like bottleneck that I faced.
cc: @SamuelCahyawijaya @holylovenia @bryanwilie
I think you can continue discussing it in slack for the faster response kak @madenindya, thank you!
@muhsatrio : Sorry I missed this PR. Has this one been finalized? I can check it right away
@muhsatrio : Sorry I missed this PR. Has this one been finalized? I can check it right away
I think still there is no any changes kak
IMO, for this dataset we can just implement the source schema for now, as it will be complicated to extend it to the KB schema. What do you think? @madenindya @muhsatrio
Closes #305
Checkbox
nusantara/nusa_datasets/my_dataset/my_dataset.py
(please use only lowercase and underscore for dataset naming)._CITATION
,_DATASETNAME
,_DESCRIPTION
,_HOMEPAGE
,_LICENSE
,_URLs
,_SUPPORTED_TASKS
,_SOURCE_VERSION
, and_NUSANTARA_VERSION
variables._info()
,_split_generators()
and_generate_examples()
in dataloader script.BUILDER_CONFIGS
class attribute is a list with at least oneNusantaraConfig
for the source schema and one for a nusantara schema.datasets.load_dataset
function.python -m tests.test_nusantara --path=nusantara/nusa_datasets/my_dataset/my_dataset.py
.