Closed zwenyu closed 4 months ago
Hi @zwenyu, thanks for the dataloader submission! Dataloader works fine for me, just have some suggestions and comments
@elyanah-aco Thanks for the comments. I've pushed changes according to the suggestions. For the returned text, I've removed the timestamps.
Hi @zwenyu , the data loader works for me! So no issues with implementation!
As for the timestamps, I think we should include it in the text
. Without it, can we still map the audio to the transcription? If not then maybe the timestamps are relevant 🤔 . Thoughts? @elyanah-aco
@ljvmiranda921 Honestly either is okay for me - I listened to some of the audio files and there's just silence in periods without timestamps. Each utterance is separated by \n
right now which I think is enough. But I defer to what option is most useful to researchers at the end of the day, haha
Each utterance is separated by \n right now which I think is enough
Ah I see if that's the case then I think it's all good. LGTM!
@zwenyu Could you please add
timestamp
undermetadata
in theseacrowd_sptext
schema as well? It should be fine as long as the unit test doesn't raise an error.Added.
Thanks for the addition, @zwenyu!
@elyanah-aco, may I know if this PR is ready for merge?
Closes #442
Checkbox
seacrowd/sea_datasets/{my_dataset}/{my_dataset}.py
(please use only lowercase and underscore for dataset folder naming, as mentioned in dataset issue) and its__init__.py
within{my_dataset}
folder._CITATION
,_DATASETNAME
,_DESCRIPTION
,_HOMEPAGE
,_LICENSE
,_LOCAL
,_URLs
,_SUPPORTED_TASKS
,_SOURCE_VERSION
, and_SEACROWD_VERSION
variables._info()
,_split_generators()
and_generate_examples()
in dataloader script.BUILDER_CONFIGS
class attribute is a list with at least oneSEACrowdConfig
for the source schema and one for a seacrowd schema.datasets.load_dataset
function.python -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py
orpython -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py --subset_id {subset_name_without_source_or_seacrowd_suffix}
.