Add Nusacrowd filtering module

IndoNLP / nusa-crowd

A collaborative project to collect datasets in Indonesian languages.

Apache License 2.0

261 stars 61 forks source link

Add Nusacrowd filtering module #229

Closed SamuelCahyawijaya closed 2 years ago

SamuelCahyawijaya commented 2 years ago

Add config_helper.py => add module NusantaraConfigHelper and NusantaraMetadataHelper
Standardize dataset => many datasets don't have _LANGUAGES and _LOCAL variables, adding these variables on the currently existing datasets
Move variables from constant variables on test/test_nusantara.py to nusantara/utils/constant.py

bryanwilie commented 2 years ago

I think we also need to change the PR template and the dataloader code template, so it encourages the _LANGUAGES and _LOCAL variables to be filled. But maybe this can be done in a separate PR.