Closes #228 | Add M3LS dataloader

[sabilmakbar]

Please name your PR title and the first line of PR message after the issue it will close. You can use the following examples: Closes #228

Checkbox

• [x] Confirm that this PR is linked to the dataset issue.
• [x] Create the dataloader script seacrowd/sea_datasets/{my_dataset}/{my_dataset}.py (please use only lowercase and underscore for dataset folder naming, as mentioned in dataset issue) and its __init__.py within {my_dataset} folder.
• [x] Provide values for the _CITATION, _DATASETNAME, _DESCRIPTION, _HOMEPAGE, _LICENSE, _LOCAL, _URLs, _SUPPORTED_TASKS, _SOURCE_VERSION, and _SEACROWD_VERSION variables.
• [x] Implement _info(), _split_generators() and _generate_examples() in dataloader script.
• [x] Make sure that the BUILDER_CONFIGS class attribute is a list with at least one SEACrowdConfig for the source schema and one for a seacrowd schema.
• [x] Confirm dataloader script works with datasets.load_dataset function.
• [x] Confirm that your dataloader script passes the test suite run with python -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py or python -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py --subset_id {subset_name_without_source_or_seacrowd_suffix}.
• [ ] If my dataset is local, I have provided an output of the unit-tests in the PR (please copy paste). This is OPTIONAL for public datasets, as we can test these without access to the data files.

[sabilmakbar]

Heads-up for reviewers:

1. Total data is ~14GB
2. The _generate_examples requires file reconstruction and validation (due to the nature of scraped data requires more validation), in my machine it only generate around 3-4 examples/s and has ~57K examples in total
3. It has a warning on GitGuardian (secret detection) due to the GDrive API key being used for constructing a download URL from GDrive (with bypass to large file download warning). Other methods that I tested resulted in failure.

update: the code now is optimized, can creates the examples in much better pace (300ex/s in my machine)

update 2: I find using gdown now works. prob the issue back then related to URL construction that results a warning HTML being downloaded instead of the actual file

[gitguardian[bot]]

️✅ There are no secrets present in this pull request anymore.

If these secrets were true positive and are still valid, we highly recommend you to revoke them. Once a secret has been leaked into a git repository, you should consider it compromised, even if it was deleted immediately. Find here more information about risks.

---

^{_{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}}

[sabilmakbar]

pushed-force to remove secrets in earlier commits

[holylovenia]

Hi @patrickamadeus, I would like to let you know that we plan to finalize the calculation of the open contributions (e.g., dataloader implementations) in 31 hours, so it'd be great if we could wrap up the reviewing and merge this PR before then.

cc: @sabilmakbar

[sabilmakbar]

done making changes, pls have a look @fhudi

[fhudi]

@sabilmakbar cool. thanks. The test is currently running, since it is a 15.9GB file, please wait for a while

[fhudi]

@sabilmakbar Other than the above mentioned issue, everything else work fine. Passed the test and reviewing check-list. 🙏

[sabilmakbar]

Pillow library is required. Shall we add try-except for checking?

Hi @fhudi, I thought PIL was included in SEACrowd reqs (but in fact it isn't). I'll add the try-catch exception for now so that every user can avoid downloading it only to find the error when generating the dataset due to missing PIL lib. Thanks for pointing this