Closes #569 | Add/Update Dataloader MSL4Emergency

[sabilmakbar]

Please name your PR title and the first line of PR message after the issue it will close. You can use the following examples:

Closes #569

Checkbox

• [x] Confirm that this PR is linked to the dataset issue.
• [x] Create the dataloader script seacrowd/sea_datasets/{my_dataset}/{my_dataset}.py (please use only lowercase and underscore for dataset folder naming, as mentioned in dataset issue) and its __init__.py within {my_dataset} folder.
• [x] Provide values for the _CITATION, _DATASETNAME, _DESCRIPTION, _HOMEPAGE, _LICENSE, _LOCAL, _URLs, _SUPPORTED_TASKS, _SOURCE_VERSION, and _SEACROWD_VERSION variables.
• [x] Implement _info(), _split_generators() and _generate_examples() in dataloader script.
• [x] Make sure that the BUILDER_CONFIGS class attribute is a list with at least one SEACrowdConfig for the source schema and one for a seacrowd schema.
• [x] Confirm dataloader script works with datasets.load_dataset function.
• [x] Confirm that your dataloader script passes the test suite run with python -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py or python -m tests.test_seacrowd seacrowd/sea_datasets/<my_dataset>/<my_dataset>.py --subset_id {subset_name_without_source_or_seacrowd_suffix}.
• [ ] If my dataset is local, I have provided an output of the unit-tests in the PR (please copy paste). This is OPTIONAL for public datasets, as we can test these without access to the data files.

[sabilmakbar]

Somehow, I still get error when running .load_dataset or testing via tests.test_seacrowd, the error is:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 11: character maps to <undefined>.

Do you know why this happens? Or is everything fine (OK/passed) on ​​your side? @sabilmakbar

Hmm, on my end it works fine. You can check this log file, which is a sh output from this cmd: python -m tests.test_seacrowd seacrowd/sea_datasets/msl4emergency/msl4emergency.py

test_msl4emergency.log

I'm thinking that your error might come from these lines (in L114-115):

with open(text_path, "r") as f:
    text_data = [data.split("\t") for data in f.readlines()]

You can try to print the text_path var and load it separately on your end to see if that's indeed the case.

Update: Oh perhaps it's related to _CITATION strings (if the line indicator is referring to the string decoding error in Citation lines).

let me know any of these solved the bug on this code @muhammadravi251001

[muhammadravi251001]

It's not about the _CITATION. After changing the encoding parameter on open to:

with open(text_path, "r", encoding="utf-8") as f:

I get an OK response and get the dataset perfectly with load_dataset() method.

The problem is: that I run this dataloader implementation on my Windows notebook, and Windows does not always run encoding on utf-8 in my case; while Linux and MacOS default is utf-8.

To avoid this future-encoding-error, I recommend adding an encoding value to the open method, @sabilmakbar .

[sabilmakbar]

thanks! done making changes, pls have a look @muhammadravi251001

[holylovenia]

Hi @ljvmiranda921, sorry for the sudden notice. I assigned you here in place of @luckysusanto due to the lack of response.

I would like to let you know that we plan to finalize the calculation of the open contributions (e.g., dataloader implementations) in 31 hours, so it'd be great if we could wrap up the reviewing and merge this PR before then.

[ljvmiranda921]

Sure! I'll just update the small nits myself as well just to speed things up! 👍