IndoNLP / nusa-crowd

A collaborative project to collect datasets in Indonesian languages.
Apache License 2.0
261 stars 61 forks source link

Closes #305 | Create Multilingual open relation dataloader #320

Open madenindya opened 1 year ago

madenindya commented 1 year ago

Closes #305

Checkbox

madenindya commented 1 year ago

Hi, I found many difficulties with this dataset and would like to have some advice:

  1. The data source is from Kaggle, how do I download it here? The original .zip size is huge. In local, I only download Indonesian data and use it for test reference.
  2. When creating entities offset, I found it hard to get as the original data didn't show that. When I tried to make it on my own, it was also hard to infer as there are some exact words in a sentence.
  3. And because I couldn't get the offset, I don't know whether it's better to have a different EntID for the same word in the same sentence. As it might come from the same/different position word, but I don't have that info.

I mark my line of codes as TODO for the things that need further discussion

Additional Questions:

muhsatrio commented 1 year ago

Hi, I found many difficulties with this dataset and would like to have some advice:

  1. The data source is from Kaggle, how do I download it here? The original .zip size is huge. In local, I only download Indonesian data and use it for test reference.
  2. When creating entities offset, I found it hard to get as the original data didn't show that. When I tried to make it on my own, it was also hard to infer as there are some exact words in a sentence.
  3. And because I couldn't get the offset, I don't know whether it's better to have a different EntID for the same word in the same sentence. As it might come from the same/different position word, but I don't have that info.

I mark my line of codes as TODO for the things that need further discussion

Additional Questions:

  • Why the entites.text is List & not just a single string? (context: schema's KB)

Same like bottleneck that I faced.

cc: @SamuelCahyawijaya @holylovenia @bryanwilie

muhsatrio commented 1 year ago

I think you can continue discussing it in slack for the faster response kak @madenindya, thank you!

SamuelCahyawijaya commented 1 year ago

@muhsatrio : Sorry I missed this PR. Has this one been finalized? I can check it right away

muhsatrio commented 1 year ago

@muhsatrio : Sorry I missed this PR. Has this one been finalized? I can check it right away

I think still there is no any changes kak

SamuelCahyawijaya commented 1 year ago

IMO, for this dataset we can just implement the source schema for now, as it will be complicated to extend it to the KB schema. What do you think? @madenindya @muhsatrio