wandb all dataset structure change - Githubissues

wisdomify / storyteller

An Elsaticsearch-powered forward-dictionary of Korean proverbs

3 stars 0 forks source link

wandb all dataset structure change #11

Closed ArtemisDicoTiar closed 3 years ago

ArtemisDicoTiar commented 3 years ago

TL;DR

current: train, validate, test

changed: train + validate → train.tsv / test → validate.tsv test data is now called from ultimate test queries.

WHY?

ultimate test dataset is added to wandb. Now train + validate should be train set and test should be validate.

WHAT?

changed: train + validate → train.tsv / test → validate.tsv

TODOs

[x] explore script to update all wandb dataset
[x] init_wisdom2def-raw
[x] init_wisdom2eg-raw
[x] init_kuniv_wisdom2eg-raw
[x] wikiquote_wisdom2def-raw
[x] wikiquote_wisdom2eg-raw
[x] namuwiki_wisdom2def-raw
[x] namuwiki_wisdom2eg-raw
[x] opendict_wisdom2def-raw
[x] opendict_wisdom2eg-raw
[x] init_kuniv-preprocess_wisdom_token
[x] init_kuniv-preprocess_no_wisdom

ArtemisDicoTiar commented 3 years ago

DONE