Concrete data type - Githubissues

neulab / ExplainaBoard

Interpretable Evaluation for AI Systems

MIT License

359 stars 36 forks source link

Concrete data type #466

Open odashi opened 2 years ago

odashi commented 2 years ago

FeatureTypes have the dtype member, which specifies the data type of corresponding data. However, this value is represented as str and it is unclear that what kind of values are valid for this member. Particularly, we need to:

Make a reference of valid dtype values. It should be either (1) defined by ourselves (in this repository), or (2) borrowed from other software.
Make a mechanism to restrict dtype member.

RFC: @neubig @pfliu-nlp

neubig commented 2 years ago

Yeah, this definitely seems reasonable to me. An Enum seems reasonable to me, but any other option sounds fine too.

odashi commented 2 years ago

Another problem is that JSON files involve different dtype definitions (one from ExplainaBoard, one from Datalab), which is a main source of confusion of my refactoring. We also need to consolidate them to just one type definition. I recommend not tot use Datalab's one and convert it to ExplainaBoard's data type once we loaded information from Datalab.