UI Design Suggestion of ExplainaBoard Client

pfliu-nlp commented 2 years ago

Current Implementation (in the latest PRs)

from explainaboard_client.client import ExplainaboardClient
from explainaboard_client.config import Config

client = ExplainaboardClient(Config(
    "user_email",
    "api_key",
    "environment"
))

result= client.evaluate_system_file(
    system_output_file="sst2-output.txt",
    system_output_file_type="text",
    task="text-classification",
    system_name="test_cli",
    metric_names=["Accuracy"],
    source_language="en",
    target_language="en",
    dataset="sst2",
    split="test",
    sh

Unfriendly Points

(1) explainaboard_client.client: too long and verbose and no one wants to remember this
(2) from explainaboard_client.config import Config: the additional burden to remember this (importing Config)
(3) users need to import two lines of unfamiliar packages, which are not good
(4) Config requires too much information, for example, could we remove the need of email or even remove the config!

Some potential better designs in my mind (from worse -> better)

1.

import os
from explainaboard_client import ExplainaboardClient

explainaboard_client.api_key = os.getenv("API_KEY") # api_key could be a global/system variable

result= ExplainaboardClient.evaluate_system_file(
    system_output_file="sst2-output.txt",
    system_output_file_type="text",
    task="text-classification",
    system_name="test_cli",
    metric_names=["Accuracy"],
    source_language="en",
    target_language="en",
    dataset="sst2",
    split="test",
    ..)

2.

import os
from explainaboard_client

explainaboard_client.api_key = os.getenv("API_KEY") 

result= explainaboard_client.ExplainaboardClient.evaluate_system_file(...)

3.

import os
from inspiredco import ExplainaboardClient  # from  inspiredco import EaasClient in the future

inspiredco.api_key = os.getenv("API_KEY")

result= ExplainaboardClient.evaluate_system_file(...)

3.14

import os
from inspiredco

inspiredco.api_key = os.getenv("API_KEY")

result= inspiredco.ExplainaboardClient.evaluate_system_file(...)

neubig commented 2 years ago

Thanks @pfliu-nlp! I think that all the comments here are valid.

I think we should definitely remove Config, I'll do that in a follow-up PR.

Regarding the other design decisions, I think it's reasonably easy to separate the discussion of which functions should be provided (the focus of the current PRs) and how they should be provided (the focus of your discussion here). So as a way to move forward I'd suggest that we review/merge the current PRs, and then after that decide about the design decisions in this issue. Does that sound reasonable?

pfliu-nlp commented 2 years ago

So as a way to move forward I'd suggest that we review/merge the current PRs, and then after that decide about the design decisions in this issue. Does that sound reasonable?

Sounds good!

pfliu-nlp commented 2 years ago

Also the time is ripe for us to have the following feature simultaneously in explainaboard web (displaying code block),

neulab / explainaboard_client