Usability Test (v0.11)

[pfliu-nlp]

aspect-based-sentiment-classification

• [x] :smile: custom dataset
  • dataset
  • system_output
  • Status:
    • Submit: successful
    • Single Analysis: successful

Text Chunking

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: error
  • Reason: inconsistent sample numbers between test set and system outputs

• [x] :sob: datalab dataset

  • dataset: conll00_chunk
    • system output
    • the same error
    • Reason: inconsistent sample numbers between test set and system outputs

Summarization

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: successful
    • Single analysis: failed
  • Reason: unknown

    An error occurred in the analysis.
    [500] The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
    If you found a bug, kindly open an issue on [our GitHub repo](https://github.com/neulab/explainaboard_web). Thanks!

• [x] :sob: datalab dataset

  • https://github.com/neulab/explainaboard_web/issues/308
  • Status:
    • error

named-entity-recognition

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: successful
    • Single analysis: error
    • Reason
      • unknown

        An error occurred in the analysis.
        [500] The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
        If you found a bug, kindly open an issue on [our GitHub repo](https://github.com/neulab/explainaboard_web). Thanks!

• [x] :smile: datalab dataset

  • dataset: conll2003
  • system output
  • Status:
    • Submit: successful!
    • Single analysis: successful!
    • Pair-wise analysis: successful!
    • Case analysis table should be re-formatted.

kg-link-tail-prediction

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: successful
    • Single analysis: error
    • Reason
      • unknown

        An error occurred in the analysis.
        [500] The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
        If you found a bug, kindly open an issue on [our GitHub repo](https://github.com/neulab/explainaboard_web). Thanks!

• [x] :smile: datalab dataset

  • dataset: fb15k_237 (origin)
  • system output
  • Status:
    • Submit: successful!
    • Single analysis: successful!

qa-multiple-choice

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: successful
    • Single analysis: error
    • Reason
      • unknown

        An error occurred in the analysis.
        [500] The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
        If you found a bug, kindly open an issue on [our GitHub repo](https://github.com/neulab/explainaboard_web). Thanks!

• [x] :open_mouth: datalab dataset

  • dataset: fig_qa (large)
  • system output
  • Status:
    • Submit: successful!
    • Single analysis: successful!
    • which subdataset and split should be used?

text-pair-classification

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: successful
    • Single analysis: error
    • Reason
      • unknown

        An error occurred in the analysis.
        [500] The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
        If you found a bug, kindly open an issue on [our GitHub repo](https://github.com/neulab/explainaboard_web). Thanks!

• [x] :sob: datalab dataset

  • dataset: snli
  • system output
  • Status:
    • Submit: failed
    • Reason: unknown

text-classification

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: successful
    • Single analysis: error
    • Reason
      • unknown

        An error occurred in the analysis.
        [500] The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
        If you found a bug, kindly open an issue on [our GitHub repo](https://github.com/neulab/explainaboard_web). Thanks!

• [x] :smile: datalab dataset

  • dataset: sst2
  • system output
  • Status:
    • Submit: successful!
    • Single analysis: successful!

qa-extractive

• [x] :sob: custom dataset

  • dataset
  • system output
  • Status:
    • Submit: successful
    • Single analysis: error
    • Reason
      • unknown

        An error occurred in the analysis.
        [500] The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.
        If you found a bug, kindly open an issue on [our GitHub repo](https://github.com/neulab/explainaboard_web). Thanks!

• [x] :sob: datalab dataset

  • dataset: squad
  • Status:
    • no test system

[pfliu-nlp]

@neubig Some preliminary usability tests!

(BTW, It would be nice if we could automate this part in the future!)

[neubig]

Hey @pfliu-nlp : this is a great usability test. As a preliminary step towards automation, maybe we could run many of these through explainaboard_client? It'd also be nice if we could automatically submit through the web interface as well, but I'm not familiar with the tools that would make it easy to do so.

[lyuyangh]

Yeah, I think that would be very useful. I'm not sure what would be the best way to set up integration tests but it would be nice if the tests submit systems to a separate DB+bucket so we can easily clean them up. We can even run these tests daily if the tests can automatically clean up after themselves. One relatively simple option is to use the testing feature Flask provides but this won't be able to test the py client.