OpenEval

• Project's slack channel.

About

Data scientists working on machine learning problems have historically had several issues relating to evaluating their systems: spending time individually developing evaluation frameworks for tasks, comparing results over time, and keeping evaluations consistent among teams. OpenEval is a system designed to address these problems.

In developing this system we set out to build a centralized, easy-to-use platform for groups to evaluate their models. All the user needs to do to evaluate their solver is host it on a thin server, which we provide. Then, on the web interface, they need to select their desired task and dataset to test their solver. After their solver finishes processing the dataset, the user can view the results.

Modules

The project contains two main modules.

• The OpenEval core, which contains the main functionalities and the web app.
• A Learner, which acts as a toy system to be evaluated against core.

Quick Guide on Running the Apps

Your will need Java 8 in order to run App. openjdk on Ubuntu seems to have issues.

You will also need sbt

First, run sbt from the parent directory.

• projects will show the names of the existing module names.
  • project core will take you inside the core package.
  • project learner will take you inside the examples package.
• Inside each project you can compile it, or run it.

If you run the core you can browse to localhost:9000. To run on specific port simply add the port number after run. You also should not have to start it multiple times. You can just save code, and refresh the page.

Note: OpenEval server needs to store its backend information in a SQL database. If you want to run it on your machine first create a SQL DB, rename core/conf/application.conf.sample to core/conf/application.conf, and add the DB information (url, username and password).