ζ₯ζ¬θͺη π
OptHub Runner
Opthub Runner is a Python package that provides the following two features:
- Evaluator: Feature to evaluate user-submitted solutions on Opthub using Docker Image.
- Scorer: Feature to calculate scores from a series of evaluations using Docker Image.
This repository describes how to install and run Opthub Runner.
Getting Started
Prerequisites
- Install Python 3.10 or newer.
- Set up pip for package management.
- Install and start Docker.*
*For Mac users, you can install and start Docker Desktop.
1. Installation
Install opthub-runner-admin from PyPI.
pip install opthub-runner-admin2. Configuring YAML File
Specify the options for starting the Evaluator and Scorer in a YAML file. For detailed information on the options set in the file, please refer to YAML File Options.
3. Execution
Running Evaluator
First, start the Evaluator using the following command.
Set the path to the YAML file configured with the options in --config.
opthub-runner evaluator --config (yaml file path)You will be prompted to enter a username and password. Please enter the administrator username and password for the competition.
Username: (your username)
Password: (your password)Running Scorer
First, start the Scorer using the following command.
Set the path to the YAML file configured with the options in --config.
opthub-runner scorer --config (yaml file path)You will be prompted to enter a username and password. Please enter the administrator username and password for the competition.
Username: (your username)
Password: (your password)YAML File Options
The following table describes the options to be specified in the YAML file, including their types, default values, and descriptions.
| Option | Type | Default Value | Description |
|---|---|---|---|
| interval | int | 2 | Interval to fetch messages from Amazon SQS. |
| timeout | int | 43200 | Timeout for evaluation and score calculation using Docker Image. |
| rm | bool | True | Whether to remove the Docker Container after evaluation and score calculation. |
| log_level | [DEBUG, INFO, WARNING, ERROR, CRITICAL] | INFO | Log level to output. |
| evaluator_queue_url | path | - | Amazon SQS queue URL used by the Evaluator. |
| scorer_queue_url | path | - | Amazon SQS queue URL used by the Scorer. |
| access_key_id | str | - | AWS Access Key ID. |
| secret_access_key | str | - | AWS Secret Access Key. |
| region_name | str | - | AWS default Region Name. |
| table_name | str | - | DynamoDB table name to store solutions, evaluations, and scores. |
Please check the values for the following options with the OptHub representative. Contact information is here.
- evaluator_queue_url
- scorer_queue_url
- access_key_id
- secret_access_key
- region_name
- table_name
For Contributors
Follow these steps to set up the environment:
- Clone this repository.
- Set up Poetry.
- Run
poetry install. - Download the recommended VSCode Extensions.
- Disable the following VS Code Extensions for this workspace to avoid conflicts with other packages:
- ms-python.pylint
- ms-python.black-formatter
- ms-python.flake8
- ms-python.isort
- Create
config.ymlwith options based onconfig.default.yml.- Place
config.ymldirectly under opthub-runner-admin. - For the values of the options commented out in
config.default.yml, please contact the Opthub representative. Contact information is here.
- Place
Once you have completed the above setup, you can use the opthub-runner command in the project's root directory.
Contact
If you have any questions or concerns, please feel free to contact us (Email: dev@opthub.ai).
