Clone this repository and cd.
git clone git@github.com:langtech-bsc/argilla-eval.git
Cd into the repository and download the docker-compose.yaml
wget -O docker-compose.yaml https://raw.githubusercontent.com/argilla-io/argilla/main/examples/deployments/docker/docker-compose.yaml
Deploy the Argilla server locally.
docker compose up -d
Create a venv and install requirements
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
Create an .env file with the following variables.
Argilla server requires:
API_URL=http://localhost:6900
API_KEY=argilla.apikey
Depending on the ptoject change them:
RG_WORKSPACE=eval_workspace
NUMBER_USERS=3
DATASET_NAME=prompts-eval-dataset
DATASET_PATH=dataset.json
Once Argilla server is deployed we need to interact with it via sdk.
Run workspace.py
to create a workspace on the argilla server.
python workspace.py
Run users.py
to handle users on workspaces.
python users.py
Generated dynamically from NUMBER_USERS
.
Users and passwords are sotred at root level asusers.csv
Run datasets.py
to create and manipulate datasets.
python datasets.py
Datasets are loaded from /datasets/your_dataset.json
Run records
to query and extract records.
python records.py
Rercods are stored at root level as records.json