This PR adds batching functionality to the evaluator for query calls to the skill manager. Instead of sending a query call for each data point, data points are "batched" into a single request and the skill manager returns the predictions for all data points at once.
Who can review?
@BenediktStuhrmann
@melicoretta
@nclskfm
Test locally
To test this PR locally
change square-core/skill-manager/Dockerfile to:
FROM python:3.10.8-slim-buster as base
ENV PYTHONUNBUFFERED=1
ENV PIP_NO_CACHE_DIR=1
# required to install packages from github
RUN apt-get -y update && apt-get -y install git
RUN pip install --upgrade pip
WORKDIR /app
COPY square-core/skill-manager/requirements.txt requirements.txt
RUN pip install -r requirements.txt
COPY square-skill-api square-skill-api
RUN pip install --editable square-skill-api
COPY square-core/skill-manager skill_manager
COPY square-core/skill-manager/logging.conf logging.conf
FROM base as test
RUN apt update && apt -y install docker.io
COPY requirements.dev.txt requirements.dev.txt
RUN pip install -r requirements.dev.txt
COPY tests tests
COPY pytest.ini pytest.ini
FROM base as build
EXPOSE 8000
CMD ["uvicorn", "skill_manager.main:app", "--host", "0.0.0.0", "--port", "8000", "--log-config", "logging.conf"]
If skills not running locally (inference not possible), test batching by adding a mocked response to the extractive-qa skill in square-core/skills/extractive-qa/skill.py:
For testing batching of skill-manager ("topk" argument needs to be accounted for):
Replace the call to the model or the model_response with a call to the method mocking the response from the model in .
model_response = get_model_response_topk_1_for_evaluator()
Mocked data response is for query with batch size 2, make sure to add only 2 data points to the query, for example by setting dataset = dataset[:2] in square-core/evaluator/evaluator/tasks/predict.py before setting the query_request variable.
Clone the square-skill-api repo to the parent folder of the square-core project. Folder structure should look like this:
What does this PR do?
This PR adds batching functionality to the evaluator for query calls to the skill manager. Instead of sending a query call for each data point, data points are "batched" into a single request and the skill manager returns the predictions for all data points at once.
Who can review?
@BenediktStuhrmann @melicoretta @nclskfm
Test locally
To test this PR locally
change square-core/skill-manager/Dockerfile to:
change square-core/skills/Dockefile to
change urls to call locally running skill-manager and extractive-qa skill.
Add the following docker-compose to the project root folder:
If skills not running locally (inference not possible), test batching by adding a mocked response to the extractive-qa skill in square-core/skills/extractive-qa/skill.py:
Replace the call to the model or the model_response with a call to the method mocking the response from the model in .
model_response = get_model_response_topk_1_for_evaluator()
Mocked data response is for query with batch size 2, make sure to add only 2 data points to the query, for example by setting
dataset = dataset[:2]
in square-core/evaluator/evaluator/tasks/predict.py before setting the query_request variable.Clone the square-skill-api repo to the parent folder of the square-core project. Folder structure should look like this:
change the query attribute of the QueryRequest class in https://github.com/UKP-SQuARE/square-skill-api/blob/master/square_skill_api/models/request.py#L97 to this:
Make sure to import Union and List.
Go to root folder and run the following command from your terminal:
go to the square-core folder and run the following command:
I hope i didnt forget anything :)