Closed jfcalvo closed 4 months ago
To accommodate the proposed feature of adding a score
attribute for suggestions associated with a multi_label_selection
question in the Argilla repository, modifications would be necessary in both the SDK and the UI. Here's a breakdown of the changes required:
MultiLabelQuestion
Schema:
MultiLabelQuestion
class to hold the score for each label. This could be a dictionary mapping labels to their scores.class MultiLabelQuestion(_LabelQuestion):
# Existing attributes...
scores: Optional[Dict[str, float]] = None
@property
def server_settings(self) -> Dict[str, Any]:
settings = super().server_settings
if self.scores:
settings["scores"] = self.scores
return settings
server_settings
Property:
server_settings
property of the MultiLabelQuestion
class to include the scores in the payload if they are provided.Display Scores in the UI:
multi_label_selection
questions needs to be updated to display the score next to each label if scores are provided. This might involve adjusting the layout to accommodate the display of scores in a user-friendly manner.Update Data Handling:
multi_label_selection
questions can correctly handle and submit the scores along with the selected labels if required.Validation and Error Handling:
scores
attribute does not break existing functionality for multi_label_selection
questions that do not use this new feature.These modifications will enable the Argilla SDK and UI to support the new feature of adding scores to suggestions for multi_label_selection
questions, enhancing the flexibility and utility of the feedback collection process.
For reference, the current structure of the MultiLabelQuestion
class in the Argilla SDK is as follows:
class MultiLabelQuestion(_LabelQuestion):
"""Schema for the `FeedbackDataset` label questions, which are the ones that will
require a label response from the user. This class should be used when the user can
select multiple labels.
Args:
type: The type of the question. Defaults to 'multi_label_selection' and
cannot/shouldn't be modified.
labels: The list of labels of the label question. The labels must be unique, and
the list must contain at least two unique labels. Additionally, `labels` can
also be a dictionary of labels, where the keys are the labels, and the values
are the labels that will be shown in the UI.
visible_labels: The number of visible labels in the UI. Defaults to 20, and must
be 3 or greater.
Examples:
>>> from argilla.client.feedback.schemas.questions import MultiLabelQuestion
>>> MultiLabelQuestion(name="multi_label_question", title="Multi Label Question", labels=["label_1", "label_2"])
"""
type: Literal[QuestionTypes.multi_label_selection] = Field(
QuestionTypes.multi_label_selection.value, allow_mutation=False
)
You can find the source code for this class in the Argilla repository.
For the new feature adding support for span questions we will give support to specify a
score
for suggestions associated to a span question.For that we are following the next format (PUT to upsert a suggestion):
For this issue we should add
score
attribute for suggestions associated to amulti_label_selection
question too. With this change users can specify ascore
value for every label inside the suggestion.Additional points to research/discuss: