[FEATURE] Merging FeedbackDatasets

[mpjuhasz]

Is your feature request related to a problem? Please describe. I'm working with multiple annotators, and have given them different workspaces. This results in multiple FeedbackDatasets to aggregate. I want to look at the IAA, but the metrics as per the docs only operate on one object.

Describe the solution you'd like A method for merging multiple datasets into one would allow users to use the metrics out of the box in cases like the above.

Describe alternatives you've considered I've worked around it by transforming the dataset into the Huggingface format merging those, extracting the config, and pushing all to the Huggingface Hub. Using FeedbackDataset.from_huggingface() then results in the required single object. This is rather tedious in the long run, as it requires pushing to and pulling from the hub for each aggregation.

Additional context N/A

[burtenshaw]

Hi @mpjuhasz

Thanks for the feature suggestion. This sounds like a cool idea. One question I have is:

• Could I also see merge as add the records of dataset A to dataset B?
• What behavior would you expect if dataset A has different properties to dataset B?

Also, we have a major release of the SDK in beta right now. In this release (2.0), it will be possible to add the records of one dataset to another if their schemas are compatible. For example:

import argilla_sdk as rg

client = rg.Argilla(
    api_url="https://argilla.example.com",
    api_key="my_token",
)

dataset_a = client.datasets("dataset_a") # get the datasets from the argilla server
dataset_b = client.datasets("dataset_b") # get the datasets from the argilla server

dataset_a.records.log(list(dataset_b.records)) # add the records of dataset b to dataset a

We have a blog post on the new release that's coming at the end of the month.

[mpjuhasz]

Hi @burtenshaw,

Thanks for the quick response! My thoughts on the questions:

• absolutely: that's what I've been doing with my HF workaround, merging the records gives me what I need here (analogous to the concatenate_dataset HF function)
• I'd only expect to use this feature for compatible datasets, so it'd make sense to me to throw an error in those cases.

Looking forward to that release 🙌

[burtenshaw]

@mpjuhasz Nice. With that in mind, the 2.0 release should solve your use case. Let me know if you'd like to try out the experimental version in advance.