kedro-org / kedro-plugins

First-party plugins maintained by the Kedro team.
Apache License 2.0
94 stars 90 forks source link

feat(datasets): Added the Experimental SafetensorsDataset #898

Open MinuraPunchihewa opened 1 month ago

MinuraPunchihewa commented 1 month ago

Description

This PR adds the SafetensorsDataset to support interactions with tensors stored in files in the Safetensors format.

Fixes https://github.com/kedro-org/kedro-plugins/issues/221

Development notes

I have used the PickleDataset as a base for the implementation of this dataset (as it goes about the same manner to access files).

These changes have been tested,

  1. Manually, by running the code locally to load and save tensors from and to Safetensors files.
  2. Via the existing and newly added unit tests.

Checklist

MinuraPunchihewa commented 1 month ago

Hey @astrojuanlu, This is my implementation of the SafetensorsDataset. I would appreciate a review.

MinuraPunchihewa commented 1 week ago

Thank you for the PR, @MinuraPunchihewa! It looks great to me. I tested it manually, and everything works perfectly!

Thank you, @DimedS. I appreciate it. I just updated the release notes as well.