search

Lightning-AI / litdata

Streamline data pipelines for AI. Process datasets across 1000s of machines, and optimize data for blazing fast model training.
Apache License 2.0
249 stars 24 forks source link

Add Example to support with MinIO Configuration #170

Closed bhimrazy closed 1 week ago

bhimrazy commented 2 weeks ago

Title: Add Example to support withMinIO Configuration

Description:

The current README is great for setting up with standard AWS s3 configurations. However, including an example for users who try to use litdata with MinIO would be beneficial.

Proposed Addition:

Add a section to the README to demonstrate how to configure litdata to work with MinIO. Below is a proposed example: The content of the configuration file should be as follows:

[default]
aws_access_key_id = ACCESS_KEY_HERE
aws_secret_access_key = SECRET_KEY_HERE
endpoint_url = https://minio.your-domain.com # self deployed minio url
bhimrazy commented 2 weeks ago

Hi @tchaton, Please feel free to assign me this issue. I will be adding a PR for this issue, possibly along with a demonstrable example.