search

Lightning-AI / litdata

Transform datasets at scale. Optimize datasets for fast AI model training.
Apache License 2.0
374 stars 42 forks source link

pass storage options to s5cmd #397

Closed bhimrazy closed 1 month ago

bhimrazy commented 1 month ago
Before submitting - [x] Was this discussed/agreed via a Github issue? (no need for typos and docs improvements) - [x] Did you read the [contributor guideline](https://github.com/Lightning-AI/lit-data/blob/main/.github/CONTRIBUTING.md), Pull Request section? - [x] Did you make sure to update the docs? - [x] Did you write any new necessary tests?

What does this PR do?

Fixes #392.

PR review

Anyone in the community is free to review the PR once the tests have passed. If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

codecov[bot] commented 1 month ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 78%. Comparing base (3f47b5e) to head (4f4a1cc). Report is 1 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #397 +/- ## =================================== Coverage 78% 78% =================================== Files 34 34 Lines 5040 5044 +4 =================================== + Hits 3944 3948 +4 Misses 1096 1096 ```
bhimrazy commented 1 month ago

Can we write a mock test for this?

Should we have a test for it? What do you think!

yeah, it would be great to have one. I will look into it.