refactoring of data access code

IBM / data-prep-kit

Open source project for data preparation of LLM application builders

https://ibm.github.io/data-prep-kit/

Apache License 2.0

300 stars 132 forks source link

Closed blublinsky closed 2 months ago

blublinsky commented 2 months ago

consolidated s3 and local code fixed small issues in checkpointing