search

activeloopai / deeplake

Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai
https://activeloop.ai
Mozilla Public License 2.0
8.08k stars 622 forks source link

[FEATURE] Comparaison to LitData #2946

Open tchaton opened 4 weeks ago

tchaton commented 4 weeks ago

Description

Hey there,

I just discovered Deeplake and found it to be an amazing library. I wondered if you would be interested to compare with LitData: https://github.com/Lightning-AI/litdata

Use Cases

Better understanding of market performance.

istranical commented 4 weeks ago

Hi @tchaton you raise a great point! A research group at Yale benchmarked Deep Lake against other dataloaders including FFCV last year, and our implementation has improver further.

We are currently working on DeepLake V4 which includes major changes, and we'll make sure to update benchmarks against other dataloaders, including FFCV.

tchaton commented 4 weeks ago

Thanks @istranical, I copied the wrong link, I meant Litdata: https://github.com/Lightning-AI/litdata