Dataloader improvements

activeloopai / deeplake

Database for AI. Store Vectors, Images, Texts, Videos, etc. Use with LLMs/LangChain. Store, query, version, & visualize any AI data. Stream data in real-time to PyTorch/TensorFlow. https://activeloop.ai

https://activeloop.ai

Mozilla Public License 2.0

7.88k stars 607 forks source link

Dataloader improvements #2806

Open activesoull opened 3 months ago

activesoull commented 3 months ago

🚀 🚀 Pull Request

Impact

[ ] Bug fix (non-breaking change which fixes expected existing functionality)
[ ] Enhancement/New feature (adds functionality without impacting existing logic)
[ ] Breaking change (fix or feature that would cause existing functionality to change)

Description

Improved dataloader multiprocessing
- reduced IPC overhead
- removed ShuffleBuffer filling when the shuffle is True but the shuffle buffer logic is still there
added nii, nii.gz and dcm support

Things to be aware of

Things to worry about

Additional Context

codecov[bot] commented 3 months ago

Codecov Report

Attention: Patch coverage is `85.93750%` with `9 lines` in your changes are missing coverage. Please review.	Files	Patch %
deeplake/enterprise/dataloader.py	73.68%	5 Missing :warning:
deeplake/integrations/pytorch/common.py	77.77%	2 Missing :warning:
deeplake/core/dataset/deeplake_query_dataset.py	80.00%	1 Missing :warning:
deeplake/core/storage/gcs.py	0.00%	1 Missing :warning:

:loudspeaker: Thoughts on this report? Let us know!

sonarcloud[bot] commented 2 months ago

Quality Gate passed

Issues
3 New issues
0 Accepted issues

Measures
0 Security Hotspots
47.2% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarCloud