search

allegroai / clearml

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Apache License 2.0
5.61k stars 651 forks source link

Verify dataset in parallel #1131

Closed charlienewey-odin closed 12 months ago

charlienewey-odin commented 12 months ago

Related Issue \ discussion

Closes #1130

https://github.com/allegroai/clearml/issues/1130

Patch Description

Processes file verification in parallel.

Testing Instructions

Ensure behaviour is same as single-threaded implementation.

Other Information