search

allegroai / clearml-server

ClearML - Auto-Magical CI/CD to streamline your AI workload. Experiment Management, Data Management, Pipeline, Orchestration, Scheduling & Serving in one MLOps/LLMOps solution
https://clear.ml/docs
Other
381 stars 131 forks source link

ClearML dataset versioning/management performance #155

Closed senysenyseny16 closed 2 years ago

senysenyseny16 commented 2 years ago

hi!

At this moment in our company we use DVC for dataset versioning/management and we want to integrate in our workflow ClearML, so there are two options:

  1. Leave DVC for dataset versioning and use ClearML for tracking
  2. Set up ClearML for both

The second options looks better, but we couldn't find any information about ClearML dataset versioning performance in comparison to DVC (for example caching speed, support for large datasets, is it based on git?).

Could you clarify this?

Thanks in advance.

bmartinn commented 2 years ago

(Duplicate of clearml#750) Dateset versioning is implemented & fully supported, following the discussion in the original thread