search

eyadgaran / SimpleML

Machine learning that just works, for effortless production applications
BSD 3-Clause "New" or "Revised" License
17 stars 5 forks source link

Support for Job Management (Trainer Module) #48

Open eyadgaran opened 3 years ago

eyadgaran commented 3 years ago

Currently the management focus is on persistables and artifacts (ie once the persistables have been initialized).

Scope: Extend db tracking to include Jobs (jobs are the configs used to initialize persistables) Standardize a Job configuration standard (traitlets, yaml) Write a parser and tracking objects for jobs and configs

eyadgaran commented 2 years ago

Trainer module Training is composed of a few interchangeable components.

  1. configuration - can be yaml or json for simple references or traitlets for more granular control
  2. Job definition. All configuration sources compile to a job definition via parsers. Doesn’t matter which is used
  3. Executors. Compiled jobs are executed via the backend or framework. Adapters confer compatibility cross frameworks (eg airflow operator or Kubeflow spec)