Neural-Dragon-AI / Cynde

A Framework For Intelligence Farming
Apache License 2.0
12 stars 0 forks source link

Start working on distributed framework #12

Open furlat opened 4 months ago

furlat commented 4 months ago

All the functional methods or the at least their higher level wrappers should have a way to be executed in a distributed fashion using either:

  1. https://modal.com/docs/guide/scale
  2. Slurm in preparation of Leonardo
  3. Something simpler to handle multiple gpus in the same lan/vpn

The first focus is being able to execute the stratified cv in parallel on cpu machines, next the embeddings using transformer models.