Proposal for data_path and full training management

UETAILab / uetai

Custom ML tracking experiment and debugging tools.

https://uetailab.github.io/uetai/

MIT License

15 stars 1 forks source link

Proposal for data_path and full training management #20

Open gungui98 opened 2 years ago

gungui98 commented 2 years ago

I want to develop a solution to manage the dataset and also the development environment detection.

One solution I can think of is setup an environment like what wandb has done with WANDB_MODE.

class SummaryWriter(LightningLoggerBase):

def is_full_training(self):
     return os.get("ENVIRONMENT") == "PRODUCTION"

The self-hosted runner also should be mount to a shared NFS, eg. DATA_PATH on our lab!

gungui98 commented 2 years ago

@manhdung20112000 check this out

nmd2k commented 2 years ago

Yes, but can you describe more about these features? I'm thinking that if the development environment was detected then the full dataset will be mounted or downloaded and the full monitor/logging method will be activated. Otherwise, a sample of dataset and debugging monitor method are activated