Refactor run.py and included hyperparameter search
Added configs for task specific hyperparameter search
Updated transformers, and ray version used on aws. Known issues: I'm still having occasional issues with ray configs, where the link between /mnt/efs/results and /home/ec2-user/nta/results is not set up properly. My current workaround is to set up the link manually after spinning up a cluster: ln -s ${EFS_USER_DIR} /home/ec2-user/nta/results. Other remaining issue is that wandb is logging hyperparameter search runs, but not showing charts.
Custom tools for analyzing hyperparameter search runs
Additional checks for sparsity and callbacks
Added hpchase to experiments, which contains configs that automatically load best hyperparameters
Added hp_finetuning subdirectory to experiments where each subdirectory contains best hyperparameters for each of several milestone models. This is how hpchase gets the best parameters.
Added scripts I've been using outside of git for analyzing finetuning / hyperparameter tuning runs in batch. Analyze_all_hp_search for example writes the best hyperparameters for a series of models to hp_finetuning.
This PR involves several changes:
ln -s ${EFS_USER_DIR} /home/ec2-user/nta/results. Other remaining issue is that wandb is logging hyperparameter search runs, but not showing charts.