Closed pearlmary closed 1 year ago
Hi, Slurm is a cluster environment management tool preinstalled by the clusters. If you don't have a cluster or your cluster is using other management tools, you can also do the sweep using the regular python script.
This line is calling the sbatch xxx.sh
, where sweep_count.sh
has some slurm-specific command such as time allocation, etc. You can remove those and only use the code after this line. Also replace the batch with the regular command to execute bash file.
Thank you so much. Sure, let me check it.
Hi, Slurm is a cluster environment management tool preinstalled by the clusters. If you don't have a cluster or your cluster is using other management tools, you can also do the sweep using the regular python script.
This line is calling the
sbatch xxx.sh
, wheresweep_count.sh
has some slurm-specific command such as time allocation, etc. You can remove those and only use the code after this line. Also replace the batch with the regular command to execute bash file.
Hi Zong, In sweep_batch.py, I just replaced sbatch with bash, and as you mentioned in sweep_count.sh, I commented out the lines before 11. It ran for just two different lr and then it throws errors.
command is bash /workspace/MEDFAIR/sweep/train-sweep/sweep_count.sh --sweep_id eafkn0hh
Traceback (most recent call last):
File "/workspace/fairmed/bin/wandb", line 8, in
error None
resampling
wandb: WARNING Changes to your wandb
environment variables will be ignored because your wandb
session has already started. For more information on how to modify your settings with wandb.init()
arguments, please refer to https://wandb.me/wandb-init.
Problem at: sweep/train-sweep/sweep_batch.py 35
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "sweep/train-sweep/sweep_batch.py", line 35, in
Since I'm new to .sh file handling, can you help me what else has to be done in the .sh files?
I didn't face this error before, but this looks like an error from wandb library instead of the bash. The bash should be right as you can run experiments already. Maybe related to this. Can you try to run wandb in offline mode wandb offline
and see if it works?
Thanks for the reply. I tried the offline option as well, but it gives the same error. It seems that sweeps can't happen with offline mode. Since, you can run it without errors, I think it is the problem with the docker container's port.
Yes, it seems like the issue is with the network/ports rather than the code. Closing for now.
Hi Zong, is there a way to do sweep without using wandb for this current code?
Can you suggest one?
You can write your own script for doing a sweep. E.g., define the Hyperparameter space and loop over it where in each loop you can pass the hyperparameter to call the main.py
.
Hi Zong, I'm using docker and created a virtual environment(installed the prerequisites) to work with medfair (papila dataset). I could not resolve the error for 'sbatch'. Is there any prerequisite we should install for slurm environment? Or it would be great if you can tell me the steps to just run the sweep without using slurm cluster. Just the python way? Any possibilities?