Getting started with batch job submissions on U-M HPC Great Lakes

[DaniJonesOcean]

Issue Description: We are moving away from on-demand jobs, which have to be monitored constantly, to batch jobs submitted and managed on a queue.

The Great Lakes HPC system uses the Slurm Workload Manager for job management and scheduling. A solid understanding of Slurm and its batch scripts will help make job management easier (partly by reducing the need to "babysit" the jobs!)

Key Objectives:

• Review the Great Lakes HPC User Guide and familiarize yourself with sections 1.3.1-1.3.3 and 1.3.5 (we can skip 1.3.4 because we already have our data). Pay special attention to the examples in 1.3.5; try one or two of them out.
• Review and bookmark resources such as the Slurm User Guide for Great Lakes (https://arc.umich.edu/greatlakes/slurm-user-guide/).

• Familiarize yourself with the job submission process, using the sbatch command to submit a basic job script. A starter script might look like this:

  #!/bin/bash
  #SBATCH --job-name=deepsensor_gpu_test
  #SBATCH --mail-type=BEGIN,END,FAIL
  #SBATCH --partition=gpu
  #SBATCH --gpus=1
  #SBATCH --mem=4GB
  #SBATCH --time=00:30:00
  #SBATCH --account=[YourAccountName]
  
  module load cuda/11.8.0 cudnn/11.8-v8.7.0
  source ~/deepsensor_env_gpu/bin/activate
  python ~/your_example_code.py

  Modify the script with necessary adjustments for your GPU environment, ensuring the virtual environment setup is incorporated.

Deliverables:

• A brief written summary or checklist that confirms you have reviewed the Great Lakes HPC user guide and other resources. Highlight any questions or clarifications that may arise.
• Test and document the initial login process into Great Lakes via SSH and the successful submission of a basic Slurm job. Let me know how it goes.

Feel free to reach out if you need clarification on any steps!

[DaniJonesOcean]

If needed, run through the environment setup before doing this:

https://github.com/CIGLR-ai-lab/GreatLakes-TempSensors/blob/main/ENVIRONMENT_SETUP.md

[eredding02]

@DaniJonesOcean Prior to this issue I had attended an HPC workshop, where I somewhat understood the basics of using U-M HPC Great Lakes and downloaded resources such as the user guide and "cheat sheet". When using the HPC for this project, I used Jupyter notebooks via on-demand jobs where I was able to understand accessing a virtual environment, requesting memory, using different partitions, and using module commands. Through both of these experiences, along with the sample job script provided, I have been able to successfully submit slurm jobs. Using the nano and sbatch commands, I submitted a script that imported DeepSensor, where I found that the package was not downloaded into the environment I had been using. After appropriately downloading required packages I have started running an expensive script, where I have expectedly run into some memory issues to work on.

[DaniJonesOcean]

@eredding02 Looks good! Nice work. Feel free to close this issue when you're ready.