Closed pavlis closed 1 year ago
Yeah, I can see it will be difficult to run on different hpc systems because the job script is highly dependent on how the hpc system is configured. It is difficult to make a universal script.
With that said, I could answer your questions 2 and 3 here, and we definitely should improve the documentation accordingly.
The word "task" in the context of Slurm really means the MPI task. Therefore, this is almost completely irrelevant to MsPASS as it is not an MPI program. However, on some HPC systems, Slurm is configured to allocate resources based on the number of tasks requested in the sbatch command, which is the case for Big Red 3 (I am not sure about Big Red 200 but would guess so). This Slurm configuration is called node sharing, and if you don't request for all 24 cores (by setting the task number to 24), Slurm will run your job on a node sharing the cores left with other jobs. Since we do want the full node for Dask/Spark to run in parallel, you probably want to set --ntask_per_node
to 24 here. Note that this option is meaningless to TACC's systems because there is no node-sharing there.
The SSH tunnel is used to access the port opened by Jupyter on the compute node from the login node. This is because user accesses the Jupyter from an external IP address to the HPC cluster. Usually, the firewall policy is setup to block such external access for security reasons. That's why it is always a good idea to use the SSH tunnel from the login node as a hop to access the Jupyter Notebook on the compute node.
Very helpful.
First on the "task" issue. That is clearly something important we need to add to our documentation. What you say there is a huge inference from any documentation I've gone through. How slurm treats dask and spark is not at all clear. Good to preserve this.
On the ssh tunnel point, given what you said I think IU’s “RED” that uses Thinlinc to run a window manager on the head note makes the ssh tunnel unnecessary. I used that functionality already to get the single mode version running. I am pretty sure the script will still need to echo something to output to tell the hostname of the node running jupyter. The confusion I had about ssh running we need to make sure we clarify. Specifically that the tunnel is needed only for remote connections to the container running the jupyter notebook. A corollary, I think, is if you are running a notebook in batch mode the tunnel is not needed either. Is that true?
When I get this working on the IU system I'll update this wrt the need for the tunnel in that context. I think, however, that the IU RED system is kind of novel and not something one is likely to find on other HPC clusters.
A corollary, I think, is if you are running a notebook in batch mode the tunnel is not needed either. Is that true?
Yes!
I think this issue is solved by pull request 385 which should be merged in the next week. I'm closing this issue.
See previous comment
In the process of trying to adapt the tacc scripts to the Indiana research cluster I ran across a few obvious deficiencies in our documentation.
--ntask_per_node
. The system I'm trying to configure has dual 12-core cpus (24 total). What I have no idea is what a "task" as defined by slurm relates to dask/spark threads allowed per worker node? I would guess that bcause we use singularity there is only one real "task" per node as far as slurm is concerned but that is a pure guess. I think it would be helpful to put an answer to this question for the record here on github.So, this issue needs a box or two of explanation. The followup is all in the documentation and probably improved comments in the example scripts.