dterziev
commented
3 years ago After reaching out to the HPC Product Group via the enterprise support channels, we got a private fix for this issue.
Closed dterziev closed 3 years ago
dterziev
commented
3 years ago After reaching out to the HPC Product Group via the enterprise support channels, we got a private fix for this issue.
Problem Description
HPC Pack 2019 (6.0.7214) with Ubuntu 18.04 compute nodes repeatedly kills job tasks, fails the jobs or runs many instances of the same task shortly after compute nodes boot up. Same issue is observed with HPC Pack 2016 Update 2 with Ubuntu compute nodes.
Steps to Reproduce
Install HPC Pack 2019 refresh 6.0.7214 on a Windows 2012 R2 (using SQL Express 2017) and attach one or two Ubuntu 18.04 compute nodes and bring them Online, then shutdown all nodes.
Expected Results
All tasks from all jobs to successfully run once without failing.
Actual Results
Problem 1: The HPC node agent repeatedly starts and kills the tasks shortly after the compute node boots up (see the log below). After a few attempts some of the jobs continue running and complete successfully. Problem 2: Some of the jobs (Job 281 in the log below) fail with "Job failed to start on some nodes or some nodes became unreachable". Problem 3: Multiple instances of the same task are running at the same time on the same node (see job 281). This can lead to corrupting any state that this task may change while it is running. Problem 4: Some of the jobs (Job 280) fail because one of the tasks in the parametric sweep is reported to be running in HPC Job Manager long after it has completed. After the maximum run time for the task has elapsed, HPC cancels the task and that fails the job.
All these issues seem to happen shortly after a compute node boots up and are observed only with Linux compute nodes. It seems that the common theme is that in the first few minutes after the agent starts up, the node may become unreachable which forces the headnode to kill all tasks from the jobs currently running on that node.
Additional Logs
Job 281 failed with "Job failed to start on some nodes or some nodes became unreachable" eventhough some of its tasks have completed successfully on the node IP-172-19-3-105. Here is the same log filtered for Job 281. What is unusual here, is that task 9731 was killed quite a few times but retry Task 9731.5 and Task 9731.8 have succeeded. The log also suggests that those two were running in parallel at the same time which for tasks that depend on modifying some state will lead to corruption of that state.
Job 280, parametric sweep 1.1 - 1.85, task 1.1 executed successfully and according to the log, the headnode was notified that it ran successfuly, and yet HPC reported it as running for 2 hours before cancelling the task. The compute node logs shows that the task was started, then killed, then started again, however the requeue count is incorrect which leads to incorrect state on the headnode:
Requeue #1 isn't what the headnode state is, it is expecting requeue count #2 to complete or fail and therefore ignores the message and leaves the task state as running. The compute node will never receive another message for this task from the headnode.