Open ctgschollar opened 1 month ago
I have tried rebuilding my cluster once in case some of my experimenting to get to this point caused any issues, however on a completely new cluster, with new OS, and new nomad it still fails to start jobs that seem to have been allocated, but won't deploy. Currently sitting at about an hour and a half with all of these jobs in "starting" and allocated, but not doing anything.
Some logs from a client
Jul 22 17:39:56 k8s101 nomad[9815]: 2024-07-22T17:39:56.762Z [INFO] client: node registration complete
Jul 22 17:40:05 k8s101 nomad[9815]: 2024-07-22T17:40:05.541Z [INFO] client: node registration complete
Jul 22 17:47:15 k8s101 nomad[9815]: 2024-07-22T17:47:15.555Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Received msg="Task received by client" failed=false
Jul 22 17:47:15 k8s101 nomad[9815]: 2024-07-22T17:47:15.558Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type="Task Setup" msg="Building Task Directory" failed=false
Jul 22 17:47:15 k8s101 nomad[9815]: 2024-07-22T17:47:15.605Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
Jul 22 17:47:15 k8s101 nomad[9815]: 2024-07-22T17:47:15.605Z [INFO] agent: (runner) creating watcher
Jul 22 17:47:15 k8s101 nomad[9815]: 2024-07-22T17:47:15.605Z [INFO] agent: (runner) starting
Jul 22 17:47:15 k8s101 nomad[9815]: 2024-07-22T17:47:15.625Z [INFO] agent: (runner) rendered "(dynamic)" => "/opt/nomad/data/alloc/34be2794-a5ff-fd61-086b-c2101f432ec2/ceph-node/local/config.json"
Jul 22 17:47:15 k8s101 nomad[9815]: 2024-07-22T17:47:15.637Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Driver msg="Downloading image" failed=false
Jul 22 17:47:49 k8s101 nomad[9815]: 2024-07-22T17:47:49.053Z [INFO] client.driver_mgr.docker: created container: driver=docker container_id=5edb5d60dbe16179aa806aee1cc8e295303b7f791c58c3526ede7046e2605088
Jul 22 17:47:49 k8s101 nomad[9815]: 2024-07-22T17:47:49.265Z [INFO] client.driver_mgr.docker: started container: driver=docker container_id=5edb5d60dbe16179aa806aee1cc8e295303b7f791c58c3526ede7046e2605088
Jul 22 17:47:49 k8s101 nomad[9815]: 2024-07-22T17:47:49.301Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Started msg="Task started by client" failed=false
Jul 22 17:47:49 k8s101 nomad[9815]: 2024-07-22T17:47:49.489Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Terminated msg="Exit Code: 255, Exit Message: \"Docker container exited with non-zero exit code: >
Jul 22 17:47:49 k8s101 nomad[9815]: 2024-07-22T17:47:49.494Z [INFO] client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker plugin=/usr/bin/nomad id=10526
Jul 22 17:47:49 k8s101 nomad[9815]: 2024-07-22T17:47:49.494Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node reason="Restart within policy" delay=17.92143415s
Jul 22 17:47:49 k8s101 nomad[9815]: 2024-07-22T17:47:49.494Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Restarting msg="Task restarting in 17.92143415s" failed=false
Jul 22 17:48:07 k8s101 nomad[9815]: 2024-07-22T17:48:07.447Z [INFO] client.driver_mgr.docker: created container: driver=docker container_id=6d736fa22ef2f091605361e32700d5d2dce99ac9cf31140a982428c975511d73
Jul 22 17:48:07 k8s101 nomad[9815]: 2024-07-22T17:48:07.624Z [INFO] client.driver_mgr.docker: started container: driver=docker container_id=6d736fa22ef2f091605361e32700d5d2dce99ac9cf31140a982428c975511d73
Jul 22 17:48:07 k8s101 nomad[9815]: 2024-07-22T17:48:07.661Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Started msg="Task started by client" failed=false
Jul 22 17:48:07 k8s101 nomad[9815]: 2024-07-22T17:48:07.893Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Terminated msg="Exit Code: 255, Exit Message: \"Docker container exited with non-zero exit code: >
Jul 22 17:48:07 k8s101 nomad[9815]: 2024-07-22T17:48:07.899Z [INFO] client.alloc_runner.task_runner: restarting task: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node reason="Restart within policy" delay=18.638869179s
Jul 22 17:48:07 k8s101 nomad[9815]: 2024-07-22T17:48:07.899Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Restarting msg="Task restarting in 18.638869179s" failed=false
Jul 22 17:48:07 k8s101 nomad[9815]: 2024-07-22T17:48:07.901Z [INFO] client.driver_mgr.docker.docker_logger: plugin process exited: driver=docker plugin=/usr/bin/nomad id=10761
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.302Z [ERROR] client.alloc_runner.task_runner.task_hook: killing task because plugin failed: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node error="CSI plugin failed probe: timeout while connecting to gRPC soc>
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.302Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type="Plugin became unhealthy" msg="Error: CSI plugin failed probe: timeout while connecting to gRPC s>
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.303Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Killing msg="CSI plugin did not become healthy before configured 30s health timeout" failed=true
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.313Z [INFO] client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node plugin=/usr/bin/nomad id=10288
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.314Z [INFO] agent: (runner) stopping
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.314Z [ERROR] client.alloc_runner.task_runner.task_hook: failed to kill task: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node kill_reason="CSI plugin failed probe: timeout while connecting to gRPC socket: fail>
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.314Z [INFO] agent: (runner) received finish
Jul 22 17:48:19 k8s101 nomad[9815]: 2024-07-22T17:48:19.314Z [INFO] client.gc: marking allocation for GC: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.194Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=4bedb61d-5713-05b2-9adc-cb5671d1fe62 task=ceph-node type=Received msg="Task received by client" failed=false
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.196Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=4bedb61d-5713-05b2-9adc-cb5671d1fe62 task=ceph-node type="Task Setup" msg="Building Task Directory" failed=false
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.241Z [INFO] agent: (runner) creating new runner (dry: false, once: false)
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.241Z [INFO] agent: (runner) creating watcher
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.241Z [INFO] agent: (runner) starting
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.263Z [INFO] agent: (runner) rendered "(dynamic)" => "/opt/nomad/data/alloc/4bedb61d-5713-05b2-9adc-cb5671d1fe62/ceph-node/local/config.json"
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.285Z [INFO] client.driver_mgr.docker: created container: driver=docker container_id=41d225f208e95e8b1dc2ba31056d2300ec97dec55dacbade79e05f72f6f99c3f
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.484Z [INFO] client.driver_mgr.docker: started container: driver=docker container_id=41d225f208e95e8b1dc2ba31056d2300ec97dec55dacbade79e05f72f6f99c3f
Jul 22 17:50:05 k8s101 nomad[9815]: 2024-07-22T17:50:05.514Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=4bedb61d-5713-05b2-9adc-cb5671d1fe62 task=ceph-node type=Started msg="Task started by client" failed=false
Jul 22 17:50:10 k8s101 nomad[9815]: 2024-07-22T17:50:10.516Z [INFO] client.alloc_runner.task_runner.task_hook.csi_client: finished client unary call: alloc_id=4bedb61d-5713-05b2-9adc-cb5671d1fe62 plugin.name=ceph-csi plugin.type=node task=ceph-node grpc.code=DeadlineExceeded >
Jul 22 17:50:15 k8s101 nomad[9815]: 2024-07-22T17:50:15.521Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=4bedb61d-5713-05b2-9adc-cb5671d1fe62 task=ceph-node type="Plugin became healthy" msg="plugin: ceph-csi" failed=false
Jul 22 17:50:23 k8s101 nomad[9815]: 2024-07-22T17:50:23.799Z [INFO] client: node registration complete
Jul 22 17:53:52 k8s101 nomad[9815]: 2024-07-22T17:53:52.277Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=ed62f907-a42f-a6cf-2c22-14af2edb8031 task=reserve-volume type=Received msg="Task received by client" failed=false
Jul 22 17:53:52 k8s101 nomad[9815]: 2024-07-22T17:53:52.278Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=ed62f907-a42f-a6cf-2c22-14af2edb8031 task=release-volume type=Received msg="Task received by client" failed=false
Jul 22 17:53:52 k8s101 nomad[9815]: 2024-07-22T17:53:52.279Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=ed62f907-a42f-a6cf-2c22-14af2edb8031 task=transfer type=Received msg="Task received by client" failed=false
Jul 22 17:53:57 k8s101 nomad[9815]: 2024-07-22T17:53:57.594Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7e62615d-0fca-790b-e292-ab21b09fee14 task=reserve-volume type=Received msg="Task received by client" failed=false
Jul 22 17:53:57 k8s101 nomad[9815]: 2024-07-22T17:53:57.595Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7e62615d-0fca-790b-e292-ab21b09fee14 task=release-volume type=Received msg="Task received by client" failed=false
Jul 22 17:53:57 k8s101 nomad[9815]: 2024-07-22T17:53:57.596Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=7e62615d-0fca-790b-e292-ab21b09fee14 task=transfer type=Received msg="Task received by client" failed=false
Jul 22 17:54:04 k8s101 nomad[9815]: 2024-07-22T17:54:04.774Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=0f2059b4-ebb9-5a91-0936-5a71cd467214 task=reserve-volume type=Received msg="Task received by client" failed=false
Jul 22 17:54:04 k8s101 nomad[9815]: 2024-07-22T17:54:04.775Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=0f2059b4-ebb9-5a91-0936-5a71cd467214 task=release-volume type=Received msg="Task received by client" failed=false
Jul 22 17:54:04 k8s101 nomad[9815]: 2024-07-22T17:54:04.775Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=0f2059b4-ebb9-5a91-0936-5a71cd467214 task=transfer type=Received msg="Task received by client" failed=false
Jul 22 17:54:09 k8s101 nomad[9815]: 2024-07-22T17:54:09.661Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=dc9e05b4-dd77-c686-572a-40d76d89adc7 task=reserve-volume type=Received msg="Task received by client" failed=false
Jul 22 17:54:09 k8s101 nomad[9815]: 2024-07-22T17:54:09.662Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=dc9e05b4-dd77-c686-572a-40d76d89adc7 task=release-volume type=Received msg="Task received by client" failed=false
Jul 22 17:54:09 k8s101 nomad[9815]: 2024-07-22T17:54:09.662Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=dc9e05b4-dd77-c686-572a-40d76d89adc7 task=transfer type=Received msg="Task received by client" failed=false
Jul 22 18:54:15 k8s101 nomad[9815]: 2024-07-22T18:54:15.537Z [INFO] client.gc: garbage collecting allocation: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 reason="forced collection"
Jul 22 18:54:15 k8s101 nomad[9815]: 2024-07-22T18:54:15.538Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=34be2794-a5ff-fd61-086b-c2101f432ec2 task=ceph-node type=Killing msg="Sent interrupt. Waiting 5s before force killing" failed=false
and status of one of the allocs mentioned in the log
nomad alloc status ed62f907
ID = ed62f907-a42f-a6cf-2c22-14af2edb8031
Eval ID = 2dd12094
Name = transfer_1708650386_30486.transfer[0]
Node ID = 63eab7b8
Node Name = k8s101
Job ID = transfer_1708650386_30486
Job Version = 0
Client Status = pending
Client Description = <none>
Desired Status = run
Desired Description = <none>
Created = 1h56m ago
Modified = 1h56m ago
root@nomad-svr-3:/etc/nomad.d# nomad alloc status -verbose ed62f907
ID = ed62f907-a42f-a6cf-2c22-14af2edb8031
Eval ID = 2dd12094-d862-c4d6-7dc8-3760d590899b
Name = transfer_1708650386_30486.transfer[0]
Node ID = 63eab7b8-f12d-69d7-23f9-15312b483cf3
Node Name = k8s101
Job ID = transfer_1708650386_30486
Job Version = 0
Client Status = pending
Client Description = <none>
Desired Status = run
Desired Description = <none>
Created = 2024-07-22T17:53:52Z
Modified = 2024-07-22T17:53:52Z
Evaluated Nodes = 2
Filtered Nodes = 0
Exhausted Nodes = 0
Allocation Time = 148.233µs
Failures = 0
Placement Metrics
Node binpack job-anti-affinity node-affinity node-reschedule-penalty final score
63eab7b8-f12d-69d7-23f9-15312b483cf3 0.695 0 0 0 0.695
b59e5084-05a1-6bff-cb6d-5817bda3c0a2 0.688 0 0 0 0.688
A Job definition that works properly
job "test" {
type = "batch"
datacenters = ["chpc_nomad_18"]
group "test" {
volume "test" {
type = "csi"
source = "test"
attachment_mode = "file-system"
access_mode = "single-node-writer"
read_only = false
}
task "whalesay" {
user = "root"
driver = "docker"
config {
auth {
REDACTED
}
network_mode = "host"
image = "harbor.sdp.kat.ac.za/dockerhub_proxy/docker/whalesay"
command = "cowsay"
extra_hosts = ["REDACTED"]
args = ["yolo"]
volumes = ["/home/kat/.ssh:/root/.ssh","/home/kat/.aws/:/root/.aws"]
}
volume_mount {
volume = "test"
destination = "/scratch/kat"
read_only = false
}
resources {
cpu = 100
memory = 100
}
env {
REDACTED
}
}
}
}
All the jobs did eventually run, but it took over 2 hours for them to start.
Hi @ctgschollar! Unfortunately there's not a lot to go on here in your logs. For example, if I look at the most recent log you've posted, I see the Ceph plugin being marked healthy and fingerprinting to the server a few minutes before the allocation ed62f907
is placed:
Jul 22 17:50:15 k8s101 nomad[9815]: 2024-07-22T17:50:15.521Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=4bedb61d-5713-05b2-9adc-cb5671d1fe62 task=ceph-node type="Plugin became healthy" msg="plugin: ceph-csi" failed=false Jul 22 17:50:23 k8s101 nomad[9815]: 2024-07-22T17:50:23.799Z [INFO] client: node registration complete ... Jul 22 17:53:52 k8s101 nomad[9815]: 2024-07-22T17:53:52.277Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=ed62f907-a42f-a6cf-2c22-14af2edb8031 task=reserve-volume type=Received msg="Task received by client" failed=false Jul 22 17:53:52 k8s101 nomad[9815]: 2024-07-22T17:53:52.278Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=ed62f907-a42f-a6cf-2c22-14af2edb8031 task=release-volume type=Received msg="Task received by client" failed=false Jul 22 17:53:52 k8s101 nomad[9815]: 2024-07-22T17:53:52.279Z [INFO] client.alloc_runner.task_runner: Task event: alloc_id=ed62f907-a42f-a6cf-2c22-14af2edb8031 task=transfer type=Received msg="Task received by client" failed=false
But these are all info-level logs and don't go past that initial few minutes. What you'll want to look for are the logs that cover the whole history for this allocation (all the logs relevant to us here should have the alloc ID field), and provide the logs in debug (or even better, trace) level. My guess what's happening here is that the CSI plugin is taking a very long time to provision the volume from Ceph, but without logs that's only a guess. If that is the case, you may also want to get the logs for the Ceph node plugin allocation (via nomad alloc logs
).
Aside:
k8s101
Cheeky! :grin:
Nomad version
Nomad v1.8.0 BuildDate 2024-05-28T17:38:17Z Revision 28b82e4b2259fae5a62e2ed47395334bea5a24c4
Operating system and Environment details
Servers and clients
3 server nodes running as VMs on a proxmox cluster
3 client nodes running on hosts
All running
Distributor ID: Ubuntu Description: Ubuntu 22.04.4 LTS Release: 22.04 Codename: jammy
Volumes
I am running the ceph rbd csi for volumes for all jobs
Driver
Docker version 24.0.6, build ed223bc
Details
I am trying to run some pretty long running batch jobs to do data conversion on large datasets of up to 10TB. The nomad cluster is currently only running these jobs. All the jobs are batch jobs.
I create a new volume for each job, before I run the job using a separate api I created to manage my ceph rbd space.
Immediately after creating the volume I then run the job which seems to load fine. The job then sits in recovering for an hour with no indication of any error, however the jobs doesn't start to run.
Issue
The job is allocated, however the job goes into the "Recovering" status immediately. I can see the jobs are allocated to my clients and have enough resources, however they seem to just wait for some reason.
It then hangs out there for a really long time, from half an hour to over an hour before finally starting the tasks.
Reproduction steps
Any time I run one of my batch jobs
Expected Result
Job should start running when it is allocated. at least within a few mintues
Actual Result
Job stuck in "recovering" for over an hour. With no indication of what the issue is
Job file (if appropriate)
Inspecting jobs
Logs
For an example job transfer_1537557537_30479
Nomad Server logs (if appropriate)
Not sure how to get these
Nomad Client logs (if appropriate)