hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.94k stars 1.96k forks source link

UNIX socket path '/var/lib/nomad/alloc/9e1d5f9a-70f3-1354-077c-6947e262d845/$jobname/qemu-monitor.sock' Path must be less than 108 bytes #13943

Closed iovcho closed 9 months ago

iovcho commented 2 years ago

Nomad version

Output from nomad version Nomad v1.3.2 (bf602974112964e9691729f3f0716ff2bcdb3b44)

Operating system and Environment details

Debian 10.12 Linux 4.19.0-18-amd64 #1 SMP Debian 4.19.208-1 (2021-09-29) x86_64 GNU/Linux

Issue

job "job-test" { datacenters = ["DC"] type = "service" group "job-test" { count = 1 task "job-test" { driver = "qemu" config { image_path = "local/latest.qcow2" args = ["-drive", "file=local/cloud-init.img"] accelerator = "kvm" graceful_shutdown = "true" guest_agent = "true" port_map { ssh = 22 http = 80 https = 443 } } artifact { source = "http://192.168.0.20:8580/cloud-init.img" } artifact { source = "http://192.168.0.20:8580/latest.qcow2" } resources { cores = 4 memory = 4096 disk = 20000 network { port "ssh" {} port "http" {} port "https" {} } } } } }

nomad alloc status $ALLOCID

ID = ddf54ff5-dffb-6524-c216-0141d3c03089 Eval ID = 334d31e2 Name = job-test[0] Node ID = 09811883 Node Name = host5 Job ID = job-test Job Version = 0 Client Status = failed Client Description = Failed tasks Desired Status = run Desired Description = Created = 6m58s ago Modified = 5m30s ago Deployment ID = 903c7f9f Deployment Health = unhealthy

Task "job-test" is "dead" Task Resources CPU Memory Disk Addresses 0/9200 MHz 0 B/4.0 GiB 300 MiB ssh: 172.16.0.25:25007 http: 172.16.0.25:30685 https: 172.16.0.25:28481

Task Events: Started At = 2022-08-01T14:11:55Z Finished At = 2022-08-01T14:11:56Z Total Restarts = 2 Last Restart = 2022-08-01T17:11:40+03:00

Recent Events: Time Type Description 2022-08-01T17:11:56+03:00 Alloc Unhealthy Unhealthy because of failed task 2022-08-01T17:11:56+03:00 Not Restarting Exceeded allowed attempts 2 in interval 30m0s and mode is "fail" 2022-08-01T17:11:56+03:00 Terminated Exit Code: 1 2022-08-01T17:11:55+03:00 Started Task started by client 2022-08-01T17:11:40+03:00 Restarting Task restarting in 15.235353881s 2022-08-01T17:11:40+03:00 Terminated Exit Code: 1 2022-08-01T17:11:40+03:00 Started Task started by client 2022-08-01T17:11:40+03:00 Restarting Task restarting in 16.969039224s 2022-08-01T17:11:40+03:00 Terminated Exit Code: 1 2022-08-01T17:11:40+03:00 Started Task started by client

nomad alloc logs -stderr ddf54ff5

qemu-system-x86_64: -monitor unix:/var/lib/nomad/alloc/ddf54ff5-dffb-6524-c216-0141d3c03089/xxx/qemu-monitor.sock,server,nowait: UNIX socket path '/var/lib/nomad/alloc/ddf54ff5-dffb-6524-c216-0141d3cg Path must be less than 108 bytes qemu-system-x86_64: -monitor unix:/var/lib/nomad/alloc/ddf54ff5-dffb-6524-c216-0141d3c03089/xxx/qemu-monitor.sock,server,nowait: UNIX socket path '/var/lib/nomad/alloc/ddf54ff5-dffb-6524-c216-0141d3cg Path must be less than 108 bytes qemu-system-x86_64: -monitor unix:/var/lib/nomad/alloc/ddf54ff5-dffb-6524-c216-0141d3c03089/xxx/qemu-monitor.sock,server,nowait: UNIX socket path '/var/lib/nomad/alloc/ddf54ff5-dffb-6524-c216-0141d3cg Path must be less than 108 bytes

lgfa29 commented 2 years ago

Hi @iovcho 👋

The 108 bytes limit is set by Linux, so unfortunately there is not much we can do about it. The socket is placed inside the task directory, so maybe you can try a shorter task name?

Another thing we noticed is that the task name in your sample job file is job-test, which should leave plenty of room for the full socket path. Was this the task name you were using when you receive this error message?

To help a bit with problems like this I reduced the socket file name in In #13971 to give some characters back to other components of the path.

lgfa29 commented 9 months ago

Hello from the future 👋

I cleaning up some stale issue and we haven't had any updates on this one for a while, so I'm going to close it for now. But let me know if this is still a problem and we can reopen it.

Thank you!