pkazmierczak
commented
1 month ago Hi @josh-m-sharpe, thanks for reporting the issue. Sadly, I can't reproduce.
Here's what I did:
- installed nomad from official package
- ran nomad server with minimal config
-
created nomad client config:
client { enabled = true server_join { retry_join = ["provider=aws tag_key=Nomad_role tag_value=nomad-workstation_server"] retry_max = 5 retry_interval = "15s" } "node_class" = "es" host_volume "docker-sock-ro" { path = "/var/run/docker.sock" read_only = true } }
plugin "raw_exec" { config { enabled = true } }
data_dir = "/home/ubuntu/nomad_tmp" datacenter = "dc1"
log_level = "TRACE"
plugin "docker" { config { auth { helper = "ecr-login" } extra_labels = ["job_name", "task_group_name", "task_name", "namespace", "node_name"] allow_privileged = true } }
telemetry { collection_interval = "1s" disable_hostname = true prometheus_metrics = true publish_allocation_metrics = true publish_node_metrics = true }
- ran the client with `sudo nomad agent -config=nomad.hcl`
$ nomad version
Nomad v1.8.3
BuildDate 2024-08-13T07:37:30Z
Revision 63b636e5cbaca312cf6ea63e040f445f05f00478
$ nomad node status -self -verbose
ID = c42cb411-5dfa-1608-fc63-a513b8c3e893
Name = ip-10-0-1-160
Node Pool = default
Class = es
DC = dc1
Drain = false
Eligibility = eligible
Status = ready
CSI Controllers =
Host Volumes Name ReadOnly Source docker-sock-ro true /var/run/docker.sock
Drivers
Driver Detected Healthy Message Time
docker true true Healthy 2024-08-15T10:13:29+02:00
exec true true Healthy 2024-08-15T10:13:29+02:00
java false false
Node Events
Time Subsystem Message Details
2024-08-15T10:13:29+02:00 Cluster Node registered
Allocated Resources CPU Memory Disk 0/4600 MHz 0 B/7.8 GiB 0 B/90 GiB
Allocation Resource Utilization CPU Memory 0/4600 MHz 0 B/7.8 GiB
Host Resource Utilization CPU Memory Disk 22/4600 MHz 395 MiB/7.8 GiB (/dev/root)
Allocations No allocations placed
Attributes cpu.arch = amd64 cpu.frequency = 2300 cpu.modelname = Intel(R) Xeon(R) CPU E5-2686 v4 @ 2.30GHz cpu.numcores = 2 cpu.reservablecores = 2 cpu.totalcompute = 4600 cpu.usablecompute = 4600 driver.docker = 1 driver.docker.bridge_ip = 172.17.0.1 driver.docker.os_type = linux driver.docker.privileged.enabled = true driver.docker.runtimes = io.containerd.runc.v2,runc driver.docker.version = 24.0.7 driver.exec = 1 driver.raw_exec = 1 kernel.arch = x86_64 kernel.landlock = v4 kernel.name = linux kernel.version = 6.8.0-1008-aws memory.totalbytes = 8323854336 nomad.advertise.address = 10.0.1.160:4646 nomad.bridge.hairpin_mode = false nomad.revision = 63b636e5cbaca312cf6ea63e040f445f05f00478 nomad.service_discovery = true nomad.version = 1.8.3 numa.node.count = 1 numa.node0.cores = 0-1 os.cgroups.version = 2 os.name = ubuntu os.signals = SIGWINCH,SIGXCPU,SIGFPE,SIGILL,SIGIO,SIGPROF,SIGTSTP,SIGCONT,SIGHUP,SIGTTOU,SIGINT,SIGIOT,SIGSTOP,SIGQUIT,SIGSEGV,SIGBUS,SIGALRM,SIGPIPE,SIGSYS,SIGUSR2,SIGXFSZ,SIGNULL,SIGKILL,SIGTERM,SIGTRAP,SIGTTIN,SIGABRT,SIGUSR1 os.version = 24.04 platform.aws.ami-id = ami-01e444924a2233b07 platform.aws.instance-life-cycle = on-demand platform.aws.instance-type = m4.large platform.aws.placement.availability-zone = eu-central-1b unique.hostname = ip-10-0-1-160 unique.network.ip-address = 10.0.1.160 unique.platform.aws.hostname = ip-10-0-1-160.eu-central-1.compute.internal unique.platform.aws.instance-id = i-05a5c3592ad22b67b unique.platform.aws.local-hostname = ip-10-0-1-160.eu-central-1.compute.internal unique.platform.aws.local-ipv4 = 10.0.1.160 unique.platform.aws.mac = 06:2a:0d:82:87:b5 unique.platform.aws.public-ipv4 = 52.28.111.27 unique.storage.bytesfree = 97064935424 unique.storage.bytestotal = 102888095744 unique.storage.volume = /dev/root
Meta connect.gateway_image = docker.io/envoyproxy/envoy:v${NOMAD_envoy_version} connect.log_level = info connect.proxy_concurrency = 1 connect.sidecar_image = docker.io/envoyproxy/envoy:v${NOMAD_envoy_version} connect.transparent_proxy.default_outbound_port = 15001 connect.transparent_proxy.default_uid = 101
Can you provide more details about how you're running Nomad?
josh-m-sharpe
Nomad version
Operating system and Environment details
amazonlinux 2023
Issue
client detects 0 cpu. therefore can't alloc jobs.
Reproduction steps
run this agent as root:
Expected Result
jobs would deploy, client would report that it has cpu
Actual Result
visit client UI: /ui/clients/7ecfa3d3-91ff-71a7-b7a1-8a51c87a495d See:
0 MHz / 0 MHz TotalJob file (if appropriate)
not really important other than specifying some cpu resource requirement, e.g.:
cpu = 100Notes
It would appear that uncommenting
cpu_total_compute = 10000in that client config above "fixes" this. This is not a real fix. The agent reports that it has 10000 cpu:4% 388 MHz / 10,000 MHz Total(it doesn't really) but this at allows me to allocate jobs to these clients.Other context
I upgraded from v1.5.8 to v1.8.3. I didn't test any versions in between.