Nomad logs format json duplicated with syslog format logs

Hello guys,

I just want to report a new issue seen by one of my collegue and I, about having nomad logs in json format. Nomad version: 1.5.3 Consul versio: 1.15.2 OS: Debian 11

With a basic config, log output is fine:

{"@level":"info","@message":"blocked evals status modified","@module":"nomad","@timestamp":"2023-04-19T09:45:34.430667Z","paused":false}
{"@level":"info","@message":"initialized keyring","@module":"nomad.keyring","@timestamp":"2023-04-19T09:45:34.436925Z","id":"90209498-e373-124a-6f11-96803efb5202"}
{"@level":"info","@message":"node registration complete","@module":"client","@timestamp":"2023-04-19T09:45:34.605201Z"}
{"@level":"info","@message":"node registration complete","@module":"client","@timestamp":"2023-04-19T09:45:35.609540Z"}

Here is the basic config:

log_file   = "/tmp/nomad.json"
log_rotate_max_files       =  5
log_rotate_bytes           =  10000000
log_json = true
log_level = "INFO"

With an advanced config, log output is duplicated in syslog format AND json format:

{"@level":"info","@message":"(runner) starting","@module":"agent","@timestamp":"2023-04-19T09:49:32.571972Z"}
Apr 19 09:49:32 REDACTED nomad[160135]: {"@level":"info","@message":"(runner) starting","@module":"agent","@timestamp":"2023-04-19T09:49:32.571972Z"}

Here is the advanced config:

# Full configuration options can be found at https://www.nomadproject.io/docs/configuration

data_dir = "REDACTED"
plugin_dir = "REDACTED"
bind_addr = "0.0.0.0"
region     = "REDACTED"
datacenter = "REDACTED"
log_level = "INFO"
log_json = true
log_file = "/var/log/nomad/server.log"
log_rotate_max_files = 5

client {
  enabled = true
  server_join {
    retry_join = ["provider=gce project_name=REDACTED tag_value=nomad-server zone_pattern=REDACTED-.*"]
  }
  node_class = "none"
}

tls {
  http = true
  rpc  = true

  ca_file   = "REDACTED"
  cert_file = "REDACTED"
  key_file  = "REDACTED"

  verify_server_hostname = true
  verify_https_client    = true
}

vault {
  enabled          = true
  ca_path          = "REDACTED"
  address          = "REDACTED"
}

plugin "docker" {
  config {
    allow_privileged = true
    auth {
      config = "/etc/docker/config.json"
    }
  }
}

acl {
  enabled = true
}

consul {
  address      = "127.0.0.1:8501"
  grpc_address = "127.0.0.1:8503"
  ssl          = true
  grpc_ca_file = "REDACTED"
  ca_file      = "REDACTED"
  cert_file    = "REDACTED"
  key_file     = "REDACTED"
  token        = "REDACTED"
}

telemetry {
  publish_allocation_metrics = true
  publish_node_metrics       = true
  prometheus_metrics         = true
}

We are supposed to have json logs and instead we have a mixed of syslog logs and json logs. Waiting for your answer. I will in parallel initiate a ticket with Hashicorp Support Team.

hashicorp / nomad

Nomad logs format json duplicated with syslog format logs #16933