ambiguity in documentation around NOMAD_TASK_DIR vs "task directory"

[tgross]

The task directories documentation contains the following text:

Nomad makes the following directories available to tasks:

• alloc/: This directory is shared across all tasks in a task group and can be used to store data that needs to be used by multiple tasks, such as a log shipper.
• local/: This directory is private to each task. It can be used to store arbitrary data that should not be shared by tasks in the task group. ...

Depending on the driver and operating system being targeted, the directories are made available in various ways. For example, on docker the directories are bound to the container, while on exec on Linux the directories are mounted into the chroot.

Meanwhile, the template documentation includes this bit:

destination (string: <required>) - Specifies the location where the resulting template should be rendered, relative to the task directory.

The NOMAD_TASK_DIR is the local/ directory provided to each task, whereas the "task directory" is the directory that Nomad uses to set up the task, and may not be the actual filesystem of the task, depending on the driver. This turns out to be quite different between raw_exec/exec and docker tasks.

There's nothing we're doing wrong here per se, but the behavior could be better described, particularly around the use of templates where it's possible to write a template that can't be seen from inside a Docker container. This issue is to anchor a discussion about how we want to document this better.

---

The following jobspec illustrates some of the interesting behavioral differences:

job "dirs" {
  datacenters = ["dc1"]

  group "group" {

    task "exectask" {
      driver = "exec"
      config {
        command = "/bin/sh"
        args    = ["./script.sh"]
      }
      resources {
        cpu    = 256
        memory = 128
      }

      template {

        data = <<EOT
#!/bin/sh
touch ${NOMAD_TASK_DIR}/exectask-was-here

# make sure all the tasks have started and touched their taskdir
sleep 30

echo "--- environment ----------------------------"
echo NOMAD_ALLOC_DIR: ${NOMAD_ALLOC_DIR}
echo NOMAD_TASK_DIR:  ${NOMAD_TASK_DIR}
echo "--- root dir ----------------------------"
ls /
echo "--- alloc dir ----------------------------"
find ${NOMAD_ALLOC_DIR}
echo "--- task dir ----------------------------"
find ${NOMAD_TASK_DIR}
sleep 600

EOT

        # this destination is relative to the *root*
        destination = "script.sh"
      }

    }

    task "rawexectask" {
      driver = "exec"
      config {
        command = "/bin/sh"
        args    = ["./script.sh"]
      }
      resources {
        cpu    = 256
        memory = 128
      }

      template {

        data = <<EOT
#!/bin/sh
touch ${NOMAD_TASK_DIR}/rawexectask-was-here

# make sure all the tasks have started and touched their taskdir
sleep 30

echo "--- environment ----------------------------"
echo NOMAD_ALLOC_DIR: ${NOMAD_ALLOC_DIR}
echo NOMAD_TASK_DIR:  ${NOMAD_TASK_DIR}
echo "--- root dir ----------------------------"
ls /
echo "--- alloc dir ----------------------------"
find ${NOMAD_ALLOC_DIR}
echo "--- task dir ----------------------------"
find ${NOMAD_TASK_DIR}
sleep 600

EOT

        destination = "script.sh"
      }

    }

    task "dockertask" {
      driver = "docker"
      config {
        image   = "busybox:1"
        command = "/bin/sh"
        args    = ["/local/script.sh"]
      }
      resources {
        cpu    = 256
        memory = 128
      }

      template {

        data = <<EOT
#!/bin/sh
touch ${NOMAD_TASK_DIR}/dockertask-was-here

# make sure all the tasks have started and touched their taskdir
sleep 30

echo "--- environment ----------------------------"
echo NOMAD_ALLOC_DIR: ${NOMAD_ALLOC_DIR}
echo NOMAD_TASK_DIR:  ${NOMAD_TASK_DIR}
echo "--- root dir ----------------------------"
ls /
echo "--- alloc dir ----------------------------"
find ${NOMAD_ALLOC_DIR}
echo "--- task dir ----------------------------"
find ${NOMAD_TASK_DIR}
sleep 600

EOT

        # note: if we use "script.sh" here without "local/" it's not visible
        # to the container at all! it's found where the task can't see it at:
        # /var/nomad/alloc/:id/dockertask/script.sh
        # see weirddockertask, below.
        destination = "local/script.sh"
      }

    }

    task "weirddockertask" {
      driver = "docker"
      config {
        image   = "busybox:1"
        command = "/bin/sh"
        args    = ["-c", "echo '-- root ---; ls /; echo '-- local ---'; ls /local; sleep 600"]
      }
      resources {
        cpu    = 256
        memory = 128
      }

      template {

        data = <<EOT
weird-dockertask-was-here

EOT

        # this is relative to the root of the container, at:
        # /var/nomad/alloc/:id/dockertask/weird-dockertask-was-here
        # but the docker task driver unpacks an overlay filesystem
        # elsewhere and we can't see anything that isn't
        # explicitly bound into it (the /alloc, /local, and /secrets dirs)
        # so this file is invisible to us.
        destination = "weird-dockertask-was-here"
      }

    }

  }
}

If we look at the logs for the three tasks, we see that the raw_exec and exec jobs have their script file at /script.sh, whereas the Docker task has its at /local/script.sh. We had to do this because the "relative" path of ./script.sh is not available inside the Docker container. First, the logs:

nomad alloc logs dockertask

``` $ nomad alloc logs a2c dockertask --- environment ---------------------------- NOMAD_ALLOC_DIR: /alloc NOMAD_TASK_DIR: /local --- root dir ---------------------------- alloc bin dev etc home local proc root secrets sys tmp usr var --- alloc dir ---------------------------- /alloc /alloc/container /alloc/container/871cc932_8058_22a4_f90c_a238fec755bc /alloc/container/871cc932_8058_22a4_f90c_a238fec755bc/state.json /alloc/container/fd83a19b_7dfb_f90e_361c_1fa06e7b7c8a /alloc/container/fd83a19b_7dfb_f90e_361c_1fa06e7b7c8a/state.json /alloc/data /alloc/logs /alloc/logs/.rawexectask.stdout.fifo /alloc/logs/exectask.stdout.0 /alloc/logs/exectask.stderr.0 /alloc/logs/dockertask.stdout.0 /alloc/logs/rawexectask.stdout.0 /alloc/logs/.exectask.stdout.fifo /alloc/logs/rawexectask.stderr.0 /alloc/logs/.dockertask.stderr.fifo /alloc/logs/.rawexectask.stderr.fifo /alloc/logs/.exectask.stderr.fifo /alloc/logs/.dockertask.stdout.fifo /alloc/logs/dockertask.stderr.0 /alloc/tmp --- task dir ---------------------------- /local /local/script.sh /local/dockertask-was-here ```

nomad alloc logs exectask

``` $ nomad alloc logs a2c exectask --- environment ---------------------------- NOMAD_ALLOC_DIR: /alloc NOMAD_TASK_DIR: /local --- root dir ---------------------------- alloc bin dev etc executor.out lib lib32 lib64 local proc run sbin script.sh secrets sys tmp usr --- alloc dir ---------------------------- /alloc /alloc/container /alloc/data /alloc/logs /alloc/logs/.rawexectask.stdout.fifo /alloc/logs/exectask.stdout.0 /alloc/logs/exectask.stderr.0 /alloc/logs/dockertask.stdout.0 /alloc/logs/rawexectask.stdout.0 /alloc/logs/.exectask.stdout.fifo /alloc/logs/rawexectask.stderr.0 /alloc/logs/.dockertask.stderr.fifo /alloc/logs/.rawexectask.stderr.fifo /alloc/logs/.exectask.stderr.fifo /alloc/logs/.dockertask.stdout.fifo /alloc/logs/dockertask.stderr.0 /alloc/tmp --- task dir ---------------------------- /local /local/exectask-was-here ```

nomad alloc logs exectask

``` $ nomad alloc logs a2c rawexectask --- environment ---------------------------- NOMAD_ALLOC_DIR: /alloc NOMAD_TASK_DIR: /local --- root dir ---------------------------- alloc bin dev etc executor.out lib lib32 lib64 local proc run sbin script.sh secrets sys tmp usr --- alloc dir ---------------------------- /alloc /alloc/container /alloc/data /alloc/logs /alloc/logs/.rawexectask.stdout.fifo /alloc/logs/exectask.stdout.0 /alloc/logs/exectask.stderr.0 /alloc/logs/dockertask.stdout.0 /alloc/logs/rawexectask.stdout.0 /alloc/logs/.exectask.stdout.fifo /alloc/logs/rawexectask.stderr.0 /alloc/logs/.dockertask.stderr.fifo /alloc/logs/.rawexectask.stderr.fifo /alloc/logs/.exectask.stderr.fifo /alloc/logs/.dockertask.stdout.fifo /alloc/logs/dockertask.stderr.0 /alloc/tmp --- task dir ---------------------------- /local /local/rawexectask-was-here ```

If we then go into the Nomad datadir for the allocation, we see the following:

root@linux:/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332# ls
alloc  dockertask  exectask  rawexectask

The rawexectask directory is the working directory of our raw_exec task, full of hardlinks back to the host:

root@linux:/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332# find rawexectask/ -maxdepth 1
rawexectask/
rawexectask/lib
rawexectask/lib64
rawexectask/etc
rawexectask/executor.out
rawexectask/sys
rawexectask/dev
rawexectask/sbin
rawexectask/local
rawexectask/proc
rawexectask/lib32
rawexectask/bin
rawexectask/usr
rawexectask/script.sh
rawexectask/run
rawexectask/alloc
rawexectask/tmp
rawexectask/secrets

The exectask directory is the working directory of our exec task, full of bind-mounts from the host:

root@linux:/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332# find exectask/ -maxdepth 1
exectask/
exectask/lib
exectask/lib64
exectask/etc
exectask/executor.out
exectask/sys
exectask/dev
exectask/sbin
exectask/local
exectask/proc
exectask/lib32
exectask/bin
exectask/usr
exectask/script.sh
exectask/run
exectask/alloc
exectask/tmp
exectask/secrets

But the dockertask directory is the working directory of the task, but includes only the bind-mounts that we add for /alloc, /local, and /secrets. The rest of the overlay filesystem unpacked by the Docker driver is elsewhere. So we can see the weird-dockertask-was-here in the allocation's "task directory" on the host, but not inside the running container.

root@linux:/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332# find dockertask/ -maxdepth 1
dockertask/
dockertask/weird-dockertask-was-here
dockertask/local
dockertask/tmp
dockertask/secrets

The Nomad-defined mounts for the container:

root@linux:/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332# docker inspect 1da | jq '.[0].Mounts'
[
  {
    "Type": "bind",
    "Source": "/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332/dockertask/secrets",
    "Destination": "/secrets",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332/alloc",
    "Destination": "/alloc",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  },
  {
    "Type": "bind",
    "Source": "/var/nomad/alloc/a2c02305-f397-e72a-55c1-79728d7ad332/dockertask/local",
    "Destination": "/local",
    "Mode": "",
    "RW": true,
    "Propagation": "rprivate"
  }
]

cc @angrycub @notnoop

[tgross]

Oh, and for completeness, nomad alloc fs sees the allocation's directory on the host (which isn't the same as NOMAD_ALLOC_DIR), and can see our "hidden" file as a result, but the task can't do anything with it!

$ nomad alloc fs a2c
Mode        Size     Modified Time         Name
drwxrwxrwx  4.0 KiB  2020-09-17T19:12:32Z  alloc/
drwxrwxrwx  4.0 KiB  2020-09-17T19:12:29Z  dockertask/
drwxrwxrwx  4.0 KiB  2020-09-17T19:12:32Z  exectask/
drwxrwxrwx  4.0 KiB  2020-09-17T19:12:32Z  rawexectask/

$ nomad alloc fs a2c dockertask
Mode         Size     Modified Time         Name
drwxrwxrwx   4.0 KiB  2020-09-17T19:12:31Z  local/
drwxrwxrwx   60 B     2020-09-17T19:12:29Z  secrets/
dtrwxrwxrwx  4.0 KiB  2020-09-17T19:12:29Z  tmp/
-rw-r--r--   27 B     2020-09-17T19:12:29Z  weird-dockertask-was-here

[tgross]

This also has some interesting impact with templates and absolute directories, such as https://github.com/hashicorp/nomad/issues/8962

[tgross]

Cross-referencing https://github.com/hashicorp/nomad/issues/2948

[tgross]

Working up a PR for this.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.