search

hashicorp / nomad

Nomad is an easy-to-use, flexible, and performant workload orchestrator that can deploy a mix of microservice, batch, containerized, and non-containerized applications. Nomad is easy to operate and scale and has native Consul and Vault integrations.
https://www.nomadproject.io/
Other
14.86k stars 1.95k forks source link

Artifacts are not executable #2676

Open sebholstein opened 7 years ago

sebholstein commented 7 years ago

Nomad version

Nomad v0.5.6

Operating system and Environment details

Issue

Downloaded artifacts are not executable in a docker container. It works with raw_fork but not with the Docker driver. I get the following message when using Ubuntu with Vagrant. (I also tested it with Docker for Mac):

f0b494360b302168d8334382dbba185c66b77143729a4c1b6b223d10e44a26cd: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}

Reproduction steps

My go program I'm trying to run as "app":

package main

import "log"

func main() {
  log.Println("DONE! :)")
}

I build this program with the following command:

GOOS=linux GOARCH=amd64 go build -o app app.go

Then I serve the binary via a simple HTTP sever and run the job file (below).

Nomad Server logs (if appropriate)

2017/05/26 19:21:59.951926 [DEBUG] http: Request /v1/evaluation/2cfa116e-4c0b-3b14-6c22-84acf6a5137f/allocations (157.131µs)
    2017/05/26 19:22:00.999816 [DEBUG] driver.docker: docker pull alpine:latest succeeded
    2017/05/26 19:22:01.001897 [DEBUG] driver.docker: image "alpine" (sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118) reference count incremented: 1
    2017/05/26 19:22:01 [DEBUG] plugin: starting plugin: /home/ubuntu/nomad []string{"/home/ubuntu/nomad", "executor", "{\"LogFile\":\"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/executor.out\",\"LogLevel\":\"DEBUG\"}"}
    2017/05/26 19:22:01 [DEBUG] plugin: waiting for RPC address for: /home/ubuntu/nomad
    2017/05/26 19:22:01 [DEBUG] plugin: nomad: 2017/05/26 19:22:01 [DEBUG] plugin: plugin address: unix /tmp/plugin476509569
    2017/05/26 19:22:01.043571 [DEBUG] driver.docker: Setting default logging options to syslog and unix:///tmp/plugin112291564
    2017/05/26 19:22:01.043616 [DEBUG] driver.docker: Using config for logging: {Type:syslog ConfigRaw:[] Config:map[syslog-address:unix:///tmp/plugin112291564]}
    2017/05/26 19:22:01.043622 [DEBUG] driver.docker: using 268435456 bytes memory for app
    2017/05/26 19:22:01.043626 [DEBUG] driver.docker: using 500 cpu shares for app
    2017/05/26 19:22:01.043664 [DEBUG] driver.docker: binding directories []string{"/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/alloc:/alloc", "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local:/local", "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/secrets:/secrets"} for app
    2017/05/26 19:22:01.043676 [DEBUG] driver.docker: networking mode not specified; defaulting to bridge
    2017/05/26 19:22:01.043684 [DEBUG] driver.docker: allocated port 192.168.178.29:40699 -> 40699 (mapped)
    2017/05/26 19:22:01.043690 [DEBUG] driver.docker: exposed port 40699
    2017/05/26 19:22:01.043744 [DEBUG] driver.docker: setting container startup command to: ./local/app
    2017/05/26 19:22:01.043760 [DEBUG] driver.docker: setting container name to: app-1ae5668e-74c0-050d-627d-fd236faeecd1
    2017/05/26 19:22:01.084600 [INFO] driver.docker: created container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112
    2017/05/26 19:22:01.507951 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 1): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:02.939463 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 2): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:04.331313 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 3): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:05.746869 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 4): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:07.171912 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 5): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:08.579853 [DEBUG] driver.docker: failed to start container "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112" (attempt 6): API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:08.579976 [ERR] driver.docker: failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:08 [DEBUG] plugin: /home/ubuntu/nomad: plugin process exited
    2017/05/26 19:22:08.582071 [WARN] client: failed to start task "app" for alloc "1ae5668e-74c0-050d-627d-fd236faeecd1": Failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
    2017/05/26 19:22:08.582646 [INFO] client: Not restarting task: app for alloc: 1ae5668e-74c0-050d-627d-fd236faeecd1
    2017/05/26 19:22:08.582877 [DEBUG] driver.docker: image id "sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118" reference count decremented: 0
    2017/05/26 19:22:08.583423 [INFO] client: marking allocation 1ae5668e-74c0-050d-627d-fd236faeecd1 for GC
    2017/05/26 19:22:08.584152 [INFO] client: marking allocation 1ae5668e-74c0-050d-627d-fd236faeecd1 for GC
    2017/05/26 19:22:08.746745 [DEBUG] client: updated allocations at index 12 (total 1) (pulled 0) (filtered 1)
    2017/05/26 19:22:08.746897 [DEBUG] client: allocs: (added 0) (removed 0) (updated 0) (ignore 1)

Nomad Client logs (if appropriate)

➜  Desktop ./nomad alloc-status --address=http://192.168.178.29:4646 1ae5668e
ID                  = 1ae5668e
Eval ID             = 2cfa116e
Name                = app.app[0]
Node ID             = e6e35f7b
Job ID              = app
Client Status       = failed
Client Description  = <none>
Desired Status      = run
Desired Description = <none>
Created At          = 05/26/17 21:21:58 CEST

Task "app" is "dead"
Task Resources
CPU      Memory   Disk     IOPS  Addresses
500 MHz  256 MiB  300 MiB  0     db: 192.168.178.29:40699

Recent Events:
Time                    Type                   Description
05/26/17 21:22:08 CEST  Not Restarting         Policy allows no restarts
05/26/17 21:22:08 CEST  Driver Failure         failed to start task "app" for alloc "1ae5668e-74c0-050d-627d-fd236faeecd1": Failed to start container f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112: API error (500): {"message":"oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n"}
05/26/17 21:21:58 CEST  Driver                 Downloading image alpine:latest
05/26/17 21:21:58 CEST  Downloading Artifacts  Client is downloading artifacts
05/26/17 21:21:58 CEST  Task Setup             Building Task Directory
05/26/17 21:21:58 CEST  Received               Task received by client

Job file (if appropriate)

job "app" {
  datacenters = ["dc1"]

  type = "batch"

  update {
    stagger = "10s"
    max_parallel = 1
  }

  group "app" {
    count = 1

    restart {
      attempts = 0
      interval = "5m"
      delay = "25s"
      mode = "fail"
    }

    task "app" {
      driver = "docker"

      config {
        image = "alpine"
        command = "./local/app"
      }

      artifact {
        source = "http://localhost:8333/app"
     }

     logs {
        max_files     = 1
        max_file_size = 15
      }

      resources {
        cpu    = 500 # 500 MHz
        memory = 256 # 256MB
        network {
          mbits = 10
          port "db" {}
        }
      }
    }
  }
}
sebholstein commented 7 years ago

When I look at the rights of binary in the local dir/mount, it has no x flag:

ubuntu@ubuntu-xenial:~$ cd /tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local
ubuntu@ubuntu-xenial:/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local$ ls -la
total 1600
drwxrwxrwx 2 ubuntu ubuntu    4096 May 26 19:21 .
drwxrwxrwx 5 ubuntu ubuntu    4096 May 26 19:22 ..
-rw-rw-r-- 1 ubuntu ubuntu 1627996 May 26 19:21 app

Here's the docker inspect output of the container:

ubuntu@ubuntu-xenial:~$ docker inspect f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112
[
    {
        "Id": "f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112",
        "Created": "2017-05-26T19:22:01.046659025Z",
        "Path": "./local/app",
        "Args": [],
        "State": {
            "Status": "created",
            "Running": false,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 0,
            "ExitCode": 126,
            "Error": "oci runtime error: container_linux.go:247: starting container process caused \"exec: \\\"./local/app\\\": permission denied\"\n",
            "StartedAt": "0001-01-01T00:00:00Z",
            "FinishedAt": "0001-01-01T00:00:00Z"
        },
        "Image": "sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118",
        "ResolvConfPath": "/var/lib/docker/containers/f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112/resolv.conf",
        "HostnamePath": "/var/lib/docker/containers/f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112/hostname",
        "HostsPath": "/var/lib/docker/containers/f909e9e0fc21cf502b542d24e1ce3733b97ff4dfcddd2805195f7605679e5112/hosts",
        "LogPath": "",
        "Name": "/app-1ae5668e-74c0-050d-627d-fd236faeecd1",
        "RestartCount": 0,
        "Driver": "aufs",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": [
                "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/alloc:/alloc",
                "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local:/local",
                "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/secrets:/secrets"
            ],
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "syslog",
                "Config": {
                    "syslog-address": "unix:///tmp/plugin112291564"
                }
            },
            "NetworkMode": "bridge",
            "PortBindings": {
                "40699/tcp": [
                    {
                        "HostIp": "192.168.178.29",
                        "HostPort": "40699"
                    }
                ],
                "40699/udp": [
                    {
                        "HostIp": "192.168.178.29",
                        "HostPort": "40699"
                    }
                ]
            },
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "",
            "CpuShares": 500,
            "Memory": 268435456,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DiskQuota": 0,
            "KernelMemory": 0,
            "MemoryReservation": 0,
            "MemorySwap": -1,
            "MemorySwappiness": -1,
            "OomKillDisable": false,
            "PidsLimit": 0,
            "Ulimits": null,
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0
        },
        "GraphDriver": {
            "Data": null,
            "Name": "aufs"
        },
        "Mounts": [
            {
                "Type": "bind",
                "Source": "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/alloc",
                "Destination": "/alloc",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/local",
                "Destination": "/local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            },
            {
                "Type": "bind",
                "Source": "/tmp/NomadClient799156968/1ae5668e-74c0-050d-627d-fd236faeecd1/app/secrets",
                "Destination": "/secrets",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "f909e9e0fc21",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "40699/tcp": {},
                "40699/udp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "NOMAD_TASK_NAME=app",
                "NOMAD_HOST_PORT_db=40699",
                "NOMAD_ALLOC_DIR=/alloc",
                "NOMAD_SECRETS_DIR=/secrets",
                "NOMAD_TASK_DIR=/local",
                "NOMAD_ALLOC_ID=1ae5668e-74c0-050d-627d-fd236faeecd1",
                "NOMAD_JOB_NAME=app",
                "NOMAD_MEMORY_LIMIT=256",
                "NOMAD_ADDR_db=192.168.178.29:40699",
                "NOMAD_ALLOC_NAME=app.app[0]",
                "NOMAD_ALLOC_INDEX=0",
                "NOMAD_PORT_db=40699",
                "NOMAD_CPU_LIMIT=500",
                "NOMAD_IP_db=192.168.178.29",
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
            ],
            "Cmd": [
                "./local/app"
            ],
            "Image": "sha256:a41a7446062d197dd4b21b38122dcc7b2399deb0750c4110925a7dd37c80f118",
            "Volumes": null,
            "WorkingDir": "",
            "Entrypoint": null,
            "OnBuild": null,
            "Labels": {}
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "c552b7ada392cd8ed4676888f4c0705917755accc8e4e53c2d5ebca4d915ebeb",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {},
            "SandboxKey": "/var/run/docker/netns/c552b7ada392",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "bridge": {
                    "IPAMConfig": null,
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "fd8fde1b517481cb2d2aa6531b2d7a1d76af0add15d79f50036d934c37a6592d",
                    "EndpointID": "",
                    "Gateway": "",
                    "IPAddress": "",
                    "IPPrefixLen": 0,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": ""
                }
            }
        }
    }
]
dadgar commented 7 years ago

It is pretty odd to download the binary you are going to be running inside a docker container. You usually bake that into the image.

We set the exec bit when using raw_exec/exec automatically. You are being hit by the lack of ability to set the perms of downloaded artifacts: https://github.com/hashicorp/go-getter/issues/63

sebholstein commented 7 years ago

@dadgar I know it sounds odd :) But I'm trying to write a CI/CD server based on nomad. The user should be able to provide the docker image for the build job and I want to run a binary inside the unknown docker image that handles the build job inside.

Is there any workaround? I tried to run something like this as the command, but that didn't work (due to lstat):

chmod +x ./local/app && ./local/app
dadgar commented 7 years ago

@SebastianM I believe you can work around it by doing:

command = "/bin/bash"
args = ["-c", "chmod +x ./local/app && ./local/app"]

Oh nice! Going to open source the CI/CD?

sebholstein commented 7 years ago

@dadgar thx - works fine!

Oh nice! Going to open source the CI/CD?

Yeah that's the plan. IMO Nomad batch jobs are a perfect fit for CI workloads. I will let you know when it's public 👍

rickardrosen commented 7 years ago

We solved it by mounting a dir from the host containing deployment tools and scripts. The dir keeps containerpilot binary, pre/post scripts and alike. Works pretty well!

lanefu commented 5 years ago

another workaround that's clean is to place the executable in a zip or tar.gz

Storing the binary in the archive will preserve the executable permission. The artifact go-getter will automatically extract.