openbmc / openbmc-build-scripts

Apache License 2.0
21 stars 52 forks source link

Intermittent failures when getting docker ip address in run-qemu-robot-test.sh #16

Closed geissonator closed 6 years ago

geissonator commented 6 years ago

I've tried a couple fixes for this but not having much luck. Intermittently, our command to get the IP address of the docker container we launch QEMU in with this command:

   DOCKER_QEMU_IP_ADDR="$(docker inspect $obmc_qemu_docker |  \
                       grep -m 1 "IPAddress\":" | cut -d '"' -f 4)"

Gets this error:

13:50:55 ++ docker run --detach --user root --env HOME=/tmp/openbmc/build --env QEMU_RUN_TIMER=600 --env QEMU_ARCH=ppc64le-linux --env QEMU_BIN=./qemu-system-arm --env MACHINE=romulus --workdir /tmp/openbmc/build --volume /home/fspcibld/master-merge-ubuntu-romulus/openbmc/build:/tmp/openbmc/build --tty openbmc/ubuntu-robot-qemu /tmp/openbmc/build/boot-qemu-test.exp
13:50:56 + obmc_qemu_docker=f6111b5364d51f09f6c149688f385c1398bc9b1e96b328767f8f0738a5d9bd07
13:50:56 + DOCKER_SSH_PORT=22
13:50:56 + DOCKER_HTTPS_PORT=443
13:50:56 + trap '' PIPE
13:50:56 ++ docker inspect f6111b5364d51f09f6c149688f385c1398bc9b1e96b328767f8f0738a5d9bd07
13:50:56 ++ grep -m 1 'IPAddress":'
13:50:56 ++ cut -d '"' -f 4
13:50:56 + DOCKER_QEMU_IP_ADDR=172.17.0.3
13:50:56 + echo 141
13:50:56 141

I'm thinking we just ensure the script handles errors properly and remove the "-e" from the top of run-qemu-robot-test.sh.

geissonator commented 6 years ago

@amboar mentioned maybe just a "|| true" to the line instead of removing -e is a better option.

mdmillerii commented 6 years ago

The exit code

141 = 128 + 13 = 0x80 | SIGPIPE

so this error occurs because the

trap '' SIGPIPE

is not blocking SIGPIPE in this pipeline. The shell probably sets it to a handler which the results in default vs ignored signal behavior being inherited

This seems to match the bash documentation (under paragraph When a simple command ... is to be executed,)

       ·      traps caught by the shell are reset to the values inherited from
              the shell's parent, and traps ignored by the shell are ignored

Other options might be to use awk to consume all the output instead of the pipeline of grep and cut, or to turn off pipefail for this pipeline.

mdmillerii commented 6 years ago

Another option would be to take the last ip instead of the first so that all the output would be consumed ... remove the -m1 from grep and add tail -n1 to the pipeline.

geissonator commented 6 years ago

so like this @mdmillerii ?

docker inspect $obmc_qemu_docker | grep "IPAddress\":" | tail -n1 | cut -d '"' -f 4

seems to work as expected

$ docker inspect 58f8e191310a | grep "IPAddress\":" | cut -d '"' -f 4
172.17.0.6
172.17.0.6

$ docker inspect 58f8e191310a | grep "IPAddress\":" | tail -n1 | cut -d '"' -f 4
172.17.0.6
mdmillerii commented 6 years ago

From discussion the address appears under multiple elements but the last seems to work as well as the first, and has the advantage of consuming the entire output.