Fix Failing Builds

[rajadain]

develop is failing with this during provisioning:

TASK [bee-pollinator.beekeepers : Install NPM Dependencies] ********************
Monday 11 February 2019  15:03:54 -0500 (0:00:00.705)       0:01:50.454 ******* 
fatal: [app]: FAILED! => {"changed": false, "msg": "Error starting container 727b7db80470ede2499a69cfb9301252ecded382f07c4d9aa98b6e67a4953c80: 500 Server Error: Internal Server Error (\"OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:301: running exec setns process for init caused \\\"exit status 23\\\"\": unknown\")"}

Formatted error:

Error starting container 727b7db80470ede2499a69cfb9301252ecded382f07c4d9aa98b6e67a4953c80:
    500 Server Error: Internal Server Error (
        "OCI runtime create failed:
            container_linux.go:348:
                starting container process caused 
                    "process_linux.go:301:
                        running exec setns process for init caused 
                            "exit status 23"": unknown"
    )

[hectcastro]

I am worried that this issue is tied to the kernel running on Ubuntu 14.04. You can reproduce the failure within the virtual machine with the snippet below. No need to use Ansible as part of the feedback loop:

vagrant@app:~$ docker run node:8 yarn
docker: Error response from daemon: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:297: copying bootstrap data to pipe caused \"write init-p: broken pipe\"": unknown.

[jeancochrane]

Finally got to the bottom of this! Turns out it is in fact a kernel issue. There was a security patch release of Docker CE (18.6.2) at noon EST yesterday. From the release notes (https://docs.docker.com/engine/release-notes/#18092):

Ubuntu 14.04 customers using a 3.13 kernel will need to upgrade to a supported Ubuntu 4.x kernel

Our boxes are all running 3.13 kernels. Other users have noted that downgrading to Docker CE version 18.06.1~ce~3-0~ubuntu works around the problem. See: https://github.com/docker/for-linux/issues/591

Downgrading seems like the fastest fix, but I can't tell whether the CVE implies that this is insecure for our use-case: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-5736

[hectcastro]

😞

I think downgrading Docker should be OK since we’re really only using it to build a static bundle. The stuff in the CVE would still be possible because we don’t uninstall Docker, but there is no straightforward path for anyone to get to it.

[hectcastro]

It looks like the plan to uninstall Docker is at odds with the AMI build process. The Docker daemon configuration change queues up a service restart for the end of the Ansible run, but by that time the service no longer exists.

See: http://civicci01.internal.azavea.com/view/bees/job/bee-pollinator-app-and-worker/139/

[jeancochrane]

Ugh, I was worried something in the build step would interfere with the Docker deinstall. Are you OK moving forward with leaving Docker installed on the AMI?