SHA-based base image, 'nobody' user, and private keys cleanup for Docker image

akirillov commented 6 years ago

What changes were proposed in this pull request?

Resolves DCOS-45167

switch to SHA instead of a tag for the base image
user nobody is added to Spark Docker image
/usr/share/doc is removed fully because it contains private keys which fail twistlock checks
multiple similar RUN steps are combined to minimize the amount of layers in the image

How were these changes tested?

Integration tests from mesosphere/spark-build

Release Notes

switch to SHA instead of a tag for the base Ubuntu 18.04 image
user nobody is added to Spark Docker image
unused/example private keys causing Twistlock checks failures removed
Docker image is optimized to contain fewer layers

elezar commented 6 years ago

I see a lot of messages such as:

+ + mkdir -p /mnt/mesos/sandbox/nginx
mkdir: cannot create directory '/mnt/mesos/sandbox': Permission denied
+ exec
mkdir -p /mnt/mesos/sandbox/spark
mkdir: cannot create directory '/mnt/mesos/sandbox': Permission denied

in the Spark logs. Could it be that we haven't set up the permissions of the sandbox correctly? Is the sandbox now owned by the task user?

akirillov commented 6 years ago

@elezar the reason is in nobody's UID mismatch between a container and the host OS, working on the fix

akirillov commented 6 years ago

@elezar thanks for your comments. This PR is under active development right now and more changes coming. minDcosReleaseVersion will be 1.10 as you advised. Flag name is updated, thanks for pointing out.

mesosphere / spark-build