Refactor Dockerfile and docker build context generation

[KSAlpha]

Problem

Current implementation renders Dockerfile for build instruction and copies whole repository to temporary build context directory, which is considered inefficient in following conditions:

• Repository contains multiple problems
  • Soma will also copy other problems and make the build context bigger (slows down build speed).
  • Could be resolved by copying problem directory only.
• Repository contains non-problem files like .git
  • Build context becomes bigger (slows down build speed)
  • Could be handled by using .dockerignore.
• Problem contains a number of file entries
  • Build steps becomes bigger (creates a lot of intermediate images). Docker officially recommends to reduce build steps.

There are several 'easy' workarounds to resolve these issues, but I would like to suggest another design to generate Docker image build context.

New Design

Switching rendering subject

Current implementations of Soma renders all requirements into Dockerfile. However, we can render a shell script instead of adding several RUN instructions in Dockerfile. Following instructions can be substituted into one shell script:

• chmod
• chown

Fully utilizing temporary directory

Soma is implemented to copy the whole repository (or problem) directory into temporary directory. However, we can construct a directory structure of the image beforehand. By constructing a image root filesystem in the temporary folder, we can reduce (file )adding steps into single instruction. Also, non-related files such as .git will be excluded naturally during the process.

For example, simple-bof describes two files (flag, build/simple-bof) to be included in /home/simple-bof (i.e. default target path).

Then, the temporary folder looks like the following:

/tmp/____
├ .soma
│ ├ start.sh
│ └ configure_permissions.sh
├ build
│ └ home
│   └ simple-bof
│     ├ flag
│     └ simple-bof
└ Dockerfile

Expected Changes

• Reduced build step
• Reduced build context size

Current implementation

Dockerfile:

FROM {{ manifest.binary.os }}

RUN apt-get -qq update && apt-get -yqq upgrade && apt-get install -yqq socat

LABEL maintainer="soma_{{ username }}"
LABEL soma.username="{{ username }}"
LABEL soma.version="{{ version }}"
LABEL soma.repository="{{ repository_name }}"

{{ #with manifest }}
ENV PROB "{{ name }}"
RUN useradd -m $PROB

{{ #each executable }}
COPY "{{ path }}" "{{ target_path }}"
RUN chmod +x "{{ target_path }}"
{{ /each }}

{{ #each readonly }}
COPY "{{ path }}" "{{ target_path }}"
RUN chmod -w "{{ target_path }}"
{{ /each }}

RUN chown -R root:$PROB /home/$PROB

COPY ./start.sh /start.sh
RUN chmod 555 /start.sh

USER $PROB
WORKDIR /home/$PROB
CMD ["/start.sh"]

# TODO: Container internal port settings may be implemented afterwards
EXPOSE 1337
{{ /with }}

New Design

Dockerfile:

FROM {{ manifest.binary.os }}

LABEL maintainer="soma_{{ username }}"
LABEL soma.username="{{ username }}"
LABEL soma.version="{{ version }}"
LABEL soma.repository="{{ repository_name }}"

RUN apt-get -qq update && apt-get -yqq upgrade && apt-get install -yqq socat

ENV PROB "{{ manifest.name }}"
RUN useradd -m $PROB

ADD build/ /
ADD .soma/ /.soma/

RUN /.soma/configure_permissions.sh && rm /.soma/configure_permissions.sh

USER $PROB
WORKDIR /home/$PROB
CMD ["/.soma/start.sh"]

# TODO: Container internal port settings may be implemented afterwards
EXPOSE 1337

configure_permissions.sh

#!/bin/bash
# Exit script on error
set -e

{{ #with manifest }}
{{ #each executable }}
chmod -R +x "{{ target_path }}"
{{ /each }}

{{ #each readonly }}
chmod -R -w "{{ target_path }}"
{{ /each }}

chown -R root:$PROB /home/$PROB
chmod 555 /.soma/start.sh
{{ /with }}

[Qwaz]

The overall direction seems good. I think we should delete (or at least prevent access to) configure_permissions file because that is at a fixed location and may provide hints for a problem.

[KSAlpha]

I forgot to tell Docker to remove that script! Edited Dockerfile section :)

[KSAlpha]

I figured out that image build API includes label setting parameters, which is considered to be a good alternative to set images label as it can reduce template rendering complexity.

[Qwaz]

How build directory is handled in multi-config scenario? Maybe we should use the name of the configuration instead of static build directory.

[KSAlpha]

I think instantiating temporary folder per image (or sub configuration) can solve the problem. For now, I can't think of any drawback from this.

By the way, we should also consider multi-configuration scenario with same kind of subconfigurations (e.g. binary + binary, web + db + db). I will file an issue later related to this one.

[Qwaz]

Another question: if the value of public attribute is different among configurations, what should we do?

1. Reject
2. All
3. Any
4. Undefined Behavior

[KSAlpha]

For now, "any with exceptional rejection" seems nice to me. As public attribute has a default value of "false", setting public on any "true" value looks natural. However, if Soma detects explicit settings of both (i.e. explicit "true" and "false") we should reject it as it is apparently a bad configuration.

Or, enforcing all file entries' public attribute to be "true" or "false" seems to be a nice choice too.