blabla1337 / skf-labs

Repo for all the OWASP-SKF Docker lab examples
Apache License 2.0
439 stars 201 forks source link

SKF Labs containers needlessly large #116

Closed tsluyter closed 3 years ago

tsluyter commented 4 years ago

You may have noticed pull request 115 -> https://github.com/blabla1337/skf-labs/pull/115

I've been going over a few of your Docker files to make sure that the containers we produce are as small as possible. For students with limited resources (young students, developing countries, little expendable income) storage space can come at a premium. Shrinking our requirements by 60% seems like a very user-friendly thing to do.

With my proposed refactoring, our resulting containers go down from an average 330MB to ~77MB.

The biggest factor in the current ~330MB size per container, lies in the fact that the whole SKF Labs Git repo is pulled into the container and then never removed. By refactoring the Dockerfile to git pull, and then only copying the needed content, it's a huge space-saver already.

Another nice way to save space on our layers is to combine into one RUN the apk add, the pip intall and the apk del to immediately remove unneeded packages.

In the past, some of our students complained that editing the source files with all the ^M (Windows) line endings in place is troublesome. These occur, depending on whom built the container where. To resolve this issue I've started including dos2unix into each Dockerfile, to clean up any scripts and other text-based files. I'm considering adding nano into the containers for those completely unfamiliar with vi.

Please let me know if you'd like me to continue these activities.

tsluyter commented 4 years ago

As per the comments in my pull request about shrinking: I've rethought our Docker strategy.

By defining "skf-labs/alpine37" we can make a common underlayer that applies to almost every container we make. This means that students will only need to "pull" that layer once, in order to build 1-N containers.

This approach means that the final container space grows a little, but in reality the storage space used will be even less than before! The big underlayers will be common/shared by all containers.

tsluyter commented 4 years ago

Here's a comparison between two Docker containers built with the current Dockerfiles, and two with my adjusted Dockerfiles.

Both builds have roughly 100 MB in common: the base OS plus Python3... The big difference lies in the git clone and cleanup afterwards. Lines starting with ** are what the containers have in common.

=== Original config ===

$ docker images | grep owasp-skf-lab
blabla1337/owasp-skf-lab   cors                d38a24f88636        4 months ago        331MB
blabla1337/owasp-skf-lab   xss                 00444b7ad9e2        11 days ago         338MB

$ docker history blabla1337/owasp-skf-lab:cors
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d38a24f88636        4 months ago        CMD ["python3" "./CORS.py"]                     0B                  buildkit.dockerfile.v0
<missing>           4 months ago        RUN /bin/sh -c pip3 install -r requirements.…   8.09MB              buildkit.dockerfile.v0
<missing>           4 months ago        WORKDIR /skf-labs/CORS                          0B                  buildkit.dockerfile.v0
<missing>           4 months ago        RUN /bin/sh -c git clone https://github.com/…   221MB               buildkit.dockerfile.v0
**<missing>           4 months ago        RUN /bin/sh -c apk update --no-cache && apk …   97.9MB              buildkit.dockerfile.v0
**<missing>           4 months ago        MAINTAINER Glenn ten Cate <glenn.ten.cate@ow…   0B                  buildkit.dockerfile.v0
**<missing>           19 months ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
**<missing>           19 months ago       /bin/sh -c #(nop) ADD file:aa17928040e31624c…   4.21MB             

$ docker history blabla1337/owasp-skf-lab:xss
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
00444b7ad9e2        11 days ago         CMD ["python3" "./XSS.py"]                      0B                  buildkit.dockerfile.v0
<missing>           11 days ago         RUN /bin/sh -c chmod +x XSS.py # buildkit       0B                  buildkit.dockerfile.v0
<missing>           11 days ago         RUN /bin/sh -c pip3 install -r requirements.…   8.17MB              buildkit.dockerfile.v0
<missing>           11 days ago         WORKDIR /skf-labs/XSS                           0B                  buildkit.dockerfile.v0
<missing>           11 days ago         RUN /bin/sh -c git clone https://github.com/…   228MB               buildkit.dockerfile.v0
**<missing>           11 days ago         RUN /bin/sh -c apk update --no-cache && apk …   97.9MB              buildkit.dockerfile.v0
**<missing>           11 days ago         MAINTAINER Glenn ten Cate <glenn.ten.cate@ow…   0B                  buildkit.dockerfile.v0
**<missing>           19 months ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
**<missing>           19 months ago       /bin/sh -c #(nop) ADD file:aa17928040e31624c…   4.21MB  

=== My proposed config ===

$ docker images | grep "skf-labs/"
skf-labs/cors              latest              7e0b8fd2c881        23 minutes ago      112MB
skf-labs/xss               latest              d39fd0589ab2        32 minutes ago      112MB

$ docker history skf-labs/cors
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
7e0b8fd2c881        27 minutes ago      CMD ["python3" "./CORS.py"]                     0B                  buildkit.dockerfile.v0
<missing>           27 minutes ago      RUN /bin/sh -c find . -name "*.sh" -o -name …   850kB               buildkit.dockerfile.v0
<missing>           27 minutes ago      WORKDIR /skf-labs/CORS                          0B                  buildkit.dockerfile.v0
<missing>           27 minutes ago      RUN /bin/sh -c cd /tmp && mkdir /skf-labs &&…   1.69MB              buildkit.dockerfile.v0
<missing>           27 minutes ago      RUN /bin/sh -c wget https://raw.githubuserco…   7.1MB               buildkit.dockerfile.v0
**<missing>           40 minutes ago      MAINTAINER Glenn ten Cate <glenn.ten.cate@ow…   0B                  buildkit.dockerfile.v0
**<missing>           40 minutes ago      RUN /bin/sh -c apk add dos2unix --no-cache -…   484kB               buildkit.dockerfile.v0
**<missing>           45 minutes ago      RUN /bin/sh -c apk update --no-cache && apk …   97.9MB              buildkit.dockerfile.v0
**<missing>           45 minutes ago      MAINTAINER Glenn ten Cate <glenn.ten.cate@ow…   0B                  buildkit.dockerfile.v0
**<missing>           19 months ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
**<missing>           19 months ago       /bin/sh -c #(nop) ADD file:aa17928040e31624c…   4.21MB             

$ docker history skf-labs/xss
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
d39fd0589ab2        35 minutes ago      CMD ["python3" "./XSS.py"]                      0B                  buildkit.dockerfile.v0
<missing>           35 minutes ago      USER www                                        0B                  buildkit.dockerfile.v0
<missing>           35 minutes ago      RUN /bin/sh -c find . -name "*.sh" -o -name …   1.26MB              buildkit.dockerfile.v0
<missing>           40 minutes ago      WORKDIR /skf-labs/XSS                           0B                  buildkit.dockerfile.v0
<missing>           40 minutes ago      RUN /bin/sh -c cd /tmp && mkdir /skf-labs &&…   1.62MB              buildkit.dockerfile.v0
<missing>           40 minutes ago      RUN /bin/sh -c wget https://raw.githubuserco…   6.82MB              buildkit.dockerfile.v0
**<missing>           40 minutes ago      MAINTAINER Glenn ten Cate <glenn.ten.cate@ow…   0B                  buildkit.dockerfile.v0
**<missing>           40 minutes ago      RUN /bin/sh -c apk add dos2unix --no-cache -…   484kB               buildkit.dockerfile.v0
**<missing>           45 minutes ago      RUN /bin/sh -c apk update --no-cache && apk …   97.9MB              buildkit.dockerfile.v0
**<missing>           45 minutes ago      MAINTAINER Glenn ten Cate <glenn.ten.cate@ow…   0B                  buildkit.dockerfile.v0
**<missing>           19 months ago       /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B                  
**<missing>           19 months ago       /bin/sh -c #(nop) ADD file:aa17928040e31624c…   4.21MB 
tsluyter commented 3 years ago

You have addressed this issue for most containers through your own branches and methods.