[feature]: docker container engine

[northdpole]

Issue for the docker container engine being developed as part of the summer code sprint (proposal and timeline : https://github.com/a0xnirudh/hackademic/wiki/Summer-Code-Sprint-Proposal ) Ideas for future improvements (after the code sprint):

• Investigate the use docker compose for container management
• Allow challenge-contributors to submit complete docker images instead of code+puppet files
• Write instructions on how to easily dockerize apps (we could point them to http://jasonwilder.com/blog/2014/10/13/a-simple-way-to-dockerize-applications/)
• Show the logs of each container using javascript on the frontend OR integrate a full docker ui for the admin.
• Docker build from github (?) -- challenge repository in our github repo
• Add docker build instructions in the challenge package description (useful for various things such as creating read-only images)
• Investigate the use of Ansible for the project http://docs.ansible.com/ansible/docker_module.html

[pchaigno]

Docker build from github (?) -- challenge repository in our github repo

What's your idea on this? (I'm working on a Dockerfile to have all of Hackademic Challenges deployable in Docker quickly)

Investigate the use of Ansible for the project http://docs.ansible.com/ansible/docker_module.html

I have some experience with Ansible. What are you thinking of doing?

[northdpole]

What's your idea on this? (I'm working on a Dockerfile to have all of Hackademic Challenges deployable in Docker quickly)

I was thinking of creating a challenge repo and give the option to download and install challenges from the repo straight from the ui. (Similar to wordpress theme and plugin search functionality)

What do you mean by having all challenges deployable in docker quickly?

I have some experience with Ansible. What are you thinking of doing?

I like the monitor, maintaining container count etc functionality (the examples in the page provided above seem interesting). Maybe instead of using puppet for installing and managing the platform we could use ansible and also use an ansible python library to manage the deployed container images. But that's based on 10' research.

[a0xnirudh]

Allow challenge-contributors to submit complete docker images instead of code+puppet files

Submit complete Docker images ? Why ? Basically all they have to do is to put the challenge (as of now, lets assume that challenges are only in PHP) in a particular folder and the install script will take care of the same. Why do we need challenge Authors to submit complete docker images ?

Show the logs of each container using javascript on the frontend OR integrate a full docker ui for the admin.

This would be a really nice idea if we get someone good in designing. Hopefully we will get some for OWCS.

I was thinking of creating a challenge repo and give the option to download and install challenges from the repo straight from the ui. (Similar to wordpress theme and plugin search functionality)

Yes that would be a great idea. Before doing that can we convert Hackademic into a github organization rather than a user ? Then we can handle multiple repos, one for challenges exclusively so that things will be easier for challenge developers. Also as an organization (like zap and OWTF), things will be well organized and more professional.

Docker build from github (?) -- challenge repository in our github repo

I wonder if this is possible in github ? Can travis cl or something similar can handle this ?

What's your idea on this? (I'm working on a Dockerfile to have all of Hackademic Challenges deployable in Docker quickly)

@pchaigno why do we want to do that ? We are trying to dockerize all the common challenges of Hackademic ? That would be a hectic task + if thats the case, hackademic could take a lot of resources in the host machine (from what I understood that is).

[northdpole]

Submit complete Docker images ? Why ? Basically all they have to do is to put the challenge (as of now, lets assume that challenges are only in PHP) in a particular folder and the install script will take care of the same. Why do we need challenge Authors to submit complete docker images ?

Because the challenges aren't only simple php scripts, one may want to host other services or a complete system or a bunch of somehow connected containers. It would be cool if they could do that.

[a0xnirudh]

Because the challenges aren't only simple php scripts, one may want to host other services or a complete system or a bunch of somehow connected containers. It would be cool if they could do that.

Yes I understand, for that we can slowly provide built in support for platforms like Nodejs, and python files (I think this is easy). If we want to get another docker image, it is not easy as each of them will be another 300 MB in size + users has to download too much. I think we can handle any type of web challenges without much problem with a standard image. We can even try to automate the same image to work for even buffer overflow challenges. Unless its absolutely necessary, creating different docker images for different challenges is a bad idea IMHO.

[northdpole]

Yes I understand, for that we can slowly provide built in support for platforms like Nodejs, and python files (I think this is easy). If we want to get another docker image, it is not easy as each of them will be another 300 MB in size + users has to download too much. I think we can handle any type of web challenges without much problem with a standard image. We can even try to automate the same image to work for even buffer overflow challenges. Unless its absolutely necessary, creating different docker images for different challenges is a bad idea IMHO.

When an admin installs the challenge we could go through the process of downloading and installing whatever is needed so we have it cached locally for future use, right? Then when a user launches the challenge all wee do is copy the image to a randomized folder and launch from there. (possible?)

[a0xnirudh]

When an admin installs the challenge we could go through the process of downloading and installing whatever is needed so we have it cached locally for future use, right?

Yes once downloaded, I don;t think users have to download it again provided he won't manually delete the image.

Then when a user launches the challenge all wee do is copy the image to a randomized folder and launch from there. (possible?)

Copy the image to a randomized folder? Docker images are the operating systems used to run dockers. So what do you mean by copy the image to a randomized folder and launch from there?

[northdpole]

Copy the image to a randomized folder? Docker images are the operating systems used to run dockers. So what do you mean by copy the image to a randomized folder and launch from there?

nvm it was coffee-deprived nonsense :P

[pchaigno]

What do you mean by having all challenges deployable in docker quickly?

I meant the website in a Docker. So that Hackademic could be deployed (as it is now) with just docker run ;)

Docker build from github (?) -- challenge repository in our github repo

@northdpole Do you mean pushing an image to Docker Hub or building a the set of Docker images (with the new challenges)?

[northdpole]

@pchaigno

I meant the website in a Docker. So that Hackademic could be deployed (as it is now) with just docker run ;) having the cms part as a separate container would be interesting, but integrating the container engine in it wouldn't it introduce delay? I mean having a docker engine which fires up containers run inside docker? (Also, a dependency for a docker image which in turn has docker )

@northdpole Do you mean pushing an image to Docker Hub or building a the set of Docker images (with the new challenges)?

Last time I checked docker supports pulling images and building from github, so it would be interesting to see if it's worth having a github repo where all the challenges are and allow an admin to select the ones they want and have the platform download and build them in the background

[pchaigno]

Last time I checked docker supports pulling images and building from github

Are you referring to Dockerfiles, or is it something else?

[northdpole]

Are you referring to Dockerfiles, or is it something else?

you can issue a docker build and it will download and build an image

[AmaanC]

I'd just like to raise a few questions (might just be me being confused. If that is so, I'm sorry!)

The questions:

• Do we plan on having a separate Docker for each challenge? If yes, can we discuss that?
• What about adding challenges dynamically? When a new challenge is added, we'd need some kind of REST API to dynamically build the newly added challenge. @a0xnirudh mentioned this on IRC, but I think it's best to have things be explicit here, in the central place. (Drawback of this method: every docker image would take up a lot of space)

[northdpole]

1)Yes that's kind of the plan, challenge writers can either use one of our own or choose to submit their own Dokerfiles. What's your idea?

2) There's a repo on my profile called ctfFramework which is the start of this, and also we have a summer project for a Sandbox API. Do you have an alternative solution for the freedom to create anything vs size debate?

[AmaanC]

I thought there'd be a central Docker image, initially, but I see that that would be ridiculous now. Every challenge would need to make sure it doesn't collide with any of the others, and they'd be restricted in what they could do if we took that path. Letting each challenge have its own container makes the most sense, I think.

[arpitsingla96]

There are two main factors that need to be considered here:

1. The size of the challenges: Docker can layer images on top of each other and that enables sharing of space. It is therefore logical that we have a base image which has important stuff installed (like for challenges using php, we can install common packages like php5-cli, php5-curl, php5-dev and other important general packages like build-essential that will be common in many challenges). We can then ask the challenge creators to inherit from this base image and build their challenge on it, thus saving important server disk space for us.
2. Isolation of challenges: In case a challenge is compromised, it should not jeopardize all other challenges. Hence the single image pattern is a bad approach. Each challenge should have its own image (inheriting from base image). With different challenges running in different containers it will be easier to monitor the challenges' resource usage using something like Google CAdvisor. Proper resource constraints [as mentioned in https://docs.docker.com/engine/reference/run/#runtime-constraints-on-resources] should also be put on each container.