Add a Dockerfile

[davidrpugh]

Need to create a Dockerfile that includes the following:

1. OpenJDK8
2. Scala 2.12
3. SBT 0.13
4. Miniconda
5. Python 3
6. All dependencies necessary to run the jupyter notebook
7. matplotlib, numPy, pandas, plotly, cufflinks

I can't think of anything else at the moment. Basically we want to create a Dockerfile that will allow a user to start of a container whose entry point is a Jupyter notebook.

[ghost]

This should be pretty straight forward, I'm happy to take it on (famous last words).

[davidrpugh]

@ibillett Start by forking this repo and figuring out how to add appropriate versions of SBT and Scala and miniconda. I may have already done some of that here. The tricky part will be figuring out how to install the Jupyer notebook dependencies and get the entry point configured properly so a user can run docker run -it or something similarly trivial to start the container.

Check out the Jupyter project for ideas. In particular...

• https://github.com/jupyter/repo2docker
• https://github.com/jupyter/docker-stacks

...are relevant.

[ghost]

Will do. My initial thought is that docker compose would be the right tool for the job, though I'll have a close look after work.

[davidrpugh]

@ibillett That is a great idea! Perhaps if we isolate the Java 8 and Scala services which produce the JSON output from the Python based stack that consumes the JSON output, we can have two docker containers and compose them.

Side benefit of this approach is that we could have a another container running a full-blown web application front-end and compose it with the same Scala and Java 8 container running the back-end simulation. Developing a standalone web application which plots streaming data and computes statistics is another project worth doing.

[ghost]

Reading into docker-compose, it seems overkill for this example.

It looks like the Scala container would need some kind of networking RPC process to allow the Jupyter container to trigger the simulations, and then share a data volume so Jupyter could pick up ouput.json.

Seemingly the path of least resistance here is just a chunky Dockerfile. We can leverage existing scala-sbt images though, and build miniconda dependencies on top.

[davidrpugh]

@ibillett Let's start get a Dockerfile up and running to start. I really like your idea to use Docker compose to combine the two containers and want to come back to it later. I will do my own digging as well on Docker compose...

[ghost]

Making progress with this here.

I keep bumping into java.io.IOException: Permission denied errors when running sbt not as root. Clearly permissions are not set properly on some folder / file and I can't quite figure out which one :thinking:

[davidrpugh]

I would guess that this exception is being thrown when the simulation attempts to write output to output.json in the project root directory.

First, instead of copying over files into the user $HOME directory, I would use git to clone the repo. Something like...

RUN git clone https://github.com/EconomicSL/auctions-simulation-example.git

Depending, you might need to install git which you can do by adding git to the apt-get commands as follows.

apt-get install -y bzip2 git wget &&

This might solve the problem. If it doesn't it will simplify the Dockerfile.