Using Docker Compose instead of Run

[tomowatt]

Hey 👋

I'm a student of the course and I've came across this and I'm quite excited about using Docker for it. I did go through the README and noticed the long Run command being used, and from experience, I know this might cause issues.

So, I'm wondering if you'd be adding in Compose File instead so students only need to run: docker-compose up [--detatch]?

Something like this:

version: "3.5"

services:
  tm351:
    image: ousefuldemos/tm351-binderised:latest
    environment:
      JUPYTER_TOKEN: "letmein"
    volumes:
      - "$PWD/TM351VCE/notebooks:/home/jovyan/notebooks"
      - "$PWD/TM351VCE/openrefine_projects:/home/jovyan/openrefine"
    networks:
      - tm351
    ports:
      - 8888:8888

networks:
  tm351:

I will note, it will mean students will need to add the Notebooks files into the appropriate folders of this repository or update the pathing in the Compose File - unless the Notebooks and other files get included in the repository.

As Docker Compose is now apart of the main Docker installation, and the Compose File can be included in this repository and work as is, I feel it makes sense.

[psychemedia]

Hi @tomowatt

This is a great suggestion and one we should look at for simplifying the container management process.

One of your fellow students has posted an alternative candidate script at https://gist.github.com/robdale110/f9579713588e91c797a50de86bf723c6

I also started working on some instructions, though I still need to check them andget these into the form used for the docker commands in the Software Guide:

If you download this docker-compose.yaml file to your TM351VCE directory, you can run it in detached mode from the commandline / command prompt within that directory by issuing the command: docker-compose up -d You can stop the container by running the command: docker-compose stop and then restart it by issuing the command docker-compose start ; alternatively, to restart a running container, issue the command docker-compose restart Pause/unpause the running of the container using the commands: docker-compose pause / docker-compose unpause Force the container to stop: docker-compose kill Remove stopped container (this will not delete shared folders) : docker-compose rm Shutdown and remove the container (this will not delete shared folders): docker-compose down. To restart the VCE, you would then need to run docker-compose up again. This approach can be used if you break the environment in the container somehow and need to create a fresh one. List running containers: docker-compose ps View container logs / logfile: docker-compose logs Show public ports on the container: docker-compose port

Using docker-compose also provides additional options. For example, we could support smaller containers linked via docker compose (eg a Jupyter server container, a PostgreSQL container, a MongoDB container, an OpenRefine container).

Several reasons why we didn't go for that approach this time round:

1) the monolithic/all in one container has everything in one place;

2) it works with MyBinder (an online temporary free server provider) and the multiuser JupyterHub server;

3) I need to check that docker-compose is always installed with docker and doesn't present an additional installation requirement...

[psychemedia]

I remember now a couple more considerations that prevented me from going with docker-compose:

• there is a requirement to ship the docker-compose.yaml file; it can take weeks, if not months, to get a file onto the VLE, and days to weeks to get an updated file onto the VLE, along with a whole process relating to announcing the change and trying to mitigate against students and ALs using different versions of the file at any one time; the docker run model has no downloaded file dependencies;
• the file needs to work on a cross platform basis; Mac/Linux systems and Windows systems use a different convention for managing file paths, and that may need handling by providing two docker-compose.yaml files, one for each type of o/s, to handle the volume mount paths; docs suggest using . rather than $PWD for docker-compose; I'm not sure of the same forward slash delimiter works for compose files used on Windows and Linux/Mac?

[psychemedia]

The docker-compose.yaml file I am currently testing, and that enables the ports for exposing database service connections (if the corresponding service config files are updated to allow incoming connections from remote/external connections):

version: "3.5"

services:
  tm351:
    image: ousefulcoursecontainers/ou-tm351:current
    environment:
      JUPYTER_TOKEN: "letmein"
    volumes:
      - "./TM351VCE/notebooks:/home/jovyan/notebooks"
      - "./TM351VCE/openrefine_projects:/home/jovyan/openrefine"
    networks:
      - tm351
    ports:
      - 35180:8888
      - 35181:5432
      - 35182:27017
networks:
  tm351:

[tomowatt]

Thanks for getting back in touch, I know you must be busy with this and other important things!

I need to check that docker-compose is always installed with docker and doesn't present an additional installation requirement

In most cases, I guess students would be using either Docker Desktop for Mac/Windows so Compose is already included -> https://docs.docker.com/compose/install/

[tomowatt]

I remember now a couple more considerations that prevented me from going with docker-compose:

• there is a requirement to ship the docker-compose.yaml file; it can take weeks, if not months, to get a file onto the VLE, and days to weeks to get an updated file onto the VLE, along with a whole process relating to announcing the change and trying to mitigate against students and ALs using different versions of the file at any one time; the docker run model has no downloaded file dependencies;

That's quite annoying to hear about but understandable, didn't realise it could take so long to what seems like a simple file additional!

• the file needs to work on a cross platform basis; Mac/Linux systems and Windows systems use a different convention for managing file paths, and that may need handling by providing two docker-compose.yaml files, one for each type of o/s, to handle the volume mount paths; docs suggest using . rather than $PWD for docker-compose; I'm not sure of the same forward slash delimiter works for compose files used on Windows and Linux/Mac?

The / path delimiter for compose files does work on Windows as on Linux/Mac from what I recall - though I've not used Docker on Windows for a while so I'd need to confirm. I only used $PWD cause I find it a bit easier to read.

Using docker-compose also provides additional options. For example, we could support smaller containers linked via docker compose (eg a Jupyter server container, a PostgreSQL container, a MongoDB container, an OpenRefine container).

Yeah that would be a major benefit, I pulled down the Image as it is and it is ~1.4GB but I understand the task and pain of turning a VM into a Container. But definitely would help in future iterations!

[psychemedia]

@tomowatt I checked the docs too and it looks like docker-compose is generally available and the path thing appears not to be an issue (though I don't have a Windows machine to check). If the above docker-compose.yaml file works for you, I'll add it to the repo...

I'll work on the split container version when I get a chance, but that will have a side effect of requiring different connection strings when connecting to the database server containers from the notebook server container (i.e. it will break the connection strings currently provided as defaults in the notebooks we ship...)