Work on Docker for SLICER and PLUS #34

Closed tdowrick closed 3 years ago

In GitLab by @MattClarkson on Dec 14, 2018, 17:03

OK, so Mian was working on Docker anyway, as we want to deploy Slicer and PLUS on our demo machines in the Mock OR. When Issue was raised, we quickly installed slicer on weisslab-win and murbella. Turns out this is not sufficient, due to lack of Slicer headers apparently.

But this is where my knowledge of Slicer is lacking.

So, please can Mian talk/worth with @DavidPerez-Suarez @RolandGuichard @AnastasisGeorgoulas to work out what kind of docker image is required, how it differs from the docker image he was already working on, and then how it can be deployed onto a CI machine to enable sufficient unit testing.

In GitLab by @MianAhmad on Dec 17, 2018, 09:57

@DavidPerez-Suarez @RolandGuichard @AnastasisGeorgoulas I would like to get your requirements. We could discuss more. Here is what I have so far and what my understanding is:

Starting from base docker image:

If you would like to start from base image with minimum functionality than we could use the docker image I created. This image has ssh and systemd enabled out of the box. It is in development and therefore I pushed it to my personal dockerhub registry accounthttps://hub.docker.com/r/mianasbat/centos-systemd-ssh. The instructions to run it are there.

We could add slicer and plus etc with whatever dependencies are required and make another image for use. If you dont need ssh and systemd than we could start from clean slate too. Please let me know if you would like to meet or need more information.

Starting from slicer built docker images:

There are also pre-built slicer images available to use. So you could directly use it but again we have to check weather they have your required packages or not? or how the required packages could be added. Some of the options of docker slicer images are

mian@tdowrick2~ $ docker search slicer
NAME                                  DESCRIPTION                                     STARS               OFFICIAL            AUTOMATED
slicer/slicerexecutionmodel           Slicer command line interface                   2                                       
dit4c/dit4c-container-slicer                                                          0                                       [OK]
slicer/slicer-base                                                                    0                                       
girder/slicer_cli_web                 A Girder plugin for exposing slicer executio…   0                                       [OK]
slicer/slicer-build                   An image containing a Slicer build tree alon…   0                                       
seedinnovations/slicer                                                                0                                       
slicer/slicer-dependencies                                                            0                                       
slicer/slicer-test                                                                    0                                       
cdeepakroy/slicer_cli_web_plugin      A sample plugin of girder/slicer_cli_web wit…   0                                       [OK]
lucasgandel/slicerexecutionmodel      SlicerExecutionModel                            0                                       [OK]
mayoqin/slicer                        Grunt and Slicer                                0                                       [OK]
slicer/buildenv-qt5-centos7                                                           0                                       
iomoss/slicer-server                                                                  0                                       
turnerlabs/video-slicer                                                               0                                       
kennethjiang/octoprint-with-slicers   OctoPrint with Cura and Slic3r plugins pre-c…   0                                       
stevepieper/slicer                                                                    0                                       
saxenaamitmca/slicer3d                                                                0                                       
stevepieper/slicer-chronicle                                                          0                                       
slicer/buildenv-qt4-centos5                                                           0                                       
stuartlab/slicer                      SLICER trajectory inference algorithm           0                                       [OK]
ihnorton/slicer-build-deps                                                            0                                       
maladmin/jessie-slicer                                                                0                                       
mayeulchassagnard/slicer-cli                                                          0                                       
fedorov/slicerdockers                 unofficial automated build of Slicer docker     0                                       [OK]
stevepieper/slicer3                                                                   0

In GitLab by @MianAhmad on Dec 17, 2018, 09:57

created branch 34-work-on-docker-for-slicer-and-plus

In GitLab by @MianAhmad on Dec 17, 2018, 09:57

mentioned in merge request !3

In GitLab by @RolandGuichard on Dec 17, 2018, 11:00

Hi @MianAhmad That's basically what I am not aware of atm. I've tried to build Slicer from source and I couldn't link it yet to the OpenIGTLink plugin required here, although I also built the latter from source. Is there a way to install plugins in the Docker image ?

In GitLab by @DavidPerez-Suarez on Dec 17, 2018, 20:17

I've spent a bit of a time today with docker and family! It took me till now to find out the build docker is from 4.9.

What I was using:

slicer-build from: docker pull slicer/slicer-build - Note that this is 3GB and 9 months old. It's not maintained!

To start it:

docker run --rm -it --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" -v /tmp/.X11-unix:/tmp/.X11-unix:rw  -v $(pwd):/mnt/skull --name slicer slicer/slicer-build /bin/bash

Where my pwd is where I have cloned this repository. The x11 and display stuff got it from the lazy way on this post.

I've tried the brute force of copying my installed plugins and it didn't work. I've also tried to install them manually by loading the gui... but the gui didn't open. I stopped now trying to build them because the certificates on the container are old and it cannot download stuff from the net - unless updated the ssl.

Next step is to create one from scratch for version 4.10 - revision 27501 as indicated there. We've created a fork and updated it as we want [diff]. I'll try to create an image tomorrow.

In GitLab by @MianAhmad on Dec 17, 2018, 20:57

Hi @DavidPerez-Suarez it sounds that we are getting there. Would it not work if we pull the non maintained docker docker pull slicer/slicer-build image, log into it and update the OS then update the slicer and install the required plugin and then make a new image of it for our use.

In GitLab by @MianAhmad on Dec 19, 2018, 12:00

I am checking the following combination

To run graphical application in Docker http://fabiorehm.com/blog/2014/09/11/running-gui-apps-with-docker/
To build latest slicer instead of firefox in step 1, https://www.slicer.org/wiki/Slicer3:Build_Instructions#All-in-one_Script_to_checkout_and_build_Slicer3

Any comments?

In GitLab by @RolandGuichard on Dec 19, 2018, 12:05

I'm happy to go down that route as the Slicer build from source doesn't work on my Mac.

In GitLab by @RolandGuichard on Dec 19, 2018, 12:08

The thing is installing the required SlicerIGTLink and/or (?) the OpenIGTLink plugins is easy from the Slicer GUI. Not so much from source.

In GitLab by @MianAhmad on Dec 19, 2018, 12:09

Yes, thats what I am looking for if I get the GUI of slicer from docker. Let see how far I go :)

In GitLab by @DavidPerez-Suarez on Dec 19, 2018, 12:30

@MianAhmad

Would it not work if we pull the non maintained docker docker pull slicer/slicer-build image, log into it and update the OS then update the slicer and install the required plugin and then make a new image of it for our use.

It is easier to build from scratch as we can then have the dockerfile to generate them. I did so yesterday, but one of the commands deleted a cmake file that we need in order to build extra plugins. I'm building it again.

To run graphical application in Docker

I managed to do so as explained in my comment above and the link within. We don't really need this for production and for CI may be a bit more work if they are headless.

As soon as the build for the latest stable (4.10) completes I will push it to docker hub so all of us can use it without spending lot of time to build on our machines. Note this is only to run the testing on the same infrastructure that the rest of the community is doing. For the final user it doesn't matter how slicer is installed in the machine.

In GitLab by @MianAhmad on Dec 19, 2018, 12:33

That sounds very good to me.

In GitLab by @MianAhmad on Dec 19, 2018, 14:50

For fun and understanding I tried to run firefox and here is what worked for me on mac.

STEPS

Install Docker
Install XQuartz

Run the command and in security tab, enable “Allow connections from network clients”

open -a XQuartz

To allow connections from your local machine, first command will get you the ip and the second will add it to the list.

xhost + 127.0.0.1

Docker command

docker run -e DISPLAY=host.docker.internal:0 jess/firefox

Firefox browser will open if all goes well.

In GitLab by @MianAhmad on Dec 19, 2018, 15:59

@DavidPerez-Suarez So the image that you have shared (docker pull slicer/slicer-build) on running using the given command will start gui slicer ? because its not doing it on my mac.

docker run --rm -it --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" -v /tmp/.X11-unix:/tmp/.X11-unix:rw  -v $(pwd):/mnt/skull --name slicer slicer/slicer-build /bin/bash

After login to the container do we have to run the command to launch GUI or will it work in commandline mode?

In GitLab by @DavidPerez-Suarez on Jan 3, 2019, 18:05

The new slicer-build under uclrits (docker pull uclrits/slicer-build:410) has also the OpenIGTLink installed.

For mac, this seems to work: https://cntnr.io/running-guis-with-docker-on-mac-os-x-a14df6a76efc However, we are having OpenGL problems :disappointed:

In GitLab by @DavidPerez-Suarez on Jan 3, 2019, 18:57

Ok, I think I've solved a bit the OpenGL problem... I'm building a new version of the container. Let's see how it's tomorrow. There may still be a problem with hardware acceleration, this may be a solution: http://gernotklingler.com/blog/howto-get-hardware-accelerated-opengl-support-docker/ Maybe it also needs: --device /dev/dri additionally to the --privileged.

In GitLab by @DavidPerez-Suarez on Jan 4, 2019, 17:54

Ok, so the latest update of the docker container includes the plugins and it works on my linux machine with the following command:

docker run --rm -it --env="DISPLAY" --env="QT_X11_NO_MITSHM=1" -v /tmp/.X11-unix:/tmp/.X11-unix:rw  --device /dev/dri --privileged -v $(pwd):/mnt/skull --name slicer uclrits/slicer-build:410 /bin/bash
./Slicer

I don't know whether the hardware acceleration is host dependant, so I would need other people to try it. Meanwhile, the current version of sbn and the slicer works. Note however that the to make the Extension manager to work you may need to start slicer as:

./Slicer --no-sandbox

I think this is because the docker user is root.

In GitLab by @MianAhmad on Jan 7, 2019, 11:51

@DavidPerez-Suarez I was trying the other way, the way I ran the firefox but running the container and then from inside the container using host machine display sounds interesting.

I tried to run your image on my iMac but got the error, Any idea? I have XQuartz installed on my iMac and I ran xhost + but no luck.

Screen_Shot_2019-01-07_at_11.48.17

In GitLab by @MianAhmad on Jan 7, 2019, 12:12

Okay that is what worked for me on iMac

For newbie: Install docker on mac, make sure the service is running.
Install XQuartz, run XQuartz and from preferences, security tab, check option X11 from network clients.

After that do this

Get host ip

IP=$(ifconfig en0 | grep inet | awk '$1=="inet" {print $2}’)

Add ip to xhost
```
xhost + $IP
```

run the docker command with minor change

docker run --rm -it -e DISPLAY=$IP:0 --env="QT_X11_NO_MITSHM=1" -v /tmp/.X11-unix:/tmp/.X11-unix:rw  --device /dev/dri --privileged -v $(pwd):/mnt/skull --name slicer uclrits/slicer-build:410 /bin/bash

run slicer
```
./Slicer
```

In GitLab by @DavidPerez-Suarez on Jan 7, 2019, 17:34

Great! Happy it worked out!!

In GitLab by @MianAhmad on Jan 10, 2019, 16:15

Hi @DavidPerez-Suarez and @RolandGuichard It is interesting to play with docker and Slicer but I am wondering what is the reason of having Slicer in docker and not installing the binary directly on machine? If I remember correctly docker is not recommended in production.

In GitLab by @RolandGuichard on Jan 10, 2019, 16:20

Mostly for testing purposes. Running tests for the slicelet is embedded in CMake and uses a build version of Slicer. The docker container will allow us to run tests easily in CI, as we won't build Slicer every time we'd like to run tests. Hope it makes sense.

In GitLab by @MianAhmad on Jan 10, 2019, 16:24

Thanks, okay another question. Are we developing component for Slicer for which we need to build Slicer each time we test or we are developing a tool which will use Slicer. In the later case if we install Slicer on the runner then we dont need to build it for every test. Right?

In GitLab by @RolandGuichard on Jan 10, 2019, 16:41

Actually what we'd like is to: 1-Build Slicer in a container with the SlicerIGTLink dependencies (plus eventually the PLUS server) 2-Be able to use the container in CI, as we'd like to automate the tests and run them using CMake but ultimately we don't want the build to happen every time.

In GitLab by @MianAhmad on Jan 10, 2019, 16:49

Okay so if I correctly understand, we could have a ready made docker image to pull from docker hub and run Slicer, dependencies and PLUS server on any machine with docker installed for demo or testing.
But in our environment where we have a gitlab CI/CD setup (kind of automate the testing process) if we configure a runner machine with Slicer, dependencies and PLUS server then that will be good enough to test the tool like in production environment. Does it make sense?

In GitLab by @RolandGuichard on Jan 10, 2019, 16:54

I guess what we'd like is the gitlab CI to pull the docker container every time we commit changes and run tests automatically.

In GitLab by @DavidPerez-Suarez on Jan 10, 2019, 16:57

Both solutions are good, if you can install 3dslicer and all the other dependencies from source (so we have access to the headers) on gitlab, then that's good. The docker file was as a work around and to don't need the ci machines to have that installed. Also for us to be able to run the steps on our environments versus a standar version across the different OSs.

In GitLab by @MianAhmad on Jan 11, 2019, 09:58

Thank you guys for the detailed reply.

So if I understood correctly we can go with two options.

Option 1:

Docker image pulled every time (For second time it will be pulled from the local disk but let say pull every time) which has 3DSlicer according to your requirements and it will test your project after each commit.

Option 2:

We could install 3DSlicer according to the requirements on the physical machine and it will pull the project only to test after each commit.

Could you please share here the Dockerfile of the image. I would like to know the steps you performed to build the docker image. I will try to install a similar setup on the physical machine.

What my view is that the system cant afford latency or extra complexity of docker to begin with. We could go with both the physical runner machine and the docker system in the start if possible. whichever work well we will adopt it or if 3DSlicer with dependencies is not possible on physical machine than there is no other option except docker.

We could compare both systems behaviour and add results to publication. Does that make sense?

In GitLab by @RolandGuichard on Jan 11, 2019, 10:07

It makes sense. However, I hope there will be not too many issues installing Slicer on the runner machines. My experience of building Slicer tells me it's already a pain on my Mac, so accounting for different OSes etc might add extra pain.

In GitLab by @MianAhmad on Jan 11, 2019, 10:28

Okay sounds good.

Pain installing 3DSlicer on mac raise another issue against Docker, only if I am right. Which is

If we use the same docker image on Windows, Linux and Mac then the image is basically running in Docker which is independent from the underlying host operating system. So actually we will be running the same tests on 3 machines without considering the operating system.

So ideally we should physically install 3DSlicer on windows, mac and Linux and run the tests which will give us evaluation of 3DSlicer with respective platforms.

A docker image on any platform will have the same results. We can use if if we want performance evaluation.

In GitLab by @DavidPerez-Suarez on Jan 11, 2019, 11:40

@MianAhmad - the docker file are in https://github.com/UCL-RITS/SlicerDocker/tree/410 slicer-build it depends on the slicer-base from the same repository. I agree that ideally they are natively installed on each OS. However, since our goal is not to produce (not yet...) an application that works over all OSs, but only on Windows, I would say the more reasonable approach is to install it natively on Windows. Mac and Linux could use the docker.

In GitLab by @MianAhmad on Jan 11, 2019, 11:51

Thanks for the link.
Okay sounds good.

In GitLab by @AnastasisGeorgoulas on Jan 15, 2019, 14:39

For future reference, if anyone else is having problems running the Docker container on Mac, I got it working by following @MianAhmad's instructions above, but I also needed to restart my machine after changing the security settings on XQuartz. Restarting XQuartz only was not enough! (inspired by https://stackoverflow.com/a/47309184)

With this, I was able to build the plugin and run ctest.

In GitLab by @MattClarkson on Feb 18, 2019, 21:04

@DavidPerez-Suarez is this one done then? @Mian is not currently working on it. There is a pull request. Can it be merged?

In GitLab by @MattClarkson on Mar 1, 2019, 13:48

We believe done. Closing.

In GitLab by @MattClarkson on Mar 1, 2019, 13:48

closed

UCL / SkullBaseNavigation

Work on Docker for SLICER and PLUS #34

Starting from base docker image:

Starting from slicer built docker images: