search

jade-hpc-gpu / jade-hpc-gpu.github.io

Joint Academic Data Science Endeavour (JADE) is the largest GPU facility in the UK supporting world-leading research in machine learning (and this is the repo that powers its website)
http://www.jade.ac.uk/
Other
24 stars 8 forks source link

Customized container with docker pull #177

Open XiaodanXing opened 2 years ago

XiaodanXing commented 2 years ago

Software Request

Hi, I'd like to know how to use customized containers. I tried to use docker pull in JADE but it was not working (the connection is refused).

Is it possible to download docker containers with "docker pull" and run it with "docker run" in JADE?

twinkarma commented 2 years ago

Instead of using Docker directly, you can use Singularity(https://docs.sylabs.io/guides/3.7/user-guide/) which can pull images from Docker hub.

Here's documentation on using Singularity specifically on JADE: https://docs.jade.ac.uk/en/latest/jade/containers.html#singularity-containers

XiaodanXing commented 2 years ago

Instead of using Docker directly, you can use Singularity(https://docs.sylabs.io/guides/3.7/user-guide/) which can pull images from Docker hub.

Here's documentation on using Singularity specifically on JADE: https://docs.jade.ac.uk/en/latest/jade/containers.html#singularity-containers

Thank you for your reply. However, I couldn't pull containers from the docker hub. The connection was rejected, and it required sudo to login. For example, I tried to pull the docker container tensorflow from dockerhub with

singularity pull tensorflow.sif docker://tensorflow/tensorflow:latest

then I got: 图片

In addition, I tried to build my own singularity container using the docker tar file. First, I pulled the container into my local machine and packed it into a docker tar file. Then I build a singularity container with 

singularity build --sandbox [my_container]  docker-archive://[my_container].tar

Then I tried to use this container interactively with 

singularity run [my_container]

I could successfully enter the container, but all of my dependencies were not there. 图片

twinkarma commented 2 years ago

Which docker container are you trying to use? I can try to put a build script together for you.

XiaodanXing commented 2 years ago

Which docker container are you trying to use? I can try to put a build script together for you.

Hi, thank you for your reply. I am trying to use heatonresearch/stylegan2-ada from the docker hub, and my colleagues might want to use other dockers from the docker hub.

cemde commented 1 year ago

I get the same error message. If I am not mistaken, the compute nodes do not have internet connection. I assume that is the issue.

It would be really good to have real docker on the login node to pull and build images and then launch containerised scripts on compute nodes.