Add GitLab CI example #6

Closed tamasgal closed 5 years ago

A GitLab CI example would be great. I am currently working on pipelines which deploy tagged versions of our software to our own private docker registry. I will try to figure out a convenient way to create Singularity images and will let you know. Meanwhile if anyone has any feedback or success, please let me know ;)

hey @tamasgal I'd be glad to help you with this! Do you mean an example for a Github CI build for a Singularity Container to a Singularity Registry Server? You would need somewhere to push the final container, and if you use sregistry-cli (note, this is a different repo) you can push to AWS, Google Storage / Drive, Dropbox, etc.) What I can suggest to move forward is:

let's chat about where you are able to push your Singularity container to.
- The fallback is "nowhere" in which case you use Docker and then pull from there, singularity pull docker://username/repo.
  - If you are able to deploy a Singularity Registry Server, you can push here and then manage pulling / permissions for your users
  - If you want some kind of cloud storage, the sregistry-cli client has endpoints for that, so you can push there. You don't get a flashy web interface to show the containers aside from what the storage provides, but you do get a search interface, etc.

If it's okay with you, I'd like to transfer this issue to the sregistry-cli board (where I also manage, so don't worry discussion will keep going!)

Another note is that I'm currently working on having it easy to integrate the cloud storage (e.g., Google Cloud Storage) with Singularity Registry Server, so at some future point you can use Singularity Registry Server with something like a Google Storage backend.

Thanks Vanessa for the quick reply!

I am at the very beginning of the Singularity integration, so I still need to figure out where to store the image. The current plan is just build them and store them in the GitLab artifacts, so we can simply download them to our HPC farms, where Singularity is installed.

However, I could also set up a standalone private singularity registry and use that for the CI procedure. I guess that should be the final aim anyways, for a nice setup.

Currently I even struggle to build an image from our private docker registry due to "incorrect credentials", although it has anonymous pull access.

Sorry to mix this in, but this might be relevant. Here is the output of the build attempt:

> sudo singularity build test.img Singularity
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
ERROR Error getting token for repository test/test_image, please check your credentials.
Extra data: line 1 column 5 (char 4)
Cleaning up...

With the following Singularity file:

Bootstrap: docker
Registry: our.own.docker.registry
Namespace: test
From: test_image:latest

%setup
    mkdir -p $SINGULARITY_ROOTS/.singularity.d/env
    echo "
    source /usr/local/root/bin/thisroot.sh
    source setenv.sh
    " >> $SINGULARITY_ROOTFS/.singularity.d/env/80-custom.sh
    chmod u+x $SINGULARITY_ROOTFS/.singularity.d/env/80-custom.sh

%post
    sed -i '/Setting environment/d' /setenv.sh

If it's okay with you, I'd like to transfer this issue to the sregistry-cli board (where I also manage, so don't worry discussion will keep going!)

Absolutely!

Another note is that I'm currently working on having it easy to integrate the cloud storage (e.g., Google Cloud Storage) with Singularity Registry Server, so at some future point you can use Singularity Registry Server with something like a Google Storage backend.

Sounds great, since we also have a Google Storage backend.

fwoop we've traveled through hyperspace, and we hit another issue board planet! Here are some quick response:

I am at the very beginning of the Singularity integration, so I still need to figure out where to store the image. The current plan is just build them and store them in the GitLab artifacts, so we can simply download them to our HPC farms, where Singularity is installed.

Oh Wow! I didn't know GitLab had artifacts! This could be fantastic - a totally easy and free way to build and then store the images. We would just need a programmatic way to get them, and perhaps GitLab has an API or similar that can be used. For CircleCI artifacts, they don't advocate for using artifacts for "production" things, but it would be worth to try for a quick way to build and test an image from an environment that you don't have sudo. Let's hold off on the API side for now and I'll start to look into a GitLab build recipe.

However, I could also set up a standalone private singularity registry and use that for the CI procedure. I guess that should be the final aim anyways, for a nice setup.

The CI procedure wouldn't happen on the registry, but rather you would push there from it. For example, this would be in the deply step:

export SREGISTRY_CLIENT=registry
sregistry push --name collection/container /path/to/container.simg

Currently I even struggle to build an image from our private docker registry due to "incorrect credentials", although it has anonymous pull access.

Hmm I was going to suggest exporting SINGULARITY_DOCKER_USERNAME and SINGULARITY_DOCKER_PASSWORD but it sounds like this shouldn't be needed. Maybe try it anyway? Your build looks okay - does your registry have https? If not, try exporting SINGULARITY_NOHTTPS=true Otherwise, I think it would be worth opening an issue on the main Singularity board https://www.github.com/sylabs/singularity/issues

Oh wait, hehe. The header is not specified as it should, nevermind the above! Try this first:

Bootstrap: docker
From: our.own.docker.registry/test/test_image:latest

Try that first.

fwoop we've traveled through hyperspace, and we hit another issue board planet!

🙈

Oh Wow! I didn't know GitLab had artifacts! This could be fantastic - a totally easy and free way to build and then store the images

Yep! That's there and is fine 😄

The CI procedure wouldn't happen on the registry, but rather you would push there from it. For example, this would be in the deply step:
export SREGISTRY_CLIENT=registry
sregistry push --name collection/container /path/to/container.simg

Yeah I know, basically the same as for the docker push 👍

Oh wait, hehe. The header is not specified as it should, nevermind the above! Try this first:

Hmm, I thought it's the same, but more "verbose". However, I tried also the one-liner without the separate Namespace and Registry definition but I get the same error. Also when I define SINGULARITY_DOCKER_USERNAME and SINGULARITY_DOCKER_PASSWORD and SINGULARITY_NOHTTPS=true, I get the same token error.

Otherwise, I think it would be worth opening an issue on the main Singularity board https://www.github.com/sylabs/singularity/issues

Yes, that's what I'll do now. Thanks so far!

So I guess when this works I can proceed with playing around building the Singularity image using a privileged GitLab Runner in Docker (that one works for building and pushing Docker images too 😉).

That sounds good! For building Singularity on GitLab, hold off on this, I think I can do this for you and do a writeup. I've actually never used GitLab extensively so I want to learn the ins and outs, etc. I shouldn't take longer than a few days tops, I already have a good start! --> https://code.stanford.edu/v/singularity-ci

I'm not sure if you can see that, it's stanford's hosted Gitlab and maybe they have weird settings. Anyway - I'm excited to work on this and will likely finish up pretty quickly, I'm taking a quick break to go for a run before it gets too late - I'll be back in a few hours to keep working! Keep me updated and let me know if you have other questions I can help with.

Vanessa, thanks for joining me 😄 I really appreciate any help.

I shouldn't take longer than a few days tops, I already have a good start! --> https://code.stanford.edu/v/singularity-ci

I'm not sure if you can see that, it's stanford's hosted Gitlab and maybe they have weird settings.

Yes, I have access and the YAML looks like a good starting point for me either!

Btw. I filed an issue regarding the Singularity build: https://github.com/sylabs/singularity/issues/2377

okay, I wound up moving to GitLab.com because our hosted version doesn't have runners! It looks like the build and then artifact save that I did works! --> https://gitlab.com/singularityhub/gitlab-ci/-/jobs/122059246

I'll be back to work within the hour, and I'm going to first look into the feasibility of having interaction with artifacts, then I'll write a quick post to give more detail for you and others. I'll update you here when I finish it up.

Wow, that looks already nice! Let me know if I can somehow help.

I can also provide a runner with root privileges if needed 😄 but it seems you got all the ingredients...

Nope found a runner! I'm working on the sregistry client now - I don't think I'm going to finish tonight, but definitely won't take longer than a day or two.

hey @tamasgal I made good progress, but stopping to make dinner soon! I'll leave a question here that I can't quite find the answer to - I'm making the search endpoint and going by the docs here --> https://docs.gitlab.com/ee/user/project/pipelines/job_artifacts.html#downloading-artifacts and I'm not exactly sure what a "ref" is. If it's some random alphanumerical string we probably can't do the search. But if it is findable or predictable in any way (maybe an API endpoint that returns the latest ref?) I think we will be good to go.

Ok I found the url for the browser, but now need to figure out json...

https://gitlab.com/singularityhub/gitlab-ci/-/jobs/artifacts/master/browse?job=build

It says in the docs you can add ".json" to any endpoint and it should return it, but I got a 500 server error :thinking:

Oh nevermind, I'm looking in the wrong place entirely! Here are the docs I need: https://docs.gitlab.com/ee/api/jobs.html Yes this should fit the bill :)

hey @tamasgal work is done! https://github.com/singularityhub/sregistry-cli/pull/164 Would you care to test? Make sure to remove all old versions of the client first:

pip uninstall sregistry

Do that until there are no more. Then install the branch

cd /tmp
git clone -b add/gitlab https://www.github.com/singularityhub/sregistry-cli
cd sregistry-cli
python setup.py install

See the markdown provided in the PR description to get a walkthrough, and let me know if anything is unclear. I'm going to tweak the build a bit and then add the GitLab builder to the repo here, and with your blessing will merge the PR. I've already tested quite a bit so I'm reasonably happy with things working.

Amazing, Vanessa! Many thanks, I'll test it right now!

If it makes it easier, since it's deployed you can use the (pretty rendered) docs now https://singularityhub.github.io/sregistry-cli/client-gitlab

I'm also done with the post and writeup, so when you give your blessing I'll share that with the rest of our container friends :)

Nice! 😄

I am currently fighting with some environmental issues, since Python is not found in the build.sh although it's installed via conda.

I cloned your repository and added image: debian:stretch as the base image (otherwise it falls back to our standard CentOS image, which is used for our internal software) and had to add bzip2 to the apt install step. These are the only changes I made so far.

This is the log output (for the sake of completeness), but I will now start to debug it in a live container:

$ /bin/bash .gitlabci/build.sh Singularity

Image Recipe: Singularity
Storage Client: none
Creating Singularity.simg using Singularity...
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
/usr/bin/env: 'python': No such file or directory
Cleaning up...
ERROR: Job failed: exit code 1

I am now trying with the standard Python from the apt repository:

image: debian:stretch

before_script:
  - apt-get update -qq && apt-get install -y wget bzip2 python-pip
  - pip install sregistry[all]
  - .gitlabci/setup.sh

build:
  script:
     - .gitlabci/build.sh Singularity
     - mkdir -p build && cp *.simg build
     - mkdir -p build && cp Singularity* build

  artifacts:
      paths:
        - build/Singularity.simg
        - build/Singularity

It works, well done!

$ /bin/bash .gitlabci/build.sh Singularity

Image Recipe: Singularity
Storage Client: none
Creating Singularity.simg using Singularity...
Using container recipe deffile: Singularity
Sanitizing environment
Adding base Singularity environment to container
Docker image path: index.docker.io/library/ubuntu:16.04
Cache folder set to /root/.singularity/docker

[1/4] ||----------------------------------|   0.0% 
[1/4] |========|--------------------------|  25.0% 
[2/4] |=================|-----------------|  50.0% 
[3/4] |==========================|--------|  75.0% 
[4/4] |===================================| 100.0% 
Exploding layer: sha256:18d680d616571900d78ee1c8fff0310f2a2afe39c6ed0ba2651ff667af406c3e.tar.gz
Exploding layer: sha256:0addb6fece630456e0ab187b0aa4304d0851ba60576e7f6f9042a97ee908a796.tar.gz
Exploding layer: sha256:78e58219b215b359fe002f0ca1f416617b75ca9b36cb274c98d7a5f808711179.tar.gz
Exploding layer: sha256:eb6959a66df2ea26a26452ba11f84fb64bb45af90204374862ec71ee59f795e7.tar.gz
Exploding layer: sha256:c6a9ef4b9995d615851d7786fbc2fe72f72321bee1a87d66919b881a0336525a.tar.gz
User defined %runscript found! Taking priority.
Adding runscript
Finalizing Singularity container
Calculating final size for metadata...
Skipping checks
Building Singularity image...
Singularity container built: Singularity.simg
Cleaning up...
Testing the image... Marco!
Polo!
Skipping upload. Image Singularity.simg is finished!
$ mkdir -p build && cp *.simg build
$ mkdir -p build && cp Singularity* build
Uploading artifacts...
build/Singularity.simg: found 1 matching files     
build/Singularity: found 1 matching files          
Uploading artifacts to coordinator... ok            id=6206 responseStatus=201 Created token=RDMT4SL8
Job succeeded

Running the Singularity container:

░ tgal@staticbox:~/tmp/sing/build
░ 22:46:01 > singularity run Singularity.simg Marco
Polo Marco!

Many thanks again 😸 Now I'll prepare a base Docker image and set up a GitLab runner for our collaboration.

Awesome! Yes likely if you have your own GitLab instance there would be subtle differences. Would it help here to add a line to add the conda's python to the path first?

I just created a merge request to use the "official" Python version. I think that's the easiest solution :)

See here: https://gitlab.com/singularityhub/gitlab-ci/merge_requests/1 https://gitlab.com/singularityhub/gitlab-ci/merge_requests/1/diffs

The issue I see there is that there isn't a pip included with it, shall we add that too?

It is, I actually only added python-pip, which will pull python as dependency. It works fine on our GitLab instance!

I think we would want to install python 3 though :thinking: Many of the singularity registry client endpoints would warrant it. What about python 3 instead? I'll take a look at the PR in a few minutes, just getting home after a run!

Sure, if Python 3 is needed - I fully support it, as a Python expert - then conda is an option (I have to admit, I am not a conda friend though 🙈). Or simply using python:3 as base image (which as far I as know also runs on a Debian image).

Take your time, no hurries ;)

singularityhub / singularity-ci

Add GitLab CI example #6