Helm charts point to Docker images not available on Dockerhub

[jgerardsimcock]

xref here: https://github.com/pangeo-data/pangeo/issues/337

The two most recent helm-charts are tagged with commits that don't correspond to the image tags on dockerhub.

If I recall, chartpress is not pushing images to dockerhub, right? So this is still a manual, two-step process?

[jacobtomlinson]

Interesting. We should update this to push the docker images as a manual step is not sustainable.

[rabernat]

Yes we should. That’s what #40 should accomplish.

We just need someone with sufficient tragic experience to help get our chartpress configs sorted out. I don’t have the expertise or time to do it myself.

Sent from my iPhone

On Jul 17, 2018, at 5:25 PM, Jacob Tomlinson notifications@github.com wrote:

Interesting. We should update this to push the docker images as a manual step is not sustainable.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

[jgerardsimcock]

Ok. I am happy to help with this. I might need some assistance as I am not sure if the whole chartpress thing applies to the notebook/worker link that pangeo is trying to preserve. We've been unable to deploy both the worker and notebook using Chartpress. Its not clear to me how to add the worker 'valuesPath': value in the chartpress.yaml file. This may be a question for @yuvipanda.

My next attempt was to have travis deploy helm charts after the docker images build by checking out the helm-chart repo, updating and pushing to gh-pages branch. I've been unsuccessful at this so far. But this should in theory work. Do either of you see anything glaringly inaccurate?

sudo: required
services:
  - docker

env:
  global:
    - TAG=2018-07-18.01
  matrix:
    - IMAGE_NAME=notebook
    - IMAGE_NAME=worker
    - IMAGE_NAME=worker-tc

install:
  - cd $IMAGE_NAME
  - docker build --pull --cache-from rhodium/$IMAGE_NAME -t rhodium/$IMAGE_NAME:$TAG .

script:
  - docker images rhodium/$IMAGE_NAME:$TAG

deploy:
  provider: script
  script: docker login -u "$DOCKER_USERNAME" -p "$DOCKER_PASSWORD" && docker push "rhodium/$IMAGE_NAME:$TAG"
  skip_cleanup: true
  on:
    branch: master
after_deploy:
  script:
    - curl https://raw.githubusercontent.com/kubernetes/helm/master/scripts/get | bash
    # set-up deploy key
    - openssl aes-256-cbc -K $encrypted_19fb998dc11c_key -iv $encrypted_19fb998dc11c_iv -in github_deploy_key.enc -out github_deploy_key -d
    - chmod 600 github_deploy_key
    #clone our helm repo
    - git clone git@github.com:RhodiumGroup/helm-chart.git
    #update the tag for the helm chart
    - cd helm-chart
    - python update_helm_tag.py -$TAG
    #create the helm chart with the new tag
    - helm package rhg-hub/
    #add information to gh-pages repo
    - git checkout gh-pages
    - git add rhg-hub*.tgz
    - helm repo index .
    - git commit -am 'add new helm chart tag=$TAG'
    #push to gh-pages
    - git push 
  on:
    branch: master

@jacobtomlinson or @rabernat If I can get some help on the actual deploy, I can take it from there.

[jacobtomlinson]

@jgerardsimcock Thanks for looking at this!

I see the name rhodium littered throughout the script. Shouldn't that be pangeo?

[rabernat]

@jgerardsimcock -- thanks so much for volunteering to help with this!

I would not worry so much about automating the worker build. As discussed in several other issues (e.g. https://github.com/pangeo-data/pangeo/issues/272, https://github.com/pangeo-data/pangeo/issues/316), we would actually prefer to do away with this separation and move to a single docker image which can be used for both notebook and worker.

To create less work, I would recommend building on the almost-working travis / chartpress configuration that we already have. Your travis script above appears to bypass chartpress completely, which I'm not sure is the best way forward. The very small change to the current script introduced in PR #40 is all that is required to enable automatic pushing of the notebook image to dockerhub. This is based on the zero-to-jupyterhub chartpress script (which I recommend we try to emulate as much as possible.)

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/def23e31a3819aeb895df96d4fa26bb8820ea990/ci/deploy.sh#L1-L9

All we need to add to #40 is a line like

docker login -u "${DOCKER_USERNAME}" -p "${DOCKER_PASSWORD}"

But we need to figure out how to encrypt these credentials and store them in the repo correctly. @yuvipanda might be able to help with this.

[jacobtomlinson]

Someone with push access to the pangeo repos on Docker Hub needs to add their username and password to the "Environment Variables" section of the travis settings.

These will be automatically stored in an encrypted way and obfuscated during builds so no concerns from a security stand point.

Happy to add my credentials if someone gives me push perms on the repos.

[rabernat]

I just added @jacobtomlinson and @jgerardsimcock both to the owners list of the docker pangeo organization: https://hub.docker.com/u/pangeo/

[jacobtomlinson]

Thanks!

I've added the env vars. You should now be able to add the login command and it will work.

[rabernat]

Ok, I just tried in https://github.com/pangeo-data/helm-chart/pull/40/commits/4ca6fd11fcfe7c2f2032a3ba89666305daf3c38c

[rabernat]

Let's move this discussion to #40. I got some error messages...

[rabernat]

fixed by #40