search

jupyterhub / mybinder.org-deploy

Deployment config files for mybinder.org
https://mybinder-sre.readthedocs.io/en/latest/index.html
BSD 3-Clause "New" or "Revised" License
76 stars 74 forks source link

Enable SMTP with Grafana #365

Open choldgraf opened 6 years ago

choldgraf commented 6 years ago

If we do this, we'll be able to set up email notifications when certain metrics cross thresholds we set (e.g. % failing pods)

Here's the section we'd need to edit:

https://github.com/jupyterhub/mybinder.org-deploy/blob/staging/mybinder/values.yaml#L155

here's the configuration guide for SMTP:

http://docs.grafana.org/installation/configuration/#smtp

I'm a bit rusty on SMTP stuff...is there a way for us to use an SMTP server without setting up an email account somewhere?

We could also use SMTP for google with mybinderteam@gmail.com

choldgraf commented 6 years ago

Another snag: I can't figure out how to set this up without hard-coding the email password in our mybinder.org config file...maybe we can generate a cert/key file and have it sitting in the grafana pod?

yuvipanda commented 6 years ago

Grafana allows interpolating environment variables into its config, so you can set extra environment variables in the secrets in this repo (via https://github.com/kubernetes/charts/pull/2903) and use that in the grafana config.

I'm not sure if you have the secret key to unlock the secret config files in this repo tho...

On Tue, Feb 13, 2018 at 8:24 AM, Chris Holdgraf notifications@github.com wrote:

Another snag: I can't figure out how to set this up without hard-coding the password in our mybinder.org config file...maybe we can generate a cert/key file and have it sitting in the grafana pod?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/jupyterhub/mybinder.org-deploy/issues/365#issuecomment-365319889, or mute the thread https://github.com/notifications/unsubscribe-auth/AAB23sbi6U90p20nKfQy6FD_DAef779jks5tUbdZgaJpZM4SDUPw .

-- Yuvi Panda T http://yuvi.in/blog

minrk commented 6 years ago

@choldgraf do you want to test out the process I describe in #370 for sharing the git-crypt secret?

choldgraf commented 6 years ago

(I'll try and get to this when I have a moment...I think a first blocking step is that @yuvipanda needs to give me git-crypt access)

minrk commented 6 years ago

SMTP outbound is also blocked at the cluster-level for spam reasons in the firewall. We'll need to figure out how to let grafana send email without allowing user pods or builds to do it.

It might be time to move our firewall from node-level to NetworkPolicy, if gke's been updated to have a recent-enough calico to support egress policies yet.

minrk commented 6 years ago

@choldgraf I sent you an encrypted key via gitter PM with a link to the docs for installing it yourself. Let me know how it goes!

choldgraf commented 6 years ago

I'm going to prioritize this issue for myself, as we missed an outage because the grafana emailing wasn't set up yet. I will try to spend some time on it either this week or this weekend.

minrk commented 6 years ago

I just set up an alert with stackdriver that will hopefully let us know when launches are failing, but it would be great to have this coming from grafana.

choldgraf commented 6 years ago

cool - am gonna give the git-crypt stuff another go now that I'm back from Europe

choldgraf commented 6 years ago

hey all - so looking through this now...a couple quick questions:

  1. Can somebody generate a new ssh-vault key for me? I had to reset my SSH keys so I don't think the previous one will work.
  2. Is there any way to test this out other than just deploying to prod?
  3. I set up a google account called mybinderbot@gmail.com that we can use to send emails etc. Are folks OK with me setting a temporary password and hard-coding it in to the grafana.ini file so I can try testing this out separately from the travis git-crypt stuff?

Lemme know what you think!

cc @minrk and @yuvipanda since they've got the most experience with the git-crypt secrets etc

minrk commented 6 years ago

Can somebody generate a new ssh-vault key for me? I had to reset my SSH keys so I don't think the previous one will work.

Sent a new vault via gitter PM

Is there any way to test this out other than just deploying to prod?

You can deploy to staging and test there. Put the config in config/staging.yaml instead of the shared mybinder/values.yaml. You can also test by creating a dedicated cluster on binder-staging and deploying the chart there (I've done this with a few things in the past).

Are folks OK with me setting a temporary password and hard-coding it in to the grafana.ini file so I can try testing this out separately from the travis git-crypt stuff?

If it's not in cleartext. I'd rather not have a publicly accessible email password here. Doing a dedicated cluster should let you test without putting it on this repo.