Audit our monitoring - Githubissues

jdotpz commented 9 years ago

We want to look at our Opsview/New Relic/Pingdom monitoring, and be sure we're properly monitoring endpoints and infrastructure.

[x] Dump DNS zone lists and list them in pastebins here (mozillascience.org, hive sites, webmaker.org sites, mofoprod / mofostaging, dev, togetherjs, opennews)

Then, for each of those:

[ ] Ensure all production endpoints are monitored in Opsview Mofo Public (ie, is https://webmaker.org/en-US/ being watched?)
[ ] Ensure all databases are monitored properly
[ ] Ensure ELB's are monitored properly
[ ] Ensure we're watching things from more than one place (endpoints, at least)
[ ] Ensure alerts go out for monitors

jdotpz commented 9 years ago

https://pastebin.mozilla.org/8822218 has a list of our hosted zones.

jdotpz commented 9 years ago

Need to grab a few manual ones from: https://pastebin.mozilla.org/8822221

jdotpz commented 9 years ago

Those came from a crappy aws cli tool (cli53) which errored and lost a bunch of records. I reran.

[x] adhoctribution-db.mofoprod.net.
[ ] appmaking.mofoprod.net.
[ ] badgekit-api-mozilla.mofoprod.net.
[ ] badgekit-mozilla.mofoprod.net.
[ ] bleach.mofoprod.net.
[x] bobbydoesrule.mofoprod.net.
[ ] bsd.mofoprod.net.
[ ] butter.mofoprod.net.
[ ] butter.mofoprod.net.
[ ] callcongress.mofoprod.net.
[x] collusion.mofoprod.net.
[ ] collusiondb.mofoprod.net.
[ ] csa-ftf.mofoprod.net.
[ ] csa-wordpress-poc.mofoprod.net.
[ ] csplogger.mofoprod.net.
[ ] discourse.mofoprod.net.
[ ] discoursettw.mofoprod.net.
[ ] etherpad.mofoprod.net.
[ ] geppetto.mofoprod.net.
[ ] hackpub.mofoprod.net.
[x] hiveberlin.mofoprod.net.
[x] jenkinstest.mofoprod.net.
[ ] kount.mofoprod.net.
[ ] loggins.mofoprod.net.
[x] logs.mofoprod.net.
[ ] minecraft.mofoprod.net.
[ ] mozillawewbmakerproxy.mofoprod.net.
[x] newrelic.mofoprod.net.
[ ] node-hubble.mofoprod.net.
[ ] nominatim.mofoprod.net.
[ ] openbadger-wm.mofoprod.net.
[ ] openbadges.mofoprod.net.
[ ] profile2.mofoprod.net.
[ ] redirect.mofoprod.net.
[ ] source.mofoprod.net.
[ ] thimbleproxy.mofoprod.net.
[ ] validator.mofoprod.net.
[ ] webmaker-profile2.mofoprod.net.
[ ] wmscreenshot.mofoprod.net.
[x] 2011mozillafestival.mofostaging.net.
[x] 2013mozillafestival.mofostaging.net.
[x] aestimia-cem.mofostaging.net.
[x] badgekit-api-mozilla.mofostaging.net.
[x] badgekit-api.mofostaging.net.
[x] badgekit-mozilla.mofostaging.net.
[x] badgekit.mofostaging.net.
[x] bakery.mofostaging.net.
[ ] bleach.mofostaging.net.
[ ] blogmozillaignite.mofostaging.net.
[ ] butter.mofostaging.net.
[ ] butter.mofostaging.net.
[ ] callcongress.mofostaging.net.
[x] collusion.mofostaging.net.
[ ] collusiondb.mofostaging.net.
[x] csa-discourse.mofostaging.net.
[x] csol-aws.mofostaging.net.
[x] csol.mofostaging.net.
[x] csplogger.mofostaging.net.
[x] drumbeat.mofostaging.net.
[ ] events-api.mofostaging.net.
[ ] events.mofostaging.net.
[x] fifi.mofostaging.net.
[x] firefox10.mofostaging.net.
[ ] fundraising.mofostaging.net.
[ ] goggles.mofostaging.net.
[x] ignite.mofostaging.net.
[ ] login.mofostaging.net.
[ ] makeapi.mofostaging.net.
[ ] makedrive.mofostaging.net.
[ ] makevalet.mofostaging.net.
[ ] marketpulse.mofostaging.net.
[ ] metrics.mofostaging.net.
[ ] mobile-appmaker.mofostaging.net.
[ ] mozfest.mofostaging.net.
[x] mxmx-mars.mofostaging.net.
[x] mxmx-rover.mofostaging.net.
[ ] node-hubble.mofostaging.net.
[ ] openbadger-cem.mofostaging.net.
[ ] openbadger-csol.mofostaging.net.
[ ] openbadger-wm.mofostaging.net.
[ ] openbadgerdemo.mofostaging.net.
[ ] openbadges.mofostaging.net.
[ ] openbadgesdev.mofostaging.net.
[ ] openbadgesorg.mofostaging.net.
[ ] opennews.mofostaging.net.
[x] parapara-editor.mofostaging.net.
[x] parapara.mofostaging.net.
[ ] popcorn.mofostaging.net.
[ ] profile.mofostaging.net.
[ ] redirect.mofostaging.net.
[ ] screenshot.mofostaging.net.
[x] secretrobotron.mofostaging.net.
[ ] butter-mx.secretrobotron.mofostaging.net.
[ ] dashboard.secretrobotron.mofostaging.net.
[ ] studiomofo.mofostaging.net.
[x] teambuilding.mofostaging.net.
[ ] thimble.mofostaging.net.
[ ] validator.mofostaging.net.
[ ] webmaker-profile2.mofostaging.net.
[ ] webmaker-publisher.mofostaging.net.
[ ] webmaker.mofostaging.net.
[ ] wpchild1.mofostaging.net.
[x] blog.jdotp.org.
[x] devopsagainsthumanity.jdotp.org.
[x] journal.jdotp.org.
[x] lol.jdotp.org.
[x] pad.jdotp.org.
[x] www.jdotp.org.
[ ] chicagosummeroflearning.org.
[ ] badgestudio.chicagosummeroflearning.org.
[x] dashboard.chicagosummeroflearning.org.
[x] status.chicagosummeroflearning.org.
[ ] openbadges.org.
[ ] badgekit.openbadges.org.
[ ] badges-101.openbadges.org.
[x] dashboards.openbadges.org.
[x] dev.openbadges.org.
[ ] discovery.openbadges.org.
[x] oldbeta.openbadges.org.
[ ] planet.openbadges.org.
[ ] stage.openbadges.org.
[x] stats.openbadges.org.
[ ] traffic.openbadges.org.
[ ] validator.openbadges.org.
[ ] www.openbadges.org.
[x] dlester.mofodev.net.
[ ] login.mofodev.net.
[ ] makeapi.mofodev.net.
[x] test.mofodev.net.
[ ] webcomp.mofodev.net.
[x] chicagosummeroflearning.net.
[x] www.chicagosummeroflearning.net.
[x] chicagosummeroflearning.com.
[x] www.chicagosummeroflearning.com.
[ ] webmaker.org.
[ ] api.webmaker.org.
[ ] badges.webmaker.org.
[ ] blog.webmaker.org.
[ ] book.webmaker.org.
[ ] cinemagif.webmaker.org.
[ ] dashboards.webmaker.org.
[ ] discourse.webmaker.org.
[ ] eoy.webmaker.org.
[ ] events-api.webmaker.org.
[ ] events.webmaker.org.
[ ] goggles.webmaker.org.
[ ] hackpub.webmaker.org.
[ ] login.webmaker.org.
[ ] makeapi.webmaker.org.
[ ] mars.webmaker.org.
[ ] metrics.webmaker.org.
[ ] pad.webmaker.org.
[ ] performance.webmaker.org.
[ ] podcast.webmaker.org.
[ ] popcorn.webmaker.org.
[ ] profile.webmaker.org.
[ ] realtime.webmaker.org.
[ ] stats.webmaker.org.
[ ] status.webmaker.org.
[ ] stuff.webmaker.org.
[ ] thimble.webmaker.org.
[ ] traffic.webmaker.org.
[ ] transifex.webmaker.org.
[ ] www.webmaker.org.
[ ] mozillawebmaker.info.
[x] \052.maker.ly.
[ ] mywebmaker.org.
[ ] \052.mywebmaker.org.
[x] makes.org.
[ ] \052.makes.org.
[ ] stopwatching.us.
[ ] popcorn.webmadecontent.org.
[ ] mozillathimblelivepreview.net.
[ ] workopen.org.
[ ] opennews.org.
[x] dashboards.opennews.org.
[ ] source.opennews.org.
[x] sourcenewrelic.opennews.org.
[ ] srccon14.opennews.org.
[x] status.opennews.org.
[x] traffic.opennews.org.
[ ] www.opennews.org.
[ ] mozillascience.com.
[ ] www.mozillascience.com.
[ ] mozillascience.org.
[ ] collaborate.mozillascience.org.
[ ] dashboards.mozillascience.org.
[ ] forum.mozillascience.org.
[ ] status.mozillascience.org.
[x] wp.mozillascience.org.
[ ] badgekit.org.
[ ] api.badgekit.org.
[x] apinewrelic.badgekit.org.
[x] apiperformance.badgekit.org.
[x] dashboards.badgekit.org.
[ ] geoperformance.badgekit.org.
[x] newrelic.badgekit.org.
[x] performance.badgekit.org.
[x] status.badgekit.org.
[ ] www.badgekit.org.
[ ] hackasaurus.org.
[ ] webxray.hackasaurus.org.
[ ] lightbeamdb.org.
[ ] studiomofo.org.
[ ] dashboards.studiomofo.org.
[ ] status.studiomofo.org.
[ ] www.studiomofo.org.
[ ] mozillafestival.org.
[ ] 2011.mozillafestival.org.
[ ] 2012.mozillafestival.org.
[ ] 2013.mozillafestival.org.
[x] content-2011.mozillafestival.org.
[x] content-2012.mozillafestival.org.
[x] content-2013.mozillafestival.org.
[x] dashboards.mozillafestival.org.
[x] status.mozillafestival.org.
[x] www.mozillafestival.org.
[x] knightmozilla.org.
[x] www.knightmozilla.org.
[x] mozillapopcorn.org.
[x] mozillaopennews.org.
[x] source.mozillaopennews.org.
[x] www.mozillaopennews.org.
[ ] drumbeat.org.
[ ] beta.drumbeat.org.
[ ] www.drumbeat.org.
[ ] mozillaignite.org.
[ ] blog.mozillaignite.org.
[ ] content.mozillaignite.org.
[ ] www.mozillaignite.org.
[x] knightmozilla.com.
[x] www.knightmozilla.com.
[ ] mozillawebmakerproxy.net.
[ ] webmakerprototypes.org.
[x] dashboards.webmakerprototypes.org.
[ ] discourse.webmakerprototypes.org.
[ ] personas.webmakerprototypes.org.
[x] status.webmakerprototypes.org.
[x] traffic.webmakerprototypes.org.
[ ] webmakerconnect.org.
[ ] www.webmakerconnect.org.
[ ] togetherjs.com.
[x] dashboards.togetherjs.com.
[ ] hub.togetherjs.com.
[ ] origin.togetherjs.com.
[x] status.togetherjs.com.
[x] devopsagainsthumanity.net.
[ ] hivelearningnetworks.org.
[ ] cupcakes.hivelearningnetworks.org.
[ ] directory.hivelearningnetworks.org.
[ ] discourse.hivelearningnetworks.org.
[x] status.hivelearningnetworks.org.
[ ] www.hivelearningnetworks.org.
[x] hivelearningnetwork.org.
[ ] www.hivelearningnetwork.org.
[x] hivelearningnyc.org.
[ ] www.hivelearningnyc.org.
[x] hivelearningnyc.com.
[ ] www.hivelearningnyc.com.
[x] hivelearningchicago.com.
[ ] www.hivelearningchicago.com.
[ ] webmak.es.
[ ] \052.webmak.es.
[ ] mozilla-community.org.
[ ] discourse.mozilla-community.org.
[ ] guides.mozilla-community.org.
[ ] l10n.mozilla-community.org.
[ ] wpms.lb.mozilla-community.org.
[ ] ns1.mozilla-community.org.
[ ] ns2.mozilla-community.org.
[ ] ops.mozilla-community.org.
[ ] osticket-demo.mozilla-community.org.
[ ] wpms.mozilla-community.org.
[ ] csa-discourse-dev.communitysysadmins.org.
[ ] badg.us.
[ ] hiveberlin.org.
[ ] www.hiveberlin.org.
[ ] hivekc.org.
[ ] status.hivekc.org.
[ ] www.hivekc.org.
[ ] hivecha.org.
[ ] status.hivecha.org.
[ ] www.hivecha.org.
[ ] hivevan.org.
[ ] www.hivevan.org.
[ ] chooseindependent.org.
[ ] firefox10.org.
[ ] fantheflames.org.
[ ] adhoctribution-db.mofoprod.net.
[ ] aggredash-db.mofoprod.net.
[x] angelasreport.mofoprod.net.
[ ] badgekit-api-db.mofoprod.net.
[ ] badgekit-api-mozilla-db.mofoprod.net.
[ ] badgekit-db.mofoprod.net.
[ ] badgekit-mozilla-db.mofoprod.net.
[ ] collusion-db.mofoprod.net.
[ ] csa-discourse-cache.mofoprod.net.
[ ] csa-ftf-cache.mofoprod.net.
[ ] csa-guides-cache.mofoprod.net.
[ ] csa-msql-db.mofoprod.net.
[ ] csa-wordpress.mofoprod.net.
[ ] csol-db.mofoprod.net.
[ ] csplogger-db.mofoprod.net.
[ ] discourse-advo-cache.mofoprod.net.
[x] elk.mofoprod.net.
[ ] etherpadlite-db.mofoprod.net.
[ ] events-api-db.mofoprod.net.
[ ] gauntletdash-db.mofoprod.net.
[ ] gitribution-db.mofoprod.net.
[ ] goggles-db.mofoprod.net.
[ ] graphite.mofoprod.net.
[x] grimoire-db.mofoprod.net.
[x] huboard.mofoprod.net.
[ ] integration-db.mofoprod.net.
[ ] jenkins.mofoprod.net.
[ ] lightbeam-db.mofoprod.net.
[x] log.mofoprod.net.
[x] loggins-db.mofoprod.net.
[ ] loggins-es.mofoprod.net.
[x] loggins2.mofoprod.net.
[ ] logship.mofoprod.net.
[ ] makeapi-db.mofoprod.net.
[ ] makeapi-es.mofoprod.net.
[ ] makermind-es.mofoprod.net.
[x] makermind.mofoprod.net.
[ ] metrics-db.mofoprod.net.
[ ] mofointegration-db.mofoprod.net.
[ ] mozillawebmakerproxy.mofoprod.net.
[ ] node-hubble-cache.mofoprod.net.
[ ] openbadger-csol-db.mofoprod.net.
[ ] openbadger-production-cache-c.mofoprod.net.
[ ] openbadger-wm-cache.mofoprod.net.
[ ] openbadger-wm-db.mofoprod.net.
[ ] openbadges-cache.mofoprod.net.
[ ] openbadges-db.mofoprod.net.
[x] openbadges-read1.mofoprod.net.
[ ] openbadgesorg-cache-c.mofoprod.net.
[ ] openbadgesorg-db.mofoprod.net.
[ ] opsview-db.mofoprod.net.
[ ] opsview.mofoprod.net.
[ ] pgshared-db.mofoprod.net.
[ ] phabricator-db.mofoprod.net.
[x] s.mofoprod.net.
[x] scienceblog.mofoprod.net.
[x] scienceforum.mofoprod.net.
[x] securegeppetto.mofoprod.net.
[x] securejenkins.mofoprod.net.
[ ] source-cache.mofoprod.net.
[ ] source-db.mofoprod.net.
[ ] source-es.mofoprod.net.
[ ] statsd.mofoprod.net.
[ ] thimble-db.mofoprod.net.
[ ] togetherjs.mofoprod.net.
[x] transibution-db.mofoprod.net.
[ ] valhalla.mofoprod.net.
[ ] webmakerorg-db.mofoprod.net.
[ ] wmbadges-db.mofoprod.net.
[ ] wmlogin-cache.mofoprod.net.
[ ] wmlogin-db.mofoprod.net.
[ ] wmlogin-db2.mofoprod.net.
[x] wmprofile-service-db.mofoprod.net.
[ ] wpshared-db.mofoprod.net.
[ ] xipd.mofoprod.net.
[ ] aestimia-csol.mofostaging.net.
[ ] apps.mofostaging.net.
[ ] badgekit-api-db.mofostaging.net.
[ ] badgekit-api-mozilla-db.mofostaging.net.
[ ] badgekit-db.mofostaging.net.
[ ] badgekit-mozilla-db.mofostaging.net.
[ ] butter-cache.mofostaging.net.
[ ] butter-db.mofostaging.net.
[ ] collusion-db.mofostaging.net.
[ ] collusiondb-cache.mofostaging.net.
[ ] community.mofostaging.net.
[ ] csa-discourse-cache.mofostaging.net.
[x] csol-db.mofostaging.net.
[ ] csplogger-db.mofostaging.net.
[x] davida.mofostaging.net.
[x] dev-db.mofostaging.net.
[x] es.mofostaging.net.
[ ] events-api-db.mofostaging.net.
[x] fifiback-cache.mofostaging.net.
[x] fifiback.mofostaging.net.
[ ] firehose.mofostaging.net.
[x] fundraising2.mofostaging.net.
[ ] gallery.mofostaging.net.
[x] genopen.mofostaging.net.
[ ] goggles-db.mofostaging.net.
[ ] greece.mofostaging.net.
[ ] herokudemo-db.mofostaging.net.
[ ] hive.mofostaging.net.
[x] integration-db.mofostaging.net.
[x] ireland.mofostaging.net.
[x] macedonia.mofostaging.net.
[ ] makeapi-db.mofostaging.net.
[ ] makeapi-es.mofostaging.net.
[ ] makeapi-es1.mofostaging.net.
[ ] makedrive-cache.mofostaging.net.
[ ] makedrive-db.mofostaging.net.
[ ] metrics-db.mofostaging.net.
[ ] mongo-db.mofostaging.net.
[ ] mozillafestival.mofostaging.net.
[ ] nepal.mofostaging.net.
[ ] node-hubble-cache.mofostaging.net.
[ ] openbadges-cache.mofostaging.net.
[ ] openbadges-db.mofostaging.net.
[ ] openbadgesorg-db.mofostaging.net.
[x] parapara-db.mofostaging.net.
[x] pgshared-db.mofostaging.net.
[ ] scienceblog.mofostaging.net.
[ ] shared-db.mofostaging.net.
[ ] source-db.mofostaging.net.
[ ] source.mofostaging.net.
[ ] thimble-db.mofostaging.net.
[x] thimbleheroku.mofostaging.net.
[x] toronto.mofostaging.net.
[ ] towtruckhub.mofostaging.net.
[ ] training.mofostaging.net.
[x] ttw-discourse.mofostaging.net.
[ ] webcam.mofostaging.net.
[ ] webmaker-app.mofostaging.net.
[ ] webmakerorg-db.mofostaging.net.
[x] webmakerorg-demo-db.mofostaging.net.
[ ] welcome.mofostaging.net.
[ ] wiki.mofostaging.net.
[ ] wmlogin-cache.mofostaging.net.
[ ] wmlogin-db.mofostaging.net.
[ ] wmprofile-service-db.mofostaging.net.
[x] prop.jdotp.org.
[x] wordpress-db.jdotp.org.
[ ] assess.chicagosummeroflearning.org.
[ ] badge.chicagosummeroflearning.org.
[ ] www.chicagosummeroflearning.org.
[ ] assess.openbadges.org.
[ ] backpack.openbadges.org.
[ ] badger.openbadges.org.
[ ] badges.openbadges.org.
[ ] bakery.openbadges.org.
[ ] beta.openbadges.org.
[ ] community.openbadges.org.
[ ] directory.openbadges.org.
[ ] discover.openbadges.org.
[ ] specification.openbadges.org.
[ ] standard.openbadges.org.
[x] collusion-db.mofodev.net.
[x] csv2f2iuhb2vwqhca6i7e63fzvkibkdx._domainkey.webmaker.org.
[x] nl2juxqqjl6gjhbhicgbhuiz66ppfsaf._domainkey.webmaker.org.
[x] rcspzuxxs4pheuw4yswbu7yhatczh3bi._domainkey.webmaker.org.
[ ] apps.webmaker.org.
[ ] build.webmaker.org.
[ ] live.webmaker.org.
[x] oldparty.webmaker.org.
[ ] phabricator.webmaker.org.
[ ] planet.webmaker.org.
[ ] sendto.webmaker.org.
[ ] welcome.webmaker.org.
[x] \052.thegifreport.com.
[ ] www.mozillawebmaker.info.
[x] 9319ac9a-8ccc-4712-aad7-195f116aac6d.stopwatching.us.
[x] optin.stopwatching.us.
[x] www.stopwatching.us.
[ ] blog.workopen.org.
[x] megazord.workopen.org.
[ ] assets.opennews.org.
[ ] planet.opennews.org.
[ ] forum.mozillascience.com.
[ ] www.mozillascience.org.
[ ] \052.hackasaurus.org.
[ ] badges.mozillafestival.org.
[ ] schedule.mozillafestival.org.
[ ] sessions.mozillafestival.org.
[ ] maker.mozillapopcorn.org.
[ ] static.mozillapopcorn.org.
[ ] tedglobal.mozillapopcorn.org.
[ ] www.mozillapopcorn.org.
[ ] planet.mozillaopennews.org.
[ ] planet.drumbeat.org.
[ ] assets.mozillawebmakerproxy.net.
[ ] connect.webmakerprototypes.org.
[ ] gallery.webmakerprototypes.org.
[ ] kitbuilder.webmakerprototypes.org.
[ ] planet.webmakerprototypes.org.
[ ] training.webmakerprototypes.org.
[ ] weblitmapper.webmakerprototypes.org.
[x] oldhub.togetherjs.com.
[x] www.togetherjs.com.
[ ] www.devopsagainsthumanity.net.
[ ] planet.hivelearningnetworks.org.
[ ] 6x44acwknd7bf5kkbtmmpx5c72qdqs3i._domainkey.mozilla-community.org.
[ ] in5gnt3st4ljlynoimkaq5zic7idzfhf._domainkey.mozilla-community.org.
[ ] urgsseaev4fzp7ro3k5w6fso7u27fquu._domainkey.mozilla-community.org.
[ ] discourse.cdn.mozilla-community.org.
[x] greece.cdn.mozilla-community.org.
[x] test.cdn.mozilla-community.org.
[ ] health.mozilla-community.org.
[ ] mail.mozilla-community.org.
[ ] plesk.mozilla-community.org.
[ ] plesk2.mozilla-community.org.
[ ] relmandash.mozilla-community.org.
[ ] ovh.shared.mozilla-community.org.
[ ] www.mozilla-community.org.
[x] dev.flathead.mozillalabs.com.
[x] \052.appalot.me.
[ ] blog.communitysysadmins.org.
[ ] signup.communitysysadmins.org.
[ ] www.communitysysadmins.org.
[ ] www.badg.us.
[ ] www.chooseindependent.org.
[ ] www.firefox10.org.
[x] pdsa2g5xijdnc2ih5gbrmvjrf5dz4zjj._domainkey.fantheflames.org.
[x] s7jig6nctfjxx4jdo5nsjofvstz7mojf._domainkey.fantheflames.org.
[x] v6djva3lexqdqdtaq6w6dxuqjlfy57ey._domainkey.fantheflames.org.
[x] cdn.fantheflames.org.
[x] www.fantheflames.org

ghost commented 9 years ago

How about working off of this?

https://docs.google.com/spreadsheets/d/1QBkvhYT9twBB89246yFyyg6RVHLbsTdPvpSutbs5nsY/edit#gid=0

Also, since I'm assuming this is from Route53, I'm thinking if we get an authoritative list I can clean out all the unnecessary record sets.

jdotpz commented 9 years ago

Awesome, works from me. And yeah, that dump is all of our A and CNAME records (aliases included). Nice call, we can also tag stuff in that spreadsheet with "not needed" or some such to note it should be pulled out of R53. Thanks @chefscott !

ghost commented 9 years ago

Hey let me know how you want me to help on this -- it's difficult since I don't know which are valid domains and which are not. For instance, I know devopsagainsthumanity.net is valid but it doesn't resolve anywhere, but hackasau.usorg doesn't resolve but it seems like an invalid URL (maybe just imported incorrectly?).

I also added a new symbol for things you want deleted.

jbuck commented 9 years ago

This will be replaced with Pingdom monitoring in #30 . But that list is very helpful to use for sites to add to Pingdom!

MozillaFoundation / mofo-devops

Audit our monitoring #19