mydnshost / mydnshost-infra

mydnshost-infra

Other

1 stars 1 forks source link

Improve DNS Monitoring #1

Closed ShaneMcC closed 7 years ago

ShaneMcC commented 7 years ago

We should be dynamically changing a record under monitor.mydnshost.co.uk and updating the statuscake tests to confirm that changes are pushing properly to all the servers.

ShaneMcC commented 7 years ago

https://www.statuscake.com/api/

ShaneMcC commented 7 years ago

Prep Work

Created 3 records:

jahce2th.monitor.mydnshost.co.uk IN A 127.0.0.100
azei7rai.monitor.mydnshost.co.uk IN A 127.0.0.101
active.monitor.mydnshost.co.uk IN TXT jahce2th

Still to do

Need to create cron that does something like:

Pull all 3 records from the API
Update the non-active record to have a new random IP within 127.0.0.0/8
Update "active" TXT record to point at the currently non-active record (This is our state-tracking)
Sleep for a minute to let slaves update
Update all the "DNS" tests to point at the new active record with the new IP.

This can run every hour or so.

And we can update statuscake with something like:

curl -H "API: [API Key]" -H "Username: [Username]"  -d "TestID=2448201&DNSIP=127.0.0.100&WebsiteURL=jahce2th.monitor.mydnshost.co.uk" -X PUT https://app.statuscake.com/API/Tests/Update

ShaneMcC commented 7 years ago

This has a side-effect that it also tests that the API/DB Server is actually working.

ShaneMcC commented 7 years ago

(The reason for having 2 records and rolling around the "active" TXT record is so that we don't have a race condition between us updating the record and updating statuscake)

ShaneMcC commented 7 years ago

Done in a selection of commits:

https://github.com/ShaneMcC/mydnshost-infra/commit/f2fde70145766a2ab3697518c773e7d4aadc9128 https://github.com/ShaneMcC/mydnshost-infra/commit/b613bd13614cae5bd4da85bfa0760b8408c8a51a https://github.com/ShaneMcC/mydnshost-infra/commit/b3190638e6d0df29c5d5128ea983ce00704cc2d8 https://github.com/ShaneMcC/mydnshost-infra/commit/e38cd3b4e3d6f2adc9e5ef57e4d1f5ce75793767 https://github.com/ShaneMcC/mydnshost-infra/commit/16ca6063983fdf295b092bac83e4c5b746a11d6f

This is currently running against a series of "test" Tests on StatusCake (2450273,2450274,2448201) and updating a rotation of 6 rrnames (azei7rai,jahce2th,aiquo7ba,nuvei1lo,oogewei5,biejiam3).

6 names is probably a bit overkill, I think 2 should work but figured 6 means that we can do debugging (manual runs) without being likely to accidentally cause an error state.

ShaneMcC commented 7 years ago

Test-Run can be monitored at https://uptime.statuscake.com/?TestID=oLT7ncOZsO

ShaneMcC commented 7 years ago

Test was successful overnight, switched to production monitoring now.