Dask.org is down

[quasiben]

Cloudflare is reporting an SSL failure for dask.org

We are actively investigating

[quasiben]

Dask.org is up now for me. @jacobtomlinson and I disabled and cleared the cache on cloudflare which seems to have resolved the 525 issue. If there are any outages today please reopen

[jrbourbeau]

Thanks for handling this @quasiben @jacobtomlinson!

[mrocklin]

Indeed, many thanks

On Tue, Sep 6, 2022, 8:57 AM James Bourbeau @.***> wrote:

Thanks for handling this @quasiben https://github.com/quasiben @jacobtomlinson https://github.com/jacobtomlinson!

— Reply to this email directly, view it on GitHub https://github.com/dask/community/issues/274#issuecomment-1238345553, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTGYONKKLJGQFHP63UTV45SX5ANCNFSM6AAAAAAQF4EIKI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

[jacobtomlinson]

Here's a quick postmortem of what happened. We plan to meet later this week to go into more depth here.

Context

• When we launched the rebrand of the dask.org website we switched over to Webflow for hosting.
• Our domain name is managed by NumFOCUS who gives maintainers access via CloudFlare which allows us to set up our DNS rules and also leverage some caching, page rules and DDOS protection.
• When we switched from Readthedocs to Webflow we updated the dask.org domain to point to proxy-ssl.webflow.com as instructed by the Webflow setup guide.
• We also enabled the caching proxy in Cloudflare at this time to limit how much bandwidth Webflow would charge us for.
• The WebFlow SSL proxy uses LetsEncrypt to generate certificates. A certificate was generated between the time we update the DNS and enabled the caching.

Incident

• A user reported to us via Slack that dask.org was down.
• It appeared that Cloudflare was unable to access Webflow. Investigation began.
• It turned out that the certificate that Webflow was using had expired and because Cloudflare caching was enabled Webflow was unable to generate a new certificate resulting in Webflow being unable to securely send traffic.
• To remediate we disabled and purged the Cloudflare cache which allowed Webflow to generate a new certificate and begin serving requests again.

Follow-up tasks

• Maintainers should be notified of outages like this automatically rather than relying on user feedback. We will investigate some monitoring.
• It appears we could change the DNS rules to point to proxy.webflow.com instead and enable the caching again which would give us the bandwidth savings and DDOS protection that we had previously. The consequence of this is that traffic between Cloudflare and Webflow would be sent in plaintext.
• Our investigation was slowed by maintainers not having quick admin access to the Webflow account. We will look at transferring those rights to one of us.