☁️ Cloudflare - Add Redirects for /archive routes

amankumarrr commented 3 months ago

Description: We need to add redirects for all /archive from v1 to our new routes on Azure storage containers.

### Tasks
- [ ] https://github.com/SSWConsulting/SSW.Website/issues/2714
- [x] Add /archive redirects
- [x] Add redirects for /newsletters content from v1
- [x] Add redirects for /ssw
- [x] Add same routing on ssw.com.au

andrewwaltosssw commented 3 months ago

For newsletter asset redirects please see link at https://github.com/SSWConsulting/SSW.Website/issues/2617#issuecomment-2136275918

amankumarrr commented 2 months ago

@wicksipedia @SSWConsulting/ssw-website-maintainers

Redirects Status for tfs365.com (Staging site) Domain:

All redirects (total of 1,026) have been tested and added successfully for the following routes:

✅ legacy-redirects ✅ case-studies ✅ consulting ✅ events ✅ static ✅ archive ✅ Website-redirects ✅ Newsletters content

Remaining Redirects to be Implemented:

For the following domains, redirects are still pending:

❌ dory-redirects.json: Redirects for the sswdory.com domain are yet to be set up. ❌ prod-dotcom-redirects.json: Redirects for the ssw.com domain are not yet implemented. ❌ staging-dotcom-redirects.json: Redirects for the staging.ssw.com domain are pending.

Here is the repo - https://github.com/SSWConsulting/SSW.Cloudflare

Download this to view the result - https://github.com/SSWConsulting/SSW.Cloudflare/blob/main/test-scripts/url_redirect_report.html

amankumarrr commented 2 months ago

@wicksipedia @chrisschultzssw @bradystroud @andrewwaltosssw @Calinator444

As we are unblocked now to move forward with this task, we need to add worker routes for ssw.com.au in order to soft-turn off the VM.

Note: If we implement the route, our website v1, will not be accessible at all from /ssw.

Tasks:

[ ] Update the App Insights instrumentation key to Prod (it's currently pointing to DEV).
[ ] Update the 404 page, which currently just says "page not found."

Calinator444 commented 2 months ago

@amankumarrr I ran the URL redirect status report and I noticed a number of the redirects for /archive were returning 403s or 421s (806 pages affected)

Is this expected behaviour?

Figure: 400 level responses for archive pages

Calinator444 commented 2 months ago

Nevermind, just tried rerunning the build and deploy workflow https://github.com/SSWConsulting/SSW.Cloudflare/actions/workflows/main.yml

amankumarrr commented 2 months ago

In the past 24 hours, these are stats for Cloudflare on route /ssw/*

Figure: Stats from Cloudflare

amankumarrr commented 2 months ago

@wicksipedia @jeoffreyfischer

We can add redirects logs in App Insight.

Figure: Logs in App Insight for redirects from worker

amankumarrr commented 2 months ago

@jeoffreyfischer

Hey @wicksipedia ,

Could you please review the plan for implementing redirects through the Worker?

In the attached screenshot:

Black lines represent the current flow.
Red lines indicate changes planned for Phase 1.
Purple lines show updates for Phase 2.

Figure: Plan for the redirects and moving away from Front Door

jeoffreyfischer commented 2 months ago

As per my conversation with @wicksipedia, I will improve the drawings and discuss the plan with him and @amankumarrr on Friday.

jeoffreyfischer commented 2 months ago

As per our conversation with @wicksipedia and @amankumarrr, the Cloudflare migration will be performed in 2 steps:

Step 1: Redirects are stored in Workers and actioned by Front Door (see figure)
Step 2: Redirect are stored and managed in Workers (see figure)

Before pushing Step 1 to Prod, @amankumarrr will update the README and make a Done Video about how to use Cloudflare. This is to ensure that anyone can use it.

About Step 2, @amankumarrr and @jeoffreyfischer will investigate several FREE options and then discuss with @wicksipedia about which one we will go with.

Screenshots

Figure: Step 0 - Current workflow

Figure: Step 1 - Transition workflow

Figure: Step 2 - Final workflow

jeoffreyfischer commented 1 month ago

As per my conversation with @wicksipedia and @amankumarrr, I have investigated some CDN options:

CloudFlare Workers (Recommended)

Link to doc

https://developers.cloudflare.com/workers/platform/limits/

Price

FREE (paid options available)

Tech details

Number of env variables: 50 per Worker
Worker size: 1 MB
Request body limit: 100 MB
Number of requests per Worker: 100,000 per day - 1,000 per min
Worker memory: 128 MB
Calls (get(), put(), etc.) per request: 50
Number of routed zones per Worker: 1,000
Log size: 128 kB

Google Cloud CDN

Link to doc

https://cloud.google.com/cdn/pricing

Price

Pay by usage

Cache data transfer out: $0.10 per GiB
Cache fill: $0.02 per GiB
HTTP/HTTPS cache lookup requests: $0.0075 per 10,000 requests
Tech details

Not Found

Fastly

Link to doc

https://www.fastly.com/pricing/

Price

Lowest Tier: 1500$/month

Tech details

Number of requests: 100,000 per month
Number of TLS domains: 20

jeoffreyfischer commented 1 month ago

Below are the total monthly hits in June 2024 on:

SSW Website: 9,541,592
.../rules: 2,239,154
.../people: 374,925
.../archive: 42,018

Screenshot Figure: Monthly hits count on SSW Website, rules, people and archive

jeoffreyfischer commented 1 month ago

Summary

As per my conversation with @wicksipedia , I have investigated the limits for a $5/month plan with CloudFlare. ✅ All lights are green. ⚠️ The only slight warning is about the Routes (see below).

Requests

Number: 10 million included per month + $0.30 per additional million Note: In June 2024, SSW Website had 9.5 million requests (see previous comment)
Number of calls per request: 1000

Requests	Price	Likelihood on SSW Website
10 million	$5/month	High
20 million	$8/month	Medium
50 million	$17/month	Low

Table: Projection (if we ever exceed the 10 million requests cap)

Workers

Number: 500
Size: 100 MB

Routes

Number of routes per zone: 1000 ℹ️ Note: a "zone" refers to a specific domain or subdomain that has been added to our Cloudflare account for management. ⚠️ Warning: This can be an issue given ssw.rules has around 3500 rules. ✅ Solution: We can separate ssw.rules into several workers.

Environment variables

Number: 128/Worker
Size: 5 KB

Useful links

Link to Pricing doc: https://developers.cloudflare.com/workers/platform/pricing
Link to Limits doc: https://developers.cloudflare.com/workers/platform/limits

Calinator444 commented 1 month ago

@wicksipedia

We exceeded the request limit when the Cloudflare worker went live on production.

Had a look through the pricing doc. If we were to switch to the standard pricing plan our limit would increase to 10 million requests per month (or approximately 333,0000 requests per day over the previous 100,000)

I've included the pricing model below

jeoffreyfischer commented 1 month ago

In the previous comment, I've pointed out that the current number of requests on SSW Website is about 9.5 millions/month. So I recommend going for the 8$/month plan to allow for a maximum requests number of 20 million/month.

Figure: SSW Monthly requests + Cloudflare pricing plan (copied from previous comment)

Calinator444 commented 1 month ago

We can't push this into prod until the purchase please gets approved.

See email - Purchase Please - Request for Cloudflare workers standard (⚠️ High priority)

Calinator444 commented 1 month ago

@wicksipedia

The purchase please has been approved and the Cloudflare worker is now in production. (You can test the redirects at ssw.com.au/archive in the left hand side)

@chrisschultzssw said he was going to add a billing alerts just in case we exceed the estimate of $146.45 per year.

I also added a staging workflow for testing redirects before pushing to prod. The done video for that is below (same as the done video I did in the PBI this week)

https://youtu.be/wdZql8oAawA

chrisschultzssw commented 1 month ago

@chrisschultzssw said he was going to add a billing alerts just in case we exceed the estimate of $146.45 per year.

Done.

SSWConsulting / SSW.Website