[RESEARCH] Research rate limiting/blocking based on referrer in AWS WAF

[adambuttrick]

Background

We should investigate additional traffic control measures based on the HTTP referrer header. Specifically, we want to block or limit requests to routes such as password reset and account login that do not come from our own domain.

Objective

Research options for using WAF to block or limit traffic based on the referrer header.

Tasks

• [x] Investigate how AWS WAF handles the referrer header.
• [ ] Investigate whether we can create a rule in AWS WAF that inspects the referrer header.

If we can manage traffic in WAF based on the referrer:

• [ ] Define conditions for blocking or allowing traffic based on the referrer.
• [ ] Ensure that the rule is flexible enough to handle all EZID application referrer patterns.
• [ ] Test the rule to confirm it works as expected.
• [ ] Document the entire process, including:
  • Steps to create and configure the rule.
  • Details on modifying the rule for different referrer patterns.
  • Testing procedures and results.

References

• AWS WAF Documentation
• AWS WAF Rule Creation
• AWS WAF Rule Testing

[adambuttrick]

I discussed this issue briefly with Liz Krznarich from DataCite. She indicated that the AWS load balancers typically do not include the referrer information from the requests such that this would be accessible to the WAF for rate limiting or blocking. We should verify. Her recommendation was that we consider whether any routes that are only used by the application could be moved inside a VPC, such that they're not surfaced or publicly accessible to anything other than the application itself. Not to confirm whether we have any of these in EZID.

[jsjiang]

It looks like our WAF logs the Referer header.

• A non-legitimate request to the login page with Referer header:

httpRequest.uri: /login

{ "name": "Referer", "value": "https://44.227.124.56/login" }

httpRequest.headers 
{   "name": "Content-Length",   "value": "47" }, {   "name": "Cache-Control",   "value": "no-cache" }, {   "name": "User-Agent",   "value": "Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.19) Gecko/20081216 Fedora/2.0.0.19-1.fc8 Firefox/2.0.0.19 pango-text'))) AND 7765=CONVERT(INT,(SELECT CHAR(113)+CHAR(122)+CHAR(122)+CHAR(118)+CHAR(113)+(SELECT (CASE WHEN (7765=7765) THEN CHAR(49) ELSE CHAR(48) END))+CHAR(113)+CHAR(118)+CHAR(113)+CHAR(112)+CHAR(113))) AND ((('wweT'='wweT" }, {   "name": "Referer",   "value": "https://44.227.124.56/login" }, {   "name": "Host",   "value": "44.227.124.56" }, {   "name": "Accept",   "value": "*/*" }, {   "name": "Accept-Encoding",   "value": "gzip,deflate" }, {   "name": "Content-Type",   "value": "application/x-www-form-urlencoded; charset=utf-8" }, {   "name": "Connection",   "value": "close" }

• A legitimate request to the /learn/ page came from the /search page:

  {
  "name": "referer",
  "value": "https://ezid.cdlib.org/search"
  },

• A legitimate request without referer in the request header:

  
  httpRequest.uri:  /ark:/81235/d87s7j600

httpRequest.headers { "name": "Host", "value": "ezid.cdlib.org" }, { "name": "User-Agent", "value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/125.0.2535.92" }, { "name": "Accept", "value": "/" }, { "name": "Accept-Encoding", "value": "identity" }, { "name": "Connection", "value": "keep-alive" }, { "name": "Accept-Language", "value": "en-US;q=0.9, en;q=0.8" }

* A non-legitimate request without `referer` in the request header:

httpRequest.uri: /

httpRequest.headers | { "name": "Host", "value": "54.69.149.219" }, { "name": "User-Agent", "value": "Opera/9.80 (X11; FreeBSD 8.1-RELEASE i386; Edition Next) Presto/2.12.388 Version/12.10" }, { "name": "Accept-Charset", "value": "utf-8" }, { "name": "Accept-Encoding", "value": "gzip" }, { "name": "Connection", "value": "close" }