{{ nocache }} file disk storage optimization

dadaxr commented 4 days ago

Bug description

I may be wrong but, I figured that using nocache tag, even with the select parameter set to restrict which variable to hydrate... the whole template context seems to be stored in the disk, which impacts performance and available disk space...

Would it be possible to also take the select parameter into account, so we can avoid caching unnecessary data. (In my project, the static-url-cache folder can end up containing more than 100 go of files ... )

Kind of related to #9124 and #10281

How to reproduce

For instance, if you have a template like this :

{{ allProducts = {collection:products} }}
{{ firstProduct =  allProducts[0] }}

{{ nocache select="firstProduct" }}
  {{# do something which require the first product and need to not be cached like checking it's stock for instance #}}
  {{ app:checkProductStock :product="firstProduct" }} 
{{ /nocache }}

{{ allProducts }}
  {{# do something with cached products #}}
{{ /allProducts }}

In that case, a file in the statamic/static-url-cache will be created for the {{ nocache }} region, and will contain both "allProducts" and "firstProduct" serialized, even if "firstProduct" is needed ... It seems the "select" parameter optimization is only used when re-hydrateding the context from cache, but not before storing the context...

Logs

No response

Environment

Environment
Laravel Version: 10.48.13
PHP Version: 8.2.23
Composer Version: 1.10.27
Environment: prod
Debug Mode: ENABLED
Maintenance Mode: OFF

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: file
Database: mysql
Logs: stack / daily
Mail: smtp
Queue: sync
Session: file

Statamic
Addons: 4
Sites: 1
Stache Watcher: Disabled
Static Caching: half
Version: 5.38.0 PRO

Installation

Fresh statamic/statamic site via CLI

Additional details

No response

duncanmcclean commented 3 days ago

That's strange - it's working fine for me on a fresh install. firstProduct is saved, but allProducts isn't. 🤔

Can you try deleting the storage/statamic/static-urls-cache directory and checking again? Maybe there's some cached items left over from before you added the select?

dadaxr commented 3 days ago

OK, my bad I've oversimplified my example... and found out what the problem is.

In my above example, I only used a main template but the problem is when using nested templates (partial).

To reproduce the problem, we need 3 files :

main template :

{{ allExperts = { collection:experts } }}
{{ partial:templates/experts-list :experts="allExperts" }}

partial "experts-list" :

---
experts:
---
<ul>
    {{ view.experts scope="anExpert" }}
        <li> {{ partial:templates/expert :expert="anExpert" }}</li>
    {{ /view.experts }}
</ul>

partial "expert" :

---
expert:
---
{{ nocache select="view" }}
    expert.title : {{ view.expert.title }} 
{{ /nocache }}

Given a list of 50 experts, I should get ~50 files in static-urls-cache folder with each file containing the serialization of the view var ... which I guess would be the serialized version of an expert object... so ~ 50 "small files" (given an expert object does not contains much information) BUT ... it's not the case and 2 problems pile up here ...

First problem, related to #10835 and #10703 , the view var from the "expert" partial contains also data from the view var from the "experts-list" partial ie ... the whole "experts" array .
So we now have a serialized view var which contains the current expert + the 50 ones ... so theorically 51 objects by cached file ... (so 50 * 51 = 2250 serialized experts instead of 50). If fixing this is hard because of retro compatibiliy, maybe adding a new "self" variable, referencing only view vars from the current template could be helpfull ? ( I've added that suggestion in the #10703 opened issue)

Second "problem" (even if it's not a real bug to me but still annoying, and would be happy to have a recommanded way of handling this). In the experts-list template, we iterate over the experts collection... :

{{ view.experts scope="anExpert" }}
        <li> {{ partial:templates/expert :expert="anExpert" }}</li>
{{ /view.experts }}

When doing this, each "anExpert" variable is "augmented" with extra information like "last" / "first" but also : "prev" and "last" ... which directly targets other experts ... themselves, having their prev and next var targeting other experts and so on... That means that when we serialize the view var in the "expert" template, we serialized not 1 expert object but many many more... Actually I have files which weight ~650ko just to serialize "one expert (and related prev / next vars)" So ~ 50 files * 650 ko = ~30mo of cache space needed. That's a lot ! Maybe adding a "warning" in the {{ nocache }} tag doc about "avoiding select whole entry objects, particularly in loop, but instead being as specific as possible" could be helpful ( I saturated a 500 go file disk in 2 days because of this )

Soooo, what I think to do here to walk around this 2nd problem, is to not select "whole" object in the nocache select tag but try pick each value explicitly... for instance doing something like this :

---
expert:
---
{{ _expertTitle = view.expert.title }}
{{ nocache select="_expertTitle" }}
    expert.title : {{ _expertTitle }}
{{ /nocache }}

It's a bit cumbersome when a lot variable are needed in the nocahe tags, but it works fine. bellow an example with many variables needed in the nocache tag.

---
expert:
---
{{ _nocacheData = [
    'expertTitle" => view.expert.title,
    'expertRate" => view.expert.rate,
    ....
}}
{{ nocache select="_nocacheData" }}
    expert.title : {{  _nocacheData.expertTitle }}
    expert.rate : {{  _nocacheData.expertRate }}
    ...
{{ /nocache }}

So to sum up, maybe we can have your thoughts on that, recommended best practices, and maybe close this issue if my work arround seems good enough to be documented in the nocache tag ? What do you think ?

statamic / cms