Open janreges opened 1 year ago
Thanks for the getRootFolderByVolumeId()
suggestion! I’ve made an optimization to that for Craft 4.5 via 8305fbe36e63a620b5c5a870791536376b4d34f3.
Would you mind sharing your config/project/
folder and Composer files with us? With those in place we could recreate your setup and run it through Blackfire to look for other optimization opportunities.
The Neo and SuperTable plugins, and perhaps even the Craft CMS level fields, lack some form of internal instance cache to ensure that even if a field is used within a form dozens or hundreds of times within a single run of a PHP script (e.g. in Neo/SuperTable blocks), that they are only initialized once. That is, even if an instance of a field/element is instantiated hundreds of times in a run of a single instance of a PHP script, that initialization and retrieval of information from the DB or filesystem is done only once.
Worth noting that this is generally already the case.
Unfortunately, the Neo plugin is not tailored to higher tens or hundreds of block types. Editing such a Neo field (prescription of block types) freezes JavaScript in the browser for tens of seconds after loading.
For the Neo or SuperTable plugins, a form of "lazy loading" would be useful in some critical places, both at the block type configuration level (in the Settings section) and within the page content forms. Even though the user only uses 0 to 10% of the content types within the management of these blocks or content management on a given page, a complete initialization of all of them is performed with each request.
Both of those field types are heavily inspired by Matrix. We are working to improve Matrix scalability in Craft 5, and would expect that those plugins will end up following our lead to some extent.
The Craft CMS as a whole lacks a sophisticated cache that would be built on, for example, tagging the cache and then also have selective cache invalidation based on the tags.
Yii does in fact have robust cache invalidation, including tags. Craft uses tag dependencies for various things, such as front-end {% cache %}
tags and GraphQL queries.
Because for most projects that use Craft CMS, the CMS administration part (Settings, fields, sections, etc.) is completely disabled on production projects (CMS configuration happens more on DEV environments and is versioned). Therefore, it is completely unnecessary to have a series of DB queries, event/log throws related to loading something that is practically unchangeable in the production CMS, while doing normal content management on most forms in the CMS. The invalidation of this cache could occur by default in cache-flush commands within deployments. However, for this cache to be truly meaningful, it is not just about caching DB queries, but rather entire serialized instances of some objects that must be created over and over again for most requests in the CMS during normal operation and take tens or hundreds of milliseconds.
Caching entire objects can be error-prone, so we’ve generally avoided it, but you may be right. It’s worth looking into.
@brandonkelly - thank you very much for your reply and for quick optimization in getrootFolderByVolumeId()
.
I have already sent you an e-mail with composer.json
and the complete contents of the project/config
folder.
Considering that you have a great insight into the internal architecture of Craft CMS, I believe that you will be able to quickly find a couple of essential places to optimize.
In my opinion, the biggest quick-win can be the implementation of "field instance cache" for a place in the code with initialization/factory of fields, which are called very intensively and repeatedly even from Matrix/Neo/SuperTable.
Thank you also for referring to better options for working with the cache inside Yii. It's great that tagging support is also there and we will use it in our applications/websites. Due to scaling, it would be great if the tagging cache and ideally also the cache of entire PHP class instances were also used within the CMS. For really large projects, we use replicated databases and containerization, where the database does not run locally, so every saved DB query or CPU-cycle within PHP can represent a significant saving. Due to scaling, replicated databases or Redis clusters are often used, which also do not run locally with PHP, but on an another server. Therefore, using Redis only for DB query cache may not bring significant improvement (network latency is still there). This is the reason why, in some cases, serialized instances of ready-made classes can be stored in the cache, which had to perform a number of DB queries and other CPU/memory intensive operations.
If you import our project configuration into the CMS and see how slow the "Content editor" type page management forms are, I believe you will understand. I believe that, as an architect, you will realize very quickly that the high tens of percent of this overhead, which slows down most requests, can be efficiently cached and will not necessarily be performed for all pages/forms within the CMS movement.
Thank you very much for putting your efforts to this optimization.
I was asked to take a look at a site exhibiting similar issues this week. The developers have implemented a Neo content builder field with around 40 block types and performance in the control panel is very poor. They’ve implemented sufficient caching on the frontend that it’s fine from an end user perspective, but content managers are understandably frustrated with 5-15s load times when editing entries.
I’ve tried profiling some requests to identify obvious bottlenecks/repetition but there doesn’t seem to be any single thing that’s holding things up – more general inefficiencies.
In our case, a typical entry edit page load results in around 1k queries (of which nearly half are duplicates) and nearly 15k events logged.
If an additional test case would be useful please let me know and I can share a composer.json and project config as well.
One detail that I’m a little vague on – is Craft keeping an internal query cache per-request and returning results from that when duplicate queries are received? I assume not?
I’d also love to see support for extended caching at deploy time, as allowAdminChanges
will always be disabled in prod and staging envs, so you’d hope that a lot of the queries relating to field structure could be compiled once and then safely eliminated from individual entry edit requests.
@domstubbs Not sure why so much time is being spent in Guzzle. CP requests should generally not be making any HTTP requests directly. So it’s worth looking into why that’s happening.
The excessive time spent in Composer\Autoload\IncludeFile
seems to indicate that the classmap isn’t optimized. You can fix that by running composer dump-autoload -o
.
Beyond that, yes there’s definitely room for improvement on the caching front as @janreges pointed out, as well as generally finding ways to reduce the per-screen content complexity – which is one of our main areas of focus for Craft 5.
Thanks Brandon. I’ve just been debugging in a dev environment, hence the non-optimised classmap. I did start to dig into the Guzzle requests but once I saw they were tied to AWS Assets I assumed it was typical. I’ll take another look to see if I can see anything there.
If I disable the AWS plugin the Guzzle activity disappears but the response times don’t significantly change, so whatever it is it’s not a silver bullet.
That sounds great about Craft 5 – I look forward to giving it a try.
Just to draw a line under the Guzzle mystery, the site has some Neo fields with icons and their Twig svg() call is what’s generating the S3 requests.
Removing the svg()
call is turning a 4.5s request into a 3s request, so I’ll see there might be an easy <img>
replacement to be found in Neo.
Huh, so the SVGs must have remote image references within them.
Nothing that obscure – the block icon SVGs are themselves on an S3 volume, so they’re inadvertently adding 40 HTTP requests to every entry edit page (since there are ~40 block types). I had a look at refactoring Neo to use cacheable <img>
tags but it wouldn’t have been a quick change. Fortunately moving the icons volume from S3 to a local folder is an easy alternative and seems to deliver the same performance boost.
I have experienced this same behaviour on the CP side. On the front end I can optimize through twig and eager-loading, so no issue there. It's on the Admin / CP side that's slowest.
Matrix -> SuperTable -> Matrix
— but found that really slow on pages with several rows/cols/blocks. Neo -> SuperTable
— found it slightly faster because Neo was handling a layer of structure, but still quite slow on anything but a simple page.Neo -> Individual Fields
— I notice that the Debug Toolbar reports half the DB calls but maybe only a slight (10%?) reduction in time spent both loading the page and saving an entry — with saving an entry being still quite slow.I did notice that while saving an entry takes a while, when I edit an entry I see the temporary draft is saved rather quickly. Not sure if any of this helps.
Looking forward to what CraftCMS 5 has to offer!
I did notice that while saving an entry takes a while, when I edit an entry I see the temporary draft is saved rather quickly. Not sure if any of this helps.
If that happens without any form edits, are you seeing a field get the blue “edited” status indicator?
(Late follow-up) I do see the indicator, but this is with form edits.
(Update)
Neo -> Individual Fields
did improve page save time a little. Neo -> CKEditor -> Entry Blocks
and I find it better.@adrianjean Are you still getting the automatic draft creation issue?
What happened?
Hi,
for our projects we use Craft CMS (PRO) and usually we need the client to compose the individual pages himself from the individual visual blocks. We implemented the management of the structured content of these blocks using Neo and SuperTable plugins.
Note: I am creating this ticket in the Craft CMS, Neo and SuperTable GitHub repositories. I am interested in the opinion of the authors and architects of Craft CMS and these plugins. After investigating these performance issues, it is evident that some optimizations and solutions to these problems can be implemented in the Craft CMS code and then subsequently in the individual plugins.
Facts about our project
Settings -> Fields
section - 20 of them are of Asset type (typically images for visual components). We have all assets in only one common volume, which is on one local filesystem.Issues we have and perceive from the Craft CMS perspective
Our investigation of the problems and findings
How can you help us
We would like to hear an informed opinion from the authors/architects of the existing code on the possible causes of these problems, but our investigation of the existing code shows that:
getRootFolderByVolumeId($volumeId)
method is called over 1,000 times, causing over 1,000 identical queries to the database. We have therefore implemented a simple composer patch below that will reduce the number of related identical duplicate DB queries from 1,000 to units and likewise reduce the number of events or logs by thousands. A sample of this patch is at the end of this ticket. Even this one small optimization sped up page generation with page editing by 15-30%. However, this patch does not address the situation where the volume settings are modified as part of the script run, so it is not an ideal solution.How to proceed?
We really like the Craft CMS and a number of useful plugins and we are trying to promote them. Unfortunately we have now run into some very fundamental performance issues.
So the first thing we thought of was to start investigating these causes ourselves, think of and implement some optimizations and then send pull requests.
As authors, do you please have an opinion on how to solve this situation? We'd be happy to help with optimizations, but if any of the Craft CMS architects have already had some thoughts and ideas on these improvements, we'd like to know so we don't do double work. This will also increase the chances that even if we design and implement some specific optimizations ourselves, they will be incorporated into future versions faster and without major comments.
You may also be self-aware that, based on some historical architectural decisions, some forms of efficient caching are now not applicable at all. For example, I imagine that serializing some complexly loaded objects into the cache won't work because the architecture requires that lifecycle of their initialization throws a number of events on which other functionality depends.
I would be very grateful for any thoughts and ideas on how to grasp and implement these necessary optimizations.
If needed, I can also provide a complete configuration of our CMS, including a database dump to your e-mail address.
Thank you very much for your time and support.
Patch
getRootFolderByVolumeId()
Craft CMS version
4.4.13
PHP version
8.1.18
Operating system and version
Debian 11 Bullseye
Database type and version
MariaDB 10.11.2
Image driver and version
GD 8.1.18
Installed plugins and versions