statamic / eloquent-driver

Provides support for storing your Statamic data in a database, rather than flat files.
https://statamic.dev/tips/storing-content-in-a-database
MIT License
104 stars 74 forks source link

Timeout when using TermFieldtype antlers tag #343

Closed swen4 closed 2 weeks ago

swen4 commented 2 weeks ago

Bug description

When trying to query terms from a Terms Field in Antlers, my application times out. I have taxonomies and terms configured to use the eloquent driver in my config/statamic/eloquent-driver.php file.

The terms work fine in the control panel, it's only when I try to query the terms through Antlers that the issue arises.

When I revert the eloquent-driver for Terms to "file" the issue doesn't happen.

How to reproduce

  1. Create a taxonomy "Themes" and attach it to a collection.
  2. Create a "Theme" called "Example".
  3. Create an Antlers template for a single entry, and attempt to display the "Example" theme.
{{ themes }}
    {{ title }}
{{ /themes }}
  1. Configure eloquent driver for "terms" in config/statamic/eloquent-driver.php.
  2. Run migration for Taxonomies and Terms tables.
  3. Run php please eloquent:import-taxonomies and "yes" to both prompts
  4. Check entry page again, it now times out.

Logs

[26-Aug-2024 13:02:37 UTC] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 20480 bytes) in /Users/swen/Projects/statamic/vendor/laravel/framework/src/Illuminate/Database/Eloquent/Casts/Json.php on line 36
[26-Aug-2024 13:02:37 UTC] PHP Fatal error:  Allowed memory size of 536870912 bytes exhausted (tried to allocate 32768 bytes) in /Users/swen/Projects/statamic/vendor/symfony/error-handler/Error/FatalError.php on line 1

Environment

Environment
Application Name: Statamic
Laravel Version: 11.21.0
PHP Version: 8.2.22
Composer Version: 2.7.7
Environment: local
Debug Mode: ENABLED
URL: statamic.test
Maintenance Mode: OFF
Timezone: UTC
Locale: nl

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: file
Database: mysql
Logs: stack / single
Mail: log
Queue: sync
Session: file

Livewire
Livewire: v3.5.6

Statamic
Addons: 7
Sites: 1
Stache Watcher: Enabled
Static Caching: Disabled
Version: 5.23.0 PRO

Statamic Addons
jonassiewertsen/statamic-livewire: 3.6.0
rias/statamic-redirect: 3.7.2
statamic-rad-pack/runway: 7.8.0
statamic/eloquent-driver: 4.13.0
statamic/seo-pro: 6.0.3
superinteractive/statamic-super-link: 1.0.1
superinteractive/statamic-super-polls: dev-main

Statamic Eloquent Driver
Asset Containers: file
Assets: eloquent
Blueprints: file
Collection Trees: file
Collections: file
Entries: eloquent
Forms: eloquent
Global Sets: eloquent
Global Variables: eloquent
Navigation Trees: file
Navigations: file
Revisions: eloquent
Sites: file
Taxonomies: file
Terms: eloquent
Tokens: eloquent

Additional details

No response

ryanmitchell commented 2 weeks ago

Are you able to share a sample repository showing the issue? It seems to be related the type or quantity of data you are storing.

swen4 commented 2 weeks ago

@ryanmitchell I'll try to set up an example repository the coming days. Although my current project only has one "term" inside the taxonomy I'm trying to display, and only 17 terms in total, spread across other taxonomies. (Wasn't able to display those in Antlers either)

ryanmitchell commented 2 weeks ago

Hmm ok, feels like it might be a recursion issue then.

swen4 commented 2 weeks ago

@ryanmitchell After setting up an example repo, and doing some debugging, it seems to be related to how many entries my collection has.

I could not reproduce the issue with a fresh dataset, so instead of that I did an experiment:

  1. Deleted all entries in my repo
  2. Created a new entry, attached the term, and displayed it in Antlers -> This works fine
  3. Reverted to the original dataset of around 40K entries, and load a page -> Timeout
  4. Deleted entries until around 10K entries were left -> This works but loads extremely slow
  5. Removed the term query in my Antlers template -> Pages now load near-instantaneously
ryanmitchell commented 2 weeks ago

Thanks for the update - sounds similar to: https://github.com/statamic/eloquent-driver/issues/344

As I mentioned there we have a bottleneck in the code here: statamic/eloquent-driver@master/src/Taxonomies/TermQueryBuilder.php#L213-L258

I'm not entirely sure how (or even if) we can avoid it while keeping parity with the stache driver.

swen4 commented 2 weeks ago

As a temporary workaround I created a custom Antlers tag that gets the data we need Probably not a great solution, but it works well enough for our use case:

<?php

namespace App\Tags;

use Statamic\Facades\Entry;
use Statamic\Facades\Term;
use Statamic\Tags\Tags;

/**
 * Workaround for performance issues with {{ termname }} Antlers tags.
 */
class RawTerms extends Tags
{
    /**
     * The {{ raw_terms:* }} wildcard method handler.
     *
     * @return string|array
     */
    public function wildcard($method)
    {
        if ($method === 'index') {
            return '';
        }

        $taxonomy = $method;

        $entry = $this->context->value('id') ? Entry::find($this->context->value('id')) : null;

        if (!$entry) {
            return '';
        }

        $terms = $entry->get($taxonomy);

        if (!is_array($terms)) {
            $terms = [$terms];
        }

        if ($terms && count($terms)) {
            $results = collect($terms)->map(function ($slug) use ($taxonomy) {
                $term = Term::query()
                    ->where('taxonomy', $taxonomy)
                    ->where('slug', $slug)
                    ->first();
                return $term ? $term->toAugmentedArray() : null;
            })->filter()->all();
            return $results;
        }

        return [];
    }
}
ryanmitchell commented 2 weeks ago

Thanks, I'm going to try and take another look at the problem query in the next week or so.

ryanmitchell commented 2 weeks ago

@swen4 I've done a quick PR here for a potential change - https://github.com/statamic/eloquent-driver/pull/345 can you let me know if that helps with your performance issue at all?

swen4 commented 2 weeks ago

@ryanmitchell I did some tests on my database with ~10k articles and noted the loading times:

Seems like the performance hit is negligible now, awesome!

ryanmitchell commented 2 weeks ago

That’s promising. It’s the number of terms that now matter - not the number of entries. I’m working on the assumption those will always be fewer than entries.