search

laravel / nova-issues

554 stars 35 forks source link

Timeouts and out of memory errors handling very large selects with searchable and showCreateRelationbutton #3163

Closed eugenefvdm closed 3 years ago

eugenefvdm commented 3 years ago

Description:

This is exactly the same as this issue https://github.com/laravel/nova-issues/issues/1538 and as per @jbrooksuk 's request, I am opening a new issue as we're using a much later version of Nova.

In our case we're have a select that is ->searchable() and the select has 30 000 records. It's a CRM and the problems happens at the point of creating a lead. One needs to ability to select a current client, if it exists.

The resource looks like this:

BelongsTo::make('Client')
                ->searchable()
                ->withSubtitles()
                ->hideFromIndex()
                ->showCreateRelationbutton(),

The symptom of the problem varies between production and testing. In production we typically have a timeout whereas in testing we run out of memory.

Some 3rd party plugins have Ajax friendly methods of obtaining the select list, but we have the need of ->showCreateRelationButton() so they won't work.

Detailed steps to reproduce the issue on a fresh Nova installation:

Make sure inspect element is also on to see browser window errors.

Update: Additional diagnostic information

Upon closer inspection the XHR requests Inspect Element / Network, show this is the culprit taking very long:

https://dev.test/nova-api/leads/associatable/client?current=20607&first=false&search=&withTrashed=false&viaResource=&viaResourceId=&viaRelationship=

We changed the timeout in Forge from 30 seconds to 60 seconds, and now at 90 seconds it works. So after 1.2 minutes the field is populated. The packet size is also quite large, network 500K actual size 1.8 MB.

Screenshot from 2021-01-13 18-03-06

I will dive into MySQL query logging to see maybe something is amiss.

zupolgec commented 3 years ago

@davidhemphill is there a reason why this issue has been closed? I still notice the problem. Thanks!

eugenefvdm commented 3 years ago

My feedback is one has to carefully consider the application context when you are experiencing this problem. In our case we had a Client model with 25K records and only after careful analysis of performance we noticed numerous things, e.g.

I had a contractor look at the problem and he provided this early return workaround in the massive model:

public function fields(Request $request)
    {
        // Early return
        if (request()->is('*associatable*')) {
            self::$with = [];
            return [];
        }

After that the problem pretty much went away, but if I was you I could go back to my application and see if there isn't something fundamental performance wise that you could do different perhaps using the Laravel debugbar.

The fix mentioned in this issue on the Laravel Nova side is anyway related to "relatableSearchResults". You could play with that value too, or if you're using Scout, scoutSearchResults.

zupolgec commented 3 years ago

I found out that the reason is mostly that the query has no limit on the number of results and every result is an hydrated Eloquent model, thus requiring more time and throwing out of memory or time execution limit errors. At the moment I'm just limiting to 500 the number of results returned by a relatableQuery but I feel there should be a way to reduce the memory footprint by limiting the model attributes or limiting the number of results directly from the field declaration.

crynobone commented 3 years ago

Not everyone happy by enforcing a default limit on Laravel Nova 3, so we introduced a property to configure this. Please refer to documentation for details.

zupolgec commented 3 years ago

classic RTFM 😔 sorry guys for wasting your time and thank you for getting back so quickly

github-actions[bot] commented 2 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.