PHP out of memory when using the search in File Browser with TYPO3 v11.5

[mmunz]

Version: aus_driver_amazon_s3 v1.12.1 TYPO3 v11.5.26 PHP: 8.2.3

We have a S3 bucket with around 100000 files. In TYPO3 to 11.5 we experience out of memory errors when using the search for Files in the File Browser (above the file tree):

The search calls the endpoint /typo3/ajax/filestorage/tree/filterData?token=xyz&q=searchTerm with ajax.

• Without s3 driver the search takes around 1s
• With s3 driver the request is running for 30-40s and then ends up in a php memory error:

Raising the memory limit to 1G still runs into the memory limit. Raising the limit to 2G the filter works, but needs about 220s to finish.

This issue is likely related to #124

And yes, as you can see from the exception we run a bit a special setup mixed up with symfony, but i'm relatively confident that this is not the cause of these issues, because backend code is rarely touched and should run independently.

[MohsinQK]

Same Problem. TYPO3 V11.5.27 Number of files approx 61K

Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 4096 bytes) in /var/www/html/packages/aus_driver_amazon_s3/Classes/Driver/AmazonS3Driver.php on line 803

[MohsinQK]

@mmunz In my case there was a folder in the root directory and the name was just "/" that was causing this issue.

[weakbit]

@mmunz @MohsinQK You could give the branch feature/performance-improvements-caching a try, it has some smaller improvements.

Also the caching framework is used and could help. For examle it could be persistent and controlled by any other application managingn the Data in S3.

Please leave a comment if that helps.

[mmunz]

@MohsinQK thanks, but i don't have such a folder. Just many files in the root of the bucket and 5 folders.

[mmunz]

@weakbit i've now tested feature/performance-improvements-caching:

• without the feature/performance-improvements-caching branch i still get the OOM error
• with the improvements branch: no OOM, but the request to /typo3/ajax/filestorage/tree/filterData?token=xxx&q=searchWord times out after 5 Minutes: HTTP Error Code 504, "The gateway did not receive a timely response from the upstream server or application."

[Lagerregal]

Thanks for pointing this out! There was a bug in applying the search filter, which was solved with the latest commit b3802fe496b2f981e347aa132a729b5e524a4339

Could you please check if this solves the issue?

[mmunz]

@Lagerregal the search result is loading now with 512M memory limit. It is still not fast (80s), but the OOM is gone.