Bad performance of file tree with TYPO3 v11 and many files in bucket

[mmunz]

Version: aus_driver_amazon_s3 v1.12.1 TYPO3 v11.5.26 PHP: 8.2.3

We are in the progress of updating an instance from 10.4 to 11.5 and noticed some performance problems with this storage driver in v11.

We use an s3 bucket that has roughly 100000 files in it, each between some bytes to 40 MB size.

Loading the file tree in File List Module or File Browser is really slow.

Both read the file tree via an ajax request from /typo3/ajax/filestorage/tree/fetchData?token=xzy.

• Without the s3 driver this request takes about 100ms. It then only loads fileadmin which holds about 200000 files.
• With the s3 driver enabled the call will take almost 20s.

I haven't fully investigated this, but for now it seems to me that the file tree is build directly from S3 API requests instead of using data that should already be there in the sys_file table.

[cweiske]

Should be the same as #68.

[weakbit]

I have a Fix reported for TYPO3 Filelist https://review.typo3.org/c/Packages/TYPO3.CMS/+/79444

As soon as a sorting is set for a non-given property, or to be more explicit, a not yet loaded file-property, the core does a metadata lookup in the database (not dependent on this extension).

I created a folder in fileadmin with > 50k files and it also has huge performance issues.

Also a branch with some improvements is available feature/performance-improvements-caching but is still under Testing.

I would be happy to hear if that worked :)

[mmunz]

@weakbit i finally found some time and motivation to test this. In the meantime we cleaned up the s3 storage, so only about half of the files is left compared to my initial issue.

I tested both possible fixes independently and together, i.e. https://review.typo3.org/c/Packages/TYPO3.CMS/+/79444, feature/performance-improvements-caching and both at the same time:

With all three variants loading the file tree is still slow (about 17s) - so no improvement for me