Large assets crash server

aerni commented 1 year ago

Bug description

I've got a private Digital Ocean Space whose content I'd like to see in the CP. So, I created an asset container and selected my S3 disk. The asset container in the CP loads forever and eventually crashes the server. There's only one file, but it's a 10GB mp4. I suppose this isn't meant to handle large assets?

How to reproduce

Upload a huge file to a DO Space
Connect an asset container to your S3 disk
See the CP crash

Logs

No response

Environment

Environment
Application Name: Siloam Crowdfunding
Laravel Version: 9.52.15
PHP Version: 8.2.5
Composer Version: 2.5.5
Environment: local
Debug Mode: ENABLED
URL: siloam-crowdfunding.test
Maintenance Mode: OFF

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: NOT CACHED

Drivers
Broadcasting: log
Cache: statamic
Database: mysql
Logs: stack / single
Mail: smtp
Queue: redis
Session: file

Statamic
Addons: 4
Antlers: runtime
Stache Watcher: Disabled
Static Caching: Disabled
Version: 4.19.0 PRO

Statamic Addons
aerni/social-links: 3.1.0
jonassiewertsen/statamic-livewire: 2.12.0
spatie/statamic-responsive-images: 4.1.1
visuellverstehen/statamic-classify: 2.6.1

Installation

Fresh statamic/statamic site via CLI

Antlers Parser

runtime (new)

Additional details

No response

aerni commented 1 year ago

Update. After a few hours, the asset container doesn't timeout anymore and the asset now shows in the CP. I have a feeling that it took that long to analyze the asset for the metadata. However, performing a simple task like renaming the asset and we're back to timing out.

I can confirm, that the rename actually did happen when looking at the asset in the Digital Ocean dashboard. But the metadata file wasn't renamed yet. I'm pretty sure that metadata is the issue here. It's probably still analyzing the huge file which causes the server to crash.

jasonvarga commented 1 year ago

We use https://github.com/JamesHeinrich/getID3 under the hood to get meta data about videos.

https://github.com/statamic/cms/blob/e66df876e11cd8e757eed4fd0691380870e6d99b/src/Assets/ExtractInfo.php#L14

It reads the file. I'm guessing reading a 10gb file will take a while. 😄

When you say it crashes, is it just that request? Do all requests start throwing errors?

Maybe you can see something on that repo that gives a hint about handling larger files?

aerni commented 1 year ago

Yeah, all requests start spinning. Have to reboot PHP to make it work again. Can't see any errors in the console or in the logs. I'll do some digging on that repo.

aerni commented 1 year ago

I did some further testing and added the 10GB file to a local asset container. I used ray()->measure() in the fromAsset method of the ExtractInfo class to measure the performance. Generating the metadata took about 30ms. So the getID3 library doesn't seem to be the issue.

The issue seems to be the combination with S3. I've got the same file on a DO Space. I created a new asset container and used the S3 disk. I then renamed the file in the CP. Renaming took a few minutes until it even hit the fromAsset method. And the generating of metadata took 13 minutes!

I'm not sure if there is anything that can be done on the Statamic side of things? Maybe pushing things into a queue could help.

statamic / cms