search

stephenmcd / filebrowser-safe

File manager for Mezzanine
Other
41 stars 104 forks source link

Amazon S3 Performance #68

Open overshard opened 8 years ago

overshard commented 8 years ago

I've swapped over to using S3 for file storage and, as many know, it's extremely slow with filebrowser. I've taken the time to track the problem down!

The issue is the get_directory() function in filebrowser_safe/functions.py. Hover, I've been extremely lazy and not tracked the issue past this. In my fork I've just deleted everything except for dirname = DIRECTORY and return dirname because I don't use the site system anyways. https://github.com/stephenmcd/filebrowser-safe/compare/master...overshard:master

This makes my filebrowser just as fast as if I were storing files natively, a directory of 10k files loads up in less than a second for me, before it would freeze and timeout with more than 5 files in the directory. I believe the problem is too many calls are being made to S3 due to this one function.

Obviously the solution is not what I have here but I wanted to bring this to everyone's attention because I know it's a problem point for many wanting to use S3 and someone can craft a solution based on my discovery. And if other people aren't using the site system you can use this till it actually gets fixed.

stephenmcd commented 8 years ago

That's super useful, thank you.