jschneier / django-storages

https://django-storages.readthedocs.io/
BSD 3-Clause "New" or "Revised" License
2.77k stars 867 forks source link

`collectstatic` is extremely slow with S3 Manifest #1255

Open catgirlinspace opened 1 year ago

catgirlinspace commented 1 year ago

hiya! I'm deploying my app on DigitalOcean App Platform. As part of the build step there, I run python manage.py collectstatic --noinput. Before switching to storing my static assets in a S3-compatible bucket this was instant, but now takes over 10 minutes when using the S3 manifest. The S3-compatible service I am using is Backblaze B2, although I wouldn't expect this to matter.

Configuration for my static files

storages.py

class StaticStorage(S3ManifestStaticStorage):
    bucket_name = 'splashcat-static'
    custom_domain = 'static.splashcat.ink'

settings.py

if not DEBUG:
    STORAGES = global_settings.STORAGES | {
        "default": {"BACKEND": "storages.backends.s3boto3.S3Boto3Storage"},
        "staticfiles": {"BACKEND": "splashcat.storages.StaticStorage"}
    }
DuncanTormey commented 1 year ago

@catgirlinspace I am experiencing this too, did you find a solution?

DuncanTormey commented 1 year ago

I ended up doing this.

python -m cosmos.manage collectstatic --no-input & # background the static collection
gunicorn cosmos.cosmos_site.wsgi --config cosmos/gunicorn_conf.py
catgirlinspace commented 1 year ago

@DuncanTormey hi, sorry for the late reply. my workaround for this was to just use collectstatic and serve it without S3 and then stick a CDN in front.

robd003 commented 8 months ago

I'm also experiencing a 10 minute duration when trying to upload ~2,700 files

davidmir commented 7 months ago

+1

pablodiegoss commented 4 weeks ago

+1

Any new info for this thread?