search

theriverman / django-minio-backend

Minio Backend for Django
https://pypi.org/project/django-minio-backend/
MIT License
112 stars 22 forks source link

Uploading large files consumes a lot of memory #50

Open toabi opened 3 months ago

toabi commented 3 months ago

https://github.com/theriverman/django-minio-backend/blob/3.6.0/django_minio_backend/models.py#L166-L181

The part here could also do multipart uploads when setting length=-1 and part_size=some_buffer_size. (see https://min.io/docs/minio/linux/developers/python/API.html#put-object-bucket-name-object-name-data-length-content-type-application-octet-stream-metadata-none-sse-none-progress-none-part-size-0-num-parallel-uploads-3-tags-none-retention-none-legal-hold-false)

As a local workaround I'm currently doing this:

s3_client = default_storage.client
result = s3_client.put_object(
    bucket_name=settings.MINIO_MEDIA_FILES_BUCKET,
    object_name=output_file_path,
    data=temp_file,
    length=-1,  # enable multipart upload
    part_size=MAXIMUM_CHUNK_SIZE_BYTES,  # max size per part
)

But it would be nice to have it as an option/setting/default…

theriverman commented 2 months ago

Would you be up for opening a PR to see how would you like to get this implemented, @toabi ?

toabi commented 2 months ago

Well… currently I don't have much ressources to spare, but if another project where we use this library will require it maybe it'll be worth to implement it properly without workarounds.