search

filestack / filestack-js

Official Javascript SDK for the Filestack API and content ingestion system.
https://www.filestack.com
MIT License
206 stars 76 forks source link

FS-10615 - Allow upload files bigger than 50GB #511

Closed prem-celestial closed 1 year ago

prem-celestial commented 1 year ago

I have created this PR to fix https://filestack.atlassian.net/browse/FS-10615. Please review the same.

Yaminim07 commented 1 year ago

The original chunk size of 6MB for default case and 8MB for intelligent ingestion is kept as it is for files smaller than 60GB for default case and 80GB for intelligent ingestion.

The following calculation has been added for file size greater than 60GB (default case) and 80GB (intelligent ingestion case),

  1. The number of chunks is fixed to 10,000 which is the max number of chunks allowed over s3 bucket

  2. The size of each chunk is determined by dividing file size to max number of chunks , e.g File size = 70GB chunk size = File size / Max number of chunks chunk size = 70 GB / 10,000

    This way the chunk size is made dynamic after file size exceeds 60GB for default case and 80GB for intelligent ingestion.