department-of-veterans-affairs / caseflow-efolder

Tool for bulk download of efolder claim files
16 stars 8 forks source link

Large zip downloads are very slow #1140

Open pkarman opened 4 years ago

pkarman commented 4 years ago

One thing to consider is that we currently stream S3 downloads via our Rails app.

On a GFE VPN connection it can take 20+ minutes to download a 4G file.

Investigate whether this architecture holds open a db connection for that long, and consider instead signing the S3 request and simply redirecting to the S3 URL directly (bypassing efolder and downloading directly from S3).

https://github.com/department-of-veterans-affairs/caseflow-commons/blob/master/app/services/caseflow/s3_service.rb#L54

pkarman commented 4 years ago

Consider https://piotrmurach.com/articles/streaming-large-zip-files-in-rails/