scientist-softserv / britishlibrary

Other
3 stars 0 forks source link

Direct upload of large files #550

Closed cziaarm closed 3 weeks ago

cziaarm commented 4 months ago

Story

Refs #534

Expected Behaviour Before Changes

Files >~ 1-1.5GB do not complete upload/attachment when fcrepo backed by s3

Expected Behaviour After Changes

File > 1GB will by-pass fcrepo upload and go direct to s3.

Due to fcrepo/modeshape bug the filesize is not being sent to s3 as metadata and this will cause files to be streamed into memory so larger files will use all the available memory and cause the upload to fail (and sometimes fcrepo to restart). Working from https://github.com/scientist-softserv/utk-hyku/pull/631/files. This change will mean that files > 1GB will be sent direct to s3 by Hyrax... Then fcrepo object will be created and correctly associated with s3 bitstream as if it had gone via fcrepo.