Open carolyncole opened 7 years ago
In Sufia 7.2 Migration the version importer reads the entire content into memory. https://github.com/samvera/sufia/blob/7.2-migration/lib/sufia/import/version_builder.rb#L38
This works fine for small files, but when you start migrating larger files you can run out of memory.
Stream the file from Fedora instead of doing a single read
Reads the entire content into memory
We fixed this in ScholarSphere here: https://github.com/psu-stewardship/scholarsphere/commit/8f3f0d1#diff-2897df61a9e8723e3a0f8b8ee15638fcR51
Descriptive summary
In Sufia 7.2 Migration the version importer reads the entire content into memory. https://github.com/samvera/sufia/blob/7.2-migration/lib/sufia/import/version_builder.rb#L38
This works fine for small files, but when you start migrating larger files you can run out of memory.
Expected behavior
Stream the file from Fedora instead of doing a single read
Actual behavior
Reads the entire content into memory
Related work
We fixed this in ScholarSphere here: https://github.com/psu-stewardship/scholarsphere/commit/8f3f0d1#diff-2897df61a9e8723e3a0f8b8ee15638fcR51