pulibrary / lib_jobs

Enterprise Services batch processing tasks. Rails 7 Ruby 3.1.0
4 stars 0 forks source link

Only initialize the S3 Connection once #708

Closed sandbergja closed 5 months ago

sandbergja commented 5 months ago

Previously, we created 2 Aws::S3::Client objects for each file on the SFTP server: one for host/constitutent records, the other for bound-with records. As part of its initialization, Aws::S3::Client slurps a 800 KB JSON file into memory, which it uses to determine the endpoint it wants to use. 800 KB isn't too bad, but if we add two copies to memory 4000 times, it leads to some serious memory pressure.

related to #695