dloy
commented
4 years ago Discussions on this indicated that an AWS ALB could be used. The only potential problem would be a synchronous download of a large container file - this is deprecated.
Closed dloy closed 3 years ago
dloy
commented
4 years ago Discussions on this indicated that an AWS ALB could be used. The only potential problem would be a synchronous download of a large container file - this is deprecated.
Issue
Right now there are 2 production Storage servers that are used for both Ingest handling and presign file and object handling for the UI and Dryad.
The Dryad presign object handling has become extensive and has caused issues for both Dryad and Ingest because the 1T temp space on the two production servers has been exhausted.
Proposal
Add 2 new production storage servers:
The existing 2 servers would be dedicated to ingest only and the added 2 would be used only by the UI (with Dryad).
With this architecture Dryad would be isolated to these production servers for both presign files and presign objects, and also for any synchronous zip files (been deprecated). All of these requests use the UI, so a single RLB could be dedicated.
Ingest would continue to use the store1 and store2 servers. This has the advantage that the temp space for those servers can only be impacted by concurrent ingest updating which is pretty well understood. Lately, few of the storage server deploys have been related to ingest, most are related to presign and Dryad. Adding the 2 new servers would allow deploys to be done separately so ingest updates would not impact Dryad except for sword updates.
Because two different sets of storage servers would be used, a second RLB would be needed for the UI with its own different name from what is used by ingest. This should be able to work with the existing apache on the web server.
Cost
Basing the additional costs on our current storage servers - the estimate (thank you Martin!) is:
Total for both servers: $176.80 * 2 = $353.60/mo