search

opencadc / storage-inventory

Storage Inventory components
GNU Affero General Public License v3.0
2 stars 9 forks source link

cadc-storage-adapter-swift should support a pool of clients #319

Open pdowler opened 3 years ago

pdowler commented 3 years ago

feature: pool of connections to back end swift endpoint

configure a max pool size store pool in JNDI, assume all SwiftStorageAdapter instances in a JVM share the pool (it is the same config file) use commons-pool to implement pool (see DBUtil extension in cadc-inventory-util for example GenericObjectPool usage)

Option 1: pool internal to SwiftStorageAdapter

Option 2: create PooledSwiftStorageAdapter that simply pools instances of and delegates to SwiftStorageAdapter

side note: SwiftStorageAdapter is intended to be thread-safe and a cursory look at the underlying joss library suggests it is thread-safe as well, but if we don't want to trust that then we would go with what looks like option 1.

pdowler commented 1 year ago

internally, SwiftStorageAdapter init now avoids a costly bit so connecting/authenticating with RGW only takes milliseconds, so this is less pressing.

Also, JOSS library maintains an internal pool of http connections to RGW and makes good use of http-keep-alive... something to keep an eye on.