JBorrow
commented
11 months ago I think there is a more general issue here, and this part of the code could use a little refactoring.
There are essentially a few ways that we want to interact with the store:
- Librarian server <-> database, knowing what stores exist.
- Librarian server -> store for management (e.g. moving staged files, checking for free space).
- Client -> store for uploading data.
At the moment this is all essentially bundled up into the BaseStore class (later subclassed into Store, but that really just adds in the database ORM). Hence the BaseStore class makes a couple of tricky of assumptions about the way that stores should be interacted with:
- It assumes that the Store is a POSIX filesystem
- It assumes that all commands must/may be executed over SSH
- It assumes that all storage commands are synchronous (and makes those that are not synchronous be so; how does SneakerNet work?)
and also means that any transfer protocols must be implemented as methods on the class.
This is tricky because:
- SO:UK may not have a POSIX filesystem available
- Installing openssh-server and using it (even for local storage, i.e. ssh-ing into our own machine) might be tricky at some centers
Perhaps it makes sense to refactor into something that looks close to the following peseudocode, taking advantage of dependency injection. We want to abstract away the concept of a store and a transfer (which is not really currently the case because of its monolithic nature). At the moment the globus destination endpoint ID is stored in the config, not by the host librarian server, for instance.
# Running on librarian server
class StoreMetadata(db.Model)
""" Core metadata for store management (e.g. hostname, globus endpoint ID, etc.) """
#... Stuff that registers the store in the Librarian database
class CoreStoreManager:
""" Prototype for store management (e.g. moving staged files, finding storage). """
def move(from, to):
#...
class POSIXStoreManager(CoreStoreManager):
""" Example of a store manager that is 'used' by the librarian server """
def move(from, to):
#... ssh self.ssh_host; mv from to
class LocalPOSIXStoreManager(CoreStoreManager):
""" ... another manager """
def move(from, to):
#... os.move(from, to)
# Running on the client
class TransferManager:
""" Prototype for transfer management """
def transfer(local_path, remote_path=None):
""" Remote path can be pre-staged """
#...
class RsyncTransferManager(TransferManager):
""" Example of a transfer manager using rysnc """
def __init__(self, host_name, ...):
self.host_name = host_name
def transfer(local_path, remote_path=None):
#... rsync loac_path self.host_name:remote_path
class GlobusTransferManager(TransferManager):
""" Example of a transfer manager using globus """
def __init__(self, destination_endpoint_id, ...);
self.destination_endpoint_id = destination_endpoint_id
def transfer(local_path, remote_path=None):
#... globus_sdk.initiate_transfer(local_path, remote_path)...This would allow a flow as follows:
# Server
stores = StoreMetadata(...)
best_store: CoreStoreManager = stores.find_store_with_available(bytes=...)
staging_location: str = best_store.stage(bytes=...)
best_store.post_transfer_information(staging_location, ...)
best_store.process_staged_file(...)
# Client
client = LibrarianClient(...)
transfer_manager: TransferManager = client.get_store_for(bytes=...)
transfer_manager.transfer(local_path)In this way the transfer (e.g. globus, rsync, cp for local transfers, etc) is completely transparent.
plaplant
Starting this tracking issue as we try to remove the
sshrequirement from wherever possible in Librarian to make things NERSC-etc. friendly.@plaplant: It is my understanding that all Stores must be accessible over SSH. All inter-Librarian communication occurs from Store-Store (not Librarian to Librarian?). At the site, we have the Store url as
localhost, so we ssh into ourselves and in theory launch globus transfers from there.I would like to sub-class
BaseStoreto createLocalStorewhere all file operations are local (i.e. basically remove_ssh_slurp).A couple of quick questions: are my assertions right, and if so is there a reason that all Stores must be remote? Why is there no current option for local storage at the position that the Librarian server actually lives?