Refactor of storage service

[hekaisheng]

This proposal aims to improve the stability of storage service and clarify the responsibilities of each actors created by storage service.

Actors on main pool

• StorageManagerActor
  • Create all necessary actors for both main pool and subpools.
  • Preserve setup params and teardown params for all storage backends.
  • Report storage status to cluster API
  • main APIs
  • async def upload_storage_info(self):
  • async def upload_disk_info(self):
• RefHolderActor
  • Hold the references for stored objects to prevent being evicted by storage backends like plasma.
  • mian APIs
  • async def ref_object(session_id: str, data_key: str, level: StorageLevel, object_id: ObjectID):
  • async def release_object(session_id: str, data_key: str, level: StorageLevel):
• StorageQuotaActor
  • Manage the quotas of all storage backends.
  • main APIs
  • def request_quota(self, size: int) -> bool:
  • def release_quota(self, size: int):
  • def update_quota(self, size: int):
  • def get_quota_info(self) -> Tuple[float, float]:
• SpillManagerActor
  • Manager spill tasks and decides which keys to be spilled. It records the keys who are spilling or spilled.
  • API
  • def get_spill_keys(request_size: int, level: StorageLevel, band_name: str, multiplier=1.1)
• TransferManagerActor
  • Manage transfer tasks and records the information of transfer tasks.
  • API
  • def get_fetch_infos(session_id: str, data_keys: List[str])

Actors on subpools

• StorageHandlerActor
  • The handler actor for each subpool. It holds clients of all storage backends and provides methods like get, put, etc.
  • It keeps the same APIs as older version.
• SpillActor
  • Called by StorageHandlerActor when spilling is triggered, normally created on IO processes, also necessary for GPU band processes.
  • main APIs
  • def spill(self, session_id: str, data_keys: List[int], level: StorageLevel, block_size=None):
• SenderActor
  • Called by StorageHandlerActor, normally created on IO processes, also necessary for GPU band processes.
  • It keeps the same APIs as older version.
• ReceiverActor
  • Called by SenderActor, normally created on IO processes, also necessary for GPU band processes.
  • It keeps the same APIs as older version.

Other changes

• Remove DataManagerActor, all storage meta will be stored in meta service on each worker, use MetaAPI to delete and update storage meta.

[qinxuye]

Key part I think is dividing entire module into submodules including spill, transfer etc.

[wjsi]

We also need APIs to fetch data into a priority list of storage levels to avoid redundant RPC calls.