Several sites have asked for small file aggregation similar to HSI. Two reasons: (1) more efficient to pull the aggregate from tape, and (2) reduce the HPSS metadata overhead associated with these small files. HPSS provides built-in aggregation but it does not solve (2). And this solution could likely be generalized to POSIX storage.
Several ideas come to mind:
Who triggers the aggregation?
explicitly chosen by the user?
End point manager configures Transfer to sample transfer and tell DSI to aggregate?
System admin configures the DSI when/how to aggregate?
Is tar support the right path? It would be compatible with htar. What about zip format?
How does this work with S3 and https?
How do we handle concurrent transfers and recovery scenarios?
Several sites have asked for small file aggregation similar to HSI. Two reasons: (1) more efficient to pull the aggregate from tape, and (2) reduce the HPSS metadata overhead associated with these small files. HPSS provides built-in aggregation but it does not solve (2). And this solution could likely be generalized to POSIX storage.
Several ideas come to mind:
Who triggers the aggregation?
Is tar support the right path? It would be compatible with htar. What about zip format?
How does this work with S3 and https?
How do we handle concurrent transfers and recovery scenarios?