vkuznet / transfer2go

Distributed, loosely couple agent-based transferring system
MIT License
8 stars 2 forks source link
agent-request cms phedex priority tfc transfer-request

transfer2go

Build Status Go Report Card GoDoc DOI

Go implementation of CMS PhEDEx distributed, loosly coupled agents for CMS transfering data.

Description

The CMS experiment at the LHC proton-proton collider developed PhEDEx (Physics Experiment Data Export) service as reliable and scalable data management system to meet experiment requirements in Run I/II. Designed in 2004, and written mainly in Perl, it is still used today for managing multi-PB transfer loads per week, across a complex topology of dozen of Grid-compliant computing centres.

Its architecture, instead of having a central brain making global decisions on all CMS replica allocation, has a data management layer composed of a set of loosely coupled and stateless software agents - each managing highly specific parts of replication operations at each site in the distribution network - which communicate asynchronously through a central blackboard architecture. The system is resilient and robust against a large variety of possible failure reasons, and it has been designed by assuming a transfer will fail (thus implementing fail-over tactics) and being completely agnostic on the lower-level file transfer mechanism (thus focusing on full dataset management and detailed transfer bookkeeping). Collision data and derived data collected at LHC that allowed to achieve the Higgs boson discovery by ATLAS and CMS experiments at LHC were transferred in the CMS worldwide domain using this toolkit.

The aim of this project is to extend basic PhEDEX functionality to address up-coming challenges in exa-byte HL-HLC era via implementation of modern Go programming language.

The motivation for the effort is many fold: