So far the transfer are file based. We need an R&D for root IO event streaming and data slicing in a context of CWP:
In current paradigms, physicists consider all events at an equivalent level of detail and in the format offering the highest level of detail that needs to be considered in an analysis. However, not every event considered in analysis requires the same level of detail. One consideration to improve I/O throughput is to design event tiers with different sizes. All events can be considered at a lighter-weight tier while events of interest only can be accessed with a more information-rich tier. Machine learning and similar highly-iterative techniques also provide new challenges in data management and access. Data access that is column-based instead of row-based can be of enormous benefit to many analysis techniques. For instance, instead of reading events in sequence and searching for one particular type of object in the events, to have the data management system be able to return all objects of a certain type in a dataset. Data queries that consider histogram indexing is another feature that may or may not provide performance increases in analysis.
So far the transfer are file based. We need an R&D for root IO event streaming and data slicing in a context of CWP:
Go based root IO https://godoc.org/go-hep.org/x/hep/rootio