DiamondLightSource / httomo

High-throughput tomography pipeline
https://diamondlightsource.github.io/httomo/
BSD 3-Clause "New" or "Revised" License
4 stars 3 forks source link

Separate `DataSetBlock` from `DataSet` #233

Closed yousefmoazzam closed 4 months ago

yousefmoazzam commented 4 months ago

Changes

There's several ways to describe the main change at a high-level:

At a lower-level, the changes are:

Motivation

This change was partly motivated by the difficulties seen in trying to use FullFileDataSet in StandardTomoLoader to create blocks (ie, to create instances of DataSetBlock).

Another reason for this change was that, as development has continued, it was recognised that the main data object that methods and the task runner work with are blocks rather than chunks. In other words, blocks are the fundamental data object in httomo, not chunks. On the other hand, the original dataset design involves a class hierarchy where a chunk (represented by DataSet) is the fundamental data object, and from which other data objects derive from to get certain behaviour (DataSetBlock and FullFileDataSet). These two choices about which data object is the fundamental one clash, and consequences of this clash are likely manifested in some of the difficulties that have been encountered during development.