The egg format was designed according to the "file == run" model. Runs can be artificially stretched across multiple files just by having the user of the monarch library take care of the details of what it means to have a multi-file run.
However, if multi-file runs are now the desired mode of operation, the egg3 format should be modified to handle that. Here's a no-doubt incomplete list of things that need to be changed:
Dealing with the header should be simplified, and assumptions can be made about the structure of the files that follow the first one (with appropriate checking, though we do want things to be fast).
Acquisitions need to be able to stretch across files. The concept of an acquisition needs to be separated from the HDF5 dataset that stores records.
The egg format was designed according to the "file == run" model. Runs can be artificially stretched across multiple files just by having the user of the monarch library take care of the details of what it means to have a multi-file run.
However, if multi-file runs are now the desired mode of operation, the egg3 format should be modified to handle that. Here's a no-doubt incomplete list of things that need to be changed: