Add support for a storage container for many morphologies in one file

[matz-e]

A while ago we added support for reading morphologies from groups in #163. This has led to experimental work by @1uc to see how well HDF5 files work to store many morphologies in a single file (presentation incoming). I think it would be good to consolidate the work done in our internal MorphoKit in MorphIO by

• Specifying how many morphologies should be stored in a single HDF5 file
  • the most simplistic approach by copying the contents of morphology files into sub-groups of a larger file. This layout is already facilitated by the group access in #163 and would be quick to implement.
  • concatenating all datasets of the morphologies into larger datasets and storing the offset, count into said datasets per morphology. This layout would allow to later perform similar optimizations as has been done for BlueBrain/libsonata by reading the required data for several morphologies in one go from the larger datasets.
• Adding a MorphologyStorage (or similar) class that takes either a directory or path to an HDF5 file and exposes a .get method to obtain morphologies stored
  • this can and should be expanded to provide optimized bulk access
  • at least a utility to create containers if not an interface to write should be included
  • this would be our new preferred way of reading morphologies for large circuits with a large number of unique morphologies
• Adding any other functionality in the reader required to support the format

[mgeplf]

Everything above looks fine. Please remember that this has to build under linux/mac and windows, since we produce wheels and such for all platforms.

a get method to obtain morphologies stored

What would be the argument to the get? A number? A string?

[1uc]

The current solution in morpho-kit is essentially:

class MorphologyStorage {
public:
    virtual morphio::Morphology get(const std::string& morph_name) const = 0;
};

class DirectoryStorage {
public:
    DirectoryStorage(const std::string& dirname);
    morphio::Morphology get(const std::string& morph_name) const override;

public:
    const std::string& dirname;
};

class FileStorage {
public:
    FileStorage(const std::string& container_name);
    morphio::Morphology get(const std::string& morph_name) const override;

public:
    HighFive::File file;
};

The assumption is that the file name is {dirname}/{morph_name}.{ext} where there's some guessing an precedence order for picking the extension.

[matz-e]

Everything above looks fine. Please remember that this has to build under linux/mac and windows, since we produce wheels and such for all platforms.

a get method to obtain morphologies stored

What would be the argument to the get? A number? A string?

As @1uc pointed out in the interface snippet, the morphology name as currently stored in SONATA. If we had integration with libsonata, one could actually use a node id. Not sure that this is desired, though.

[1uc]

Multiple circuits should be able to use the same morphology container. Hence, the node ID alone doesn't identify the morphology; conversely multiple nodes can map to the same morphology.