Restructuring of the ML workflow

[chpolste]

Mainly to address peak memory consumption and the organisation (on disk and in the code) and retrieval of data.