DifferentiableUniverseInitiative / flowpm

Particle Mesh simulation in TensorFlow
https://flowpm.readthedocs.io/
MIT License
90 stars 20 forks source link

Add support for distributed I/O #26

Open EiffL opened 5 years ago

EiffL commented 5 years ago

When eventually scaling up mesh computation to very large volumes, we need to output the layed out tensors to a file system. For cori and similar systems we will need to have each process output its own slice of the data. Not sure what's the best approach to do this yet.

EiffL commented 4 years ago

At least the parallel reading can be taken care of by this: https://github.com/tensorflow/mesh/blob/master/mesh_tensorflow/experimental/input_reader.py \o/