File format for saving fluid simulation output

[vellamike]

We need a standard way to save simulation outputs. (samples in time of particle positions). Sibernetic has some rudimentary support for saving outputs (using the -l_to flag when running the executable saves some files to the buffers folder, though this is not working properly at the moment).

The file format should be shared between Geppetto and Sibernetic (allowing for simulation comparisons and visualisation of saved simulations in both frontends).

Does anyone have any thoughts on this? In particular @a-palyanov @gidili @tarelli @JimHokanson and @MichaelCurrie might have a view on the best way forward.

[MichaelCurrie]

We've followed the Schafer Lab's example and used the HDF5 format. It appears to be a standard format used for scientific data these days. Perhaps you could consider using that. We use the H5PY Python library for file i/o.

That said, as long as there is a Python library somewhere that can process the file you create, we should be able to make it work.

[slarson]

Perhaps @MichaelCurrie https://github.com/MichaelCurrie we should link to the details of the HDF5 implementation (how many points for a worm 'frame', how many FPS, what metadata, etc) to give a clearer sense? I've seen you point to this in the past, just don't have the link handy right now.

On Mon, Dec 16, 2013 at 10:11 AM, Michael Currie notifications@github.comwrote:

We've followed the Schafer Lab's example and used the HDF5 formathttp://en.wikipedia.org/wiki/Hierarchical_Data_Format. It appears to be a standard format used for scientific data these days. Perhaps you could consider using that. We use the H5PY Python library for file i/o.https://github.com/MichaelCurrie/movement_validation/blob/master/README.md#tools-used

That said, as long as there is a Python library somewhere that can process the file you create, we should be able to make it work.

— Reply to this email directly or view it on GitHubhttps://github.com/openworm/OpenWorm/issues/154#issuecomment-30684623 .

[vellamike]

HDF5 is a good container, but I'm more thinking about higher-level details like what data to save. There's stuff like the elastic connections, different types of particles timestamps and so on.

On 16 December 2013 18:38, Stephen Larson notifications@github.com wrote:

Perhaps @MichaelCurrie https://github.com/MichaelCurrie we should link to the details of the HDF5 implementation (how many points for a worm 'frame', how many FPS, what metadata, etc) to give a clearer sense? I've seen you point to this in the past, just don't have the link handy right now.

On Mon, Dec 16, 2013 at 10:11 AM, Michael Currie notifications@github.comwrote:

We've followed the Schafer Lab's example and used the HDF5 format< http://en.wikipedia.org/wiki/Hierarchical_Data_Format>. It appears to be a standard format used for scientific data these days. Perhaps you could consider using that. We use the H5PY Python library for file i/o.< https://github.com/MichaelCurrie/movement_validation/blob/master/README.md#tools-used>

That said, as long as there is a Python library somewhere that can process the file you create, we should be able to make it work.

— Reply to this email directly or view it on GitHub< https://github.com/openworm/OpenWorm/issues/154#issuecomment-30684623> .

— Reply to this email directly or view it on GitHubhttps://github.com/openworm/OpenWorm/issues/154#issuecomment-30687106 .

[MichaelCurrie]

@JimHokanson has detailed the file structure of the HDF5 file created by the Schafer Lab here. It would not be necessary for you to calculate all of the data in this structure: along the Movement Validation pipeline, this is Stage 6.

Instead, we could take the simulated input either at Stage 1 (raw video), or Stage 2 (wireframe). The wireframe is "normalized" in Stage 3 into exactly 49 measurements per frame along the worm. For instance, the skeleton (the midline of the worm body) is specified at 49 points from head to tail for each frame of video (at 25 FPS).

I'm not sure about particles, elastics connections, etc, as it would not be necessary for our movement validation. But perhaps the worm simulation could output a 25 FPS wireframe of the contour. Our processing pipeline would then take this contour data and generate all measurements, features, and statistics from that.

If this would be useful perhaps we could give you an example HDF5 file populated that way. I'm sure @JimHokanson will have some thoughts on this beyond what I've said, however.

[slarson]

I'm thinking an example HDF5 file would be very helpful.

On Mon, Dec 16, 2013 at 1:59 PM, Michael Currie notifications@github.comwrote:

@JimHokanson https://github.com/JimHokanson has detailed the file structure of the HDF5 file created by the Schafer Lab herehttps://github.com/JimHokanson/openworm_docs/blob/master/Movement/Data/MRC_HDF5.md#worm-file-structure .

It would not be necessary for you to calculate all of the data in this structure, however: we could take the simulated input either at _Stage 1_of the Movement Validation pipelinehttps://github.com/MichaelCurrie/movement_validation/blob/master/documentation/Processing%20Pipeline.md(raw video), or Stage 2 (wireframe). The wireframe is ultimately "normalized" into exactly 49 measurements per frame along the worm. For instance, the skeleton (the midline of the worm body) is specified at 49 points from head to tail for each frame of video (@25 https://github.com/25 FPS).

Perhaps the simulated worm could output a 25 FPS wireframe of the contour. Our processing pipeline would then take this contour data and generate all measurements, features, and statistics from that.

If this would be useful perhaps we could give you an example HDF5 file populated that way. I'm sure @JimHokanson https://github.com/JimHokansonwill have some thoughts on this beyond what I've said, however.

— Reply to this email directly or view it on GitHubhttps://github.com/openworm/OpenWorm/issues/154#issuecomment-30705296 .

[JimHokanson]

I think we can be pretty flexible on the format of the file. The minimum that we need is a "contour" of the worm over time. Currently we can only really handle a 2d projection of the worm resting on a surface (i.e. no swimming in 3D). In addition we also need the location of the head, tail (preferably as indices into the contour), and finally the location of the side with the vulva, which could also be an index.

The contour should contain x,y pairs over time and should be ordered, preferably by going from:

• head
• side 1
• tail
• side 2
• ending just before repeating the head point

We have been using 49 points for the skeleton, which with two sides and not duplicating the head and tail comes out to 96 points.

Jim

On Mon, Dec 16, 2013 at 9:45 PM, Stephen Larson notifications@github.comwrote:

I'm thinking an example HDF5 file would be very helpful.

On Mon, Dec 16, 2013 at 1:59 PM, Michael Currie notifications@github.comwrote:

@JimHokanson https://github.com/JimHokanson has detailed the file structure of the HDF5 file created by the Schafer Lab here< https://github.com/JimHokanson/openworm_docs/blob/master/Movement/Data/MRC_HDF5.md#worm-file-structure>

.

It would not be necessary for you to calculate all of the data in this structure, however: we could take the simulated input either at _Stage 1_of the Movement Validation pipeline< https://github.com/MichaelCurrie/movement_validation/blob/master/documentation/Processing%20Pipeline.md>(raw video), or Stage 2 (wireframe). The wireframe is ultimately "normalized" into exactly 49 measurements per frame along the worm. For instance, the skeleton (the midline of the worm body) is specified at 49 points from head to tail for each frame of video (@25 https://github.com/25 FPS).

Perhaps the simulated worm could output a 25 FPS wireframe of the contour. Our processing pipeline would then take this contour data and generate all measurements, features, and statistics from that.

If this would be useful perhaps we could give you an example HDF5 file populated that way. I'm sure @JimHokanson < https://github.com/JimHokanson>will have some thoughts on this beyond what I've said, however.

— Reply to this email directly or view it on GitHub< https://github.com/openworm/OpenWorm/issues/154#issuecomment-30705296> .

— Reply to this email directly or view it on GitHubhttps://github.com/openworm/OpenWorm/issues/154#issuecomment-30722083 .

[tarelli]

@vellamike this ties into what we discussed last time right?

[slarson]

Guys -- @a-palyanov @gidili @tarelli @JimHokanson @MichaelCurrie -- I'm pretty sure we have determined that this will be the geppetto recording format being developed by @tarelli and @jrieke over here? Closing for now as there are a bunch of other issues related to this.