Future of MDF - stability and compatibility

[profix898]

It seems that in HH and HL you are putting a lot of work into MDF in order to use it as your standard format. This also means that you require a variety of changes to MDF to support the new scenarios. For us (BS) and also Würzburg (as far as I know) the MDF format is simply an exchange format, but it is not used internally for any of our work. The point here is that I'm already struggling to get 2.0 implemented (its not likely to be ready before 2018). But MDF turns out (more and more now) to be a moving target.

I'd be very much interested to hear your perspective on how you want MDF to evolve, your use cases, and how to maintain strict compatibility with MDF v2.0 once it is released. The compatibility point is really crucial for us, but I fear (from the issues raised here) you are pushing forward. Do you plan on providing a 'fallback' on v2.0 when saving your data or will you always go with the latest version?

Looking forward to your feedback, Thilo

[tknopp]

No you are misinterpreting this. Any update to v2.0 should be non-braking we do not want a moving target. We make this now right and then never change again. Only things that will be added are extensions.

[tknopp]

And just one further note. I am not(!) happy that we have to do this move with v2. But the v1 really had some shortcomings that were not easily addressable. We provide a converter v1->v2 so that it is at least easy to convert files.

[Neumann-A]

In my opinion, v2 will not be the last breaking change because it is still not feature complete from my viewpoint. Converters between breaking changes seem to be the only sane thing to do.

I still planning to create an issue for virtual HDF5 links including some major changes on how to view the data (It could be that we do not need to change that much but that is not something i am able to predict now). What I basically want is a more composable format with very strict definitions. But currently github issues are moving to fast for me and i dont have time to write it. (I am basically only writing issues on github currently)

The point here is that I'm already struggling to get 2.0 implemented (its not likely to be ready before 2018). But MDF turns out (more and more now) to be a moving target.

My tipp here: Do not implement MDF, implement a generic HDF5 implementation. I personally did not even write a single line of MDF specific code yet. All issues raised where issues which arised due to incompabilities between a generic HDF5 implementation and the MDF format. If you want to have a generic HDF5 implementation try to copy what HDF5.jl does. It seems to be relativ generic (If I had known that beforehand I would have used their design before coming up with my own design in c++) Then build on that abstraction and write classes like MDF_FileInfo, MDF_TracerList, MDF_TracerInfo, MDF_Whatever which are able to write and read the data using the indirection you created with the generic HDF5 implement (or only use one function called serialize for read and write) [if you read the data on construction or do a lazy read is up to you]. From those classes just define functions which give you the data you need!

[tknopp]

@NeumannIMT: This rather sounds like a new file format. You are free to develop this but stability will be the absolute highest priority for the MDF. I don't want to thwart you but our users will not forgive us if we make the MDF a moving target. The only reason we do this now is because there are not much MDF files generated so far (and given to collaboration partners). We are actively supporting people using the MDF and introducing braking changes will significantly increase the maintenance level required.

[Neumann-A]

@tknopp: I don't plan to make it a moving target but for me it does not feel like the MDF specification has reached the maturity to be considered stable. Too many open questions from my point of view, too many optional fields/groups, too focused on lissajous like scanners (see /acquistion/drivefield/ its basically designed for lissajous scanners). Then there is #69 which certainly will be a braking change. Maybe you are currently trying to push MDF too fast because it directly influences your work. I also think that MDF needs alot more input from other instituts.

[Neumann-A]

@profix898 Do you have a documentation for your current format? Or an example file? Maybe MDF can extract something from it.

[hofmannmartin]

Too many open questions from my point of view, too many optional fields/groups, too focused on lissajous like scanners (see /acquistion/drivefield/ its basically designed for lissajous scanners).

I have to object. During each step of the design of the format we have had x-space scanners, MPS systems and TWMPI scanners in mind. Name one scenario, which we can not be stored in the current version apart from #69.

Then there is #69 which certainly will be a braking change.

Its true that this will be braking. But for me the question is when it will be required in practice? Do you know of any group planing or building such a receive chain? Until then I see no reason to break anything for a purely academical case.

Maybe you are currently trying to push MDF too fast because it directly influences your work. I also think that MDF needs alot more input from other instituts.

We had long discussions with Lübeck, Würzburg, Braunschweig and the people from Philips. Not all here on github, but we certainly gave the German hardware developers the chance to raise their issues. As for the people from Berkley. They do not seem to have any interest for our format at the moment as we never got any kind of response from them. Which other institutes do you have in mind?

[profix898]

@NeumannIMT: We use HDF5 internally already, but in a very different way. So providing converters/adapters from our HDF5 storage format to/from MDF is exactly what I'm doing (also did for v1). Still, there are lots of new constructs in v2 that must be supported in our framework in one way or the other, so that there is a simple conversion/adaptation possible. This is a lot of work, especially because I'm basically solo on this. From what I heard, the IMT way of handling data is somewhat 'compatible' to MDF. We have a 100% OO implementation in C#, which means that an object in our framework is usually composed of several fields in MDF, and frequently in a different representation. Also the framework handles other modalities (ACS, MRX, MPS, etc.).

We provide a converter v1->v2 so that it is at least easy to convert files.

@tknopp: Right, but for future breaking changes (if this happens), we would need the opposite, v3->v2. Otherwise, I'm really happy to read that "any update to v2.0 should be non-breaking" is a priority for you :)

No you are misinterpreting this.

Apparently, this is not entirely true ... that's why I started this issue here.

[tknopp]

@tknopp: I don't plan to make it a moving target but for me it does not feel like the MDF specification has reached the maturity to be considered stable. Too many open questions from my point of view, too many optional fields/groups, too focused on lissajous like scanners (see /acquistion/drivefield/ its basically designed for lissajous scanners). Then there is #69 which certainly will be a braking change. Maybe you are currently trying to push MDF too fast because it directly influences your work. I also think that MDF needs alot more input from other instituts.

There is quite some critique in these sentences. We asked several people, several time. in particular the IMT is involved. If you can mobilize more people to participate in discussions, please do so.

I do not object giving v2.0 a little bit more time. Please make concrete proposals how to change the specification. Lets give this time until the end of September. Within this time frame we will try addressing all issues. If there are serious issues we will extend this period. Issues where no consensus is reached will need to be decided by majority.

ping @profix898 @NeumannIMT @hofmannmartin @AvGladiss @MandyA