search

yt-project / yt

Main yt repository
http://yt-project.org
Other
469 stars 280 forks source link

Create data set format mapper #1033

Open yt-fido opened 9 years ago

yt-fido commented 9 years ago

Originally reported by: Joseph Smidt (Bitbucket: jsmidt, GitHub: jsmidt)

I am doing some code comparisons between Enzo and Gadget and it would be nice if I could map an Enzo data set to Gadget and visa versa. And I am sure it would be nice if I could translate data sets between all codes YT can read.

YT already reads in and transforms these data sets. It would be a nice feature to be able to map one data set to another for code comparison purposes.

yt-fido commented 8 years ago

Original comment by Nathan Goldbaum (Bitbucket: ngoldbaum, GitHub: ngoldbaum):

Moving to 3.4 because this is a feature request.

yt-fido commented 9 years ago

Original comment by Joseph Smidt (Bitbucket: jsmidt, GitHub: jsmidt):

If you want to try taking a stab at writing some initial support for writing out datasets in different formats I think that would be complimentary with my plans.

Right now I do intend on at least making a mapper between Enzo and Gadget since I need one. If that effort compliments your larger effort I am happy to help.

yt-fido commented 9 years ago

Original comment by Nathan Goldbaum (Bitbucket: ngoldbaum, GitHub: ngoldbaum):

That's actually been on my plate for a while and is something I want to actively develop once I start my postdoc at NCSA this fall. My idea starts with rethinking the yt frontends in terms of an abstract reader/writer pair that gets dynamically built at runtime based on a declarative data format specification. This means that a frontend author would no longer need to write a custom frontend implementation in python, instead they'd write a YAML file that fully describes their data.

Right now this is all pretty much pie-in-the-sky thinking. I'd love to hear your thoughts though. If you want to try taking a stab at writing some initial support for writing out datasets in different formats I think that would be complimentary with my plans. I'd also be happy to bring you in on design discussions and drafting the YTEP once it gets to that state, if you're up for it.

yt-fido commented 9 years ago

Original comment by Joseph Smidt (Bitbucket: jsmidt, GitHub: jsmidt):

The latter of your questions. The "read an Enzo dataset and write it to disk as a Gadget dataset" or visa versa. This way a true code comparison can be done starting from the exact same initial data.

yt-fido commented 9 years ago

Original comment by Nathan Goldbaum (Bitbucket: ngoldbaum, GitHub: ngoldbaum):

Just to clarify, what exactly do you mean when you say "map one dataset to another"? Some sort of in-memory comparison/differencing tool or do you want to be able to read an Enzo dataset and write it to disk as a Gadget dataset?

yt-fido commented 9 years ago

Original comment by John ZuHone (Bitbucket: jzuhone, GitHub: jzuhone):

Hi @jsmidt,

I want to first say that this is an excellent idea, one that we have talked about amongst the developers on several occasions.

There have been some steps towards something like it, in terms of making our own file format, Grid Data Format (GDF), as well as various things we can do with stream datasets. Unfortunately, the reason why no one has taken something like this up is probably because doing this (and doing it well) is actually a fairly large project that would probably take at least one person's-worth of time for a while.

If you are interested in developing this idea forward, I would recommend writing up a YTEP:

https://bitbucket.org/yt_analysis/ytep/src/41be8d76b2d45f814a51f43ef59e6c5e7bb7b8d8/source/YTEPs/YTEP-template.rst?at=default

This would help us to define the parameters of this idea--what we wanted such a feature to do, how it would do it, what the limitations would be, etc. I'd be happy to help write the document, and I'm sure others would chime in as well.