Make the protobuf format the default one

[ftomassetti]

When generating a world by default it is saved in pickle format. The user has the possibility to save it in protobuf format if he chooses an option. I think it should be the other way around or we could also drop the pickle format and support exclusively the protobuf format.

[psi29a]

Problem with python3, so I wouldn't make this the default.

[ftomassetti]

Mmm, in the other end pickle is limiting (not much support for other languages, not particularly efficient). Should we think about other formats like HDF5?

[psi29a]

For my world data, I used PyTables which used HDF5 to store the data. PyTables gives you a nice abstraction to work with. Bonus for being actively developed, widely used and supports Python2 and Python3. I know it works on windows because I packaged WorldSynth with it.

https://pypi.python.org/pypi/tables

[psi29a]

"PyTables is a package for managing hierarchical datasets and designed to efficently cope with extremely large amounts of data. PyTables is built on top of the HDF5 library and the NumPy package and features an object-oriented interface that, combined with C-code generated from Cython sources, makes of it a fast, yet extremely easy to use tool for interactively save and retrieve large amounts of data."

[psi29a]

This also makes it easy to hand world files off to other developers because they can read it in via PyTables or even HDF5 to just have access to the data-sets themselves.

[ftomassetti]

Now the protobuf is working of Python 3 should we make it the default format? I am also in favor of adopring HDF5, it just would require some more work while we can make protobuf the default very easily :)

[ftomassetti]

Protobuf do not seem loved enough to be the default format, closing

[psi29a]

who said this? Protobuf is loved enough to be the default format from my POV.

[ftomassetti]

Sorry, my misunderstaning. However it would probably make sense to wait that Protobuf 3 is released for all the platforms.

[psi29a]

Isn't Protobuf3 what is being pushed in a PR by @tcld ?

[psi29a]

It's already here: https://github.com/Mindwerks/worldengine/blob/master/requirements.txt#L6

:D

[tcld]

It's in there because otherwise there wouldn't be any format that is equally supported by Python 2 and 3. And it works for now, but @ftomassetti said that there were some problems with Java (which requires a newer version?). Could it be that pip is just not using the latest version?

I do think that protobuf is better than pickle, so it kind of is a good candidate for a default choice as long as there are only two options. ;P (I hope that logic isn't too broken.^^)

[psi29a]

Protobuf should be default, then we can wash our hands of pickle. HDF5 can just be another supported method until protobuf works with java.

[tcld]

That sounds like a decision to me. (So did the other five times before this too, though.^^)

Is it just about making protobuf the default and leaving pickle in? Or should pickle be completely removed? The first one I could do myself, the latter one I don't really know how to correctly approach.

[ftomassetti]

The problem with Protobuf 3 is that it has not yet properly released: alpha 3 is availlable for Python, while beta is available for Java but I did not manage to make them work together. What I like about protobuf is that it does not require you to install any system library (while HDF5 apparently requires that).

I would switch to Protobuf 3 as the default, remove pickle and introduce the optional support to HDF5 after we release this version (we have ~200 commits and endless features which are not available to people who are not willing/able to do a git clone). I would do that right now if we Protobuf 3 was properly released.

[psi29a]

I'm not scared of alpha/beta, the results speak for themselves. The only problem is java support.

I want protobuf3 as default, in this release with HDF5 as optional.

[ftomassetti]

Yes but the scaring thing is that they state they do not want to guarantee compatibility between alpha, beta and the final version 3 release.

However we can make protobuf the default format (that would be fairly easy). Add HDF5 at this stage would require convincing AppVeyor.

We could drop pickle later if it is fine for you.

[tcld]

That sounds like a decision to me.

[psi29a]

I'm fine with breaking compatibility, we're doing that anyway in dropping pickle support.

We're not exactly in 'production' ready state ourselves. ;)

I think we can afford to develop fast against moving targets until people start using us more and we are forced to be backwards compatible.

The end-game is protobuf3.

HDF5 can be optional, like GDAL. This will help appveyor.

[ftomassetti]

Ok, so marking as part of milestone 0.19.0 this one (PR sent) and the one for HDF5

[ftomassetti]

This should be done I think, closing.