search

Mindwerks / worldengine

World generator using simulation of plates, rain shadow, erosion, etc.
MIT License
981 stars 127 forks source link

Preparing for new release #184

Closed ftomassetti closed 6 years ago

ftomassetti commented 8 years ago

Version number should be updated:

Am I missing something?

rbb commented 8 years ago

Was there something special that had to happen for an OS X (Mac) build/release? I know you guys had me test something at one point.

ftomassetti commented 8 years ago

I am not sure what we need to do for OS X but we could use some help I guess :)

rbb commented 8 years ago

I have time to do more testing, but I'd need a little bit of instruction, like last time.

I'm not really aware of the potential pitfalls - especially how my MacPorts install of python might mask lack of package installs that might happen on a computer running the stock OS X version of python. In other words, I've installed a lot of extra python modules, and it seems like the worst case we need to test against is a user who is running stock python on OS X. I don't know how to make sure that only the stock python is called, to verify that all the other necessary modules are included when installing WorldEngine.

psi29a commented 8 years ago

I use brew in my OSX VirtualBox instance.

Build procedures are in the readme.

ftomassetti commented 8 years ago

And now release 0.19.0 has been tagged!

tcld commented 8 years ago

So basically the release is done? :) I guess one of you then has the honor of creating all the binary packages.^^

Is it a bad idea to commit code in the meantime? I tried hard not to start anything new, but I wasn't entirely successful.

ftomassetti commented 8 years ago

The code for release is tagged, we will now create the binary packages but any code committed in the meantime will not affect this. The work on release 0.20.0 or 1.0.0 is officially started :D

rbb commented 8 years ago

So, I've had a few spare cycles to try it on my Mac...

I didn't want to sudo pip install worldengine, because I didn't want it to mess with my MacPorts python install, which I use for lots of other things. So I went through the things that pip tried to install (without sudo), and made sure everything was installed. Then, went to my clone of the worldengine repo and tried python setup.py build, which didn't complain. Moving to the build/ directory and then trying to get it to run, I keep getting failures on some of the protobuf stuff. I've verified a couple of times that I've got protobuf installed.

>python2.7 worldengine -n myworld -g out
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/russell/projects/worldengine/build/lib/worldengine/__main__.py", line 5, in <module>
    from cli import main
  File "worldengine/cli/main.py", line 5, in <module>
    import worldengine.generation as geo
  File "/Users/russell/projects/worldengine/build/lib/worldengine/../worldengine/generation.py", line 2, in <module>
    from worldengine.world import Step
  File "/Users/russell/projects/worldengine/build/lib/worldengine/../worldengine/world.py", line 14, in <module>
    import worldengine.protobuf.World_pb2 as Protobuf
  File "/Users/russell/projects/worldengine/build/lib/worldengine/../worldengine/protobuf/World_pb2.py", line 20, in <module>
    DoubleMatrix\x12+\n\x08rivermap\x18# \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12)\n\x06icecap\x18$ \x01(\x0b\x32\x19.World.World.DoubleMatrix\x1a\x1a\n\tDoubleRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x01\x1a\x1b\n\nBooleanRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x08\x1a\x1b\n\nIntegerRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x05\x1a\x18\n\x07\x42yteRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x05\x1a\x34\n\x0c\x44oubleMatrix\x12$\n\x04rows\x18\x01 \x03(\x0b\x32\x16.World.World.DoubleRow\x1a\x36\n\rBooleanMatrix\x12%\n\x04rows\x18\x01 \x03(\x0b\x32\x17.World.World.BooleanRow\x1a\x36\n\rIntegerMatrix\x12%\n\x04rows\x18\x01 \x03(\x0b\x32\x17.World.World.IntegerRow\x1a,\n\x0e\x44oubleQuantile\x12\x0b\n\x03key\x18\x01 \x02(\x05\x12\r\n\x05value\x18\x02 \x02(\x01\x1aq\n\x19\x44oubleMatrixWithQuantiles\x12.\n\tquantiles\x18\x01 \x03(\x0b\x32\x1b.World.World.DoubleQuantile\x12$\n\x04rows\x18\x02 \x03(\x0b\x32\x16.World.World.DoubleRow\x1aS\n\x0eGenerationData\x12\x0c\n\x04seed\x18\x01 \x01(\x05\x12\x10\n\x08n_plates\x18\x02 \x01(\x05\x12\x13\n\x0bocean_level\x18\x03 \x01(\x02\x12\x0c\n\x04step\x18\x04 \x01(\t'
TypeError: __init__() got an unexpected keyword argument 'syntax'

Here is the code its referencing:

 Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: World.proto

from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
# from google.protobuf import descriptor_pb2
# @@protoc_insertion_point(imports)

_sym_db = _symbol_database.Default()

DESCRIPTOR = _descriptor.FileDescriptor(
  name='World.proto',
  package='World',
  syntax='proto2',
  serialized_pb=b'\n\x0bWorld.proto\x12\x05World\"\xf6\r\n\x05World\x12\x17\n\x0fworldengine_tag\x18\x01 \x02(\x05\x12\x1b\n\x13worldengine_version\x18\x02 \x02(\x05\x12\x0c\n\x04name\x18\x03 \x02(\t\x12\r\n\x05width\x18\x04 \x02(\x05\x12\x0e\n\x06height\x18\x05 \x02(\x05\x12\x30\n\rheightMapData\x18\x06 \x02(\x0b\x32\x19.World.World.DoubleMatrix\x12\x17\n\x0fheightMapTh_sea\x18\x07 \x02(\x01\x12\x19\n\x11heightMapTh_plain\x18\x08 \x02(\x01\x12\x18\n\x10heightMapTh_hill\x18\t \x02(\x01\x12*\n\x06plates\x18\n \x02(\x0b\x32\x1a.World.World.IntegerMatrix\x12)\n\x05ocean\x18\x0b \x02(\x0b\x32\x1a.World.World.BooleanMatrix\x12,\n\tsea_depth\x18\x0c \x02(\x0b\x32\x19.World.World.DoubleMatrix\x12)\n\x05\x62iome\x18\r \x01(\x0b\x32\x1a.World.World.IntegerMatrix\x12\x38\n\x08humidity\x18\x0e \x01(\x0b\x32&.World.World.DoubleMatrixWithQuantiles\x12-\n\nirrigation\x18\x0f \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12\x33\n\x10permeabilityData\x18\x10 \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12\x18\n\x10permeability_low\x18\x11 \x01(\x01\x12\x18\n\x10permeability_med\x18\x12 \x01(\x01\x12/\n\x0cwatermapData\x18\x13 \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12\x16\n\x0ewatermap_creek\x18\x14 \x01(\x01\x12\x16\n\x0ewatermap_river\x18\x15 \x01(\x01\x12\x1a\n\x12watermap_mainriver\x18\x16 \x01(\x01\x12\x34\n\x11precipitationData\x18\x17 \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12\x19\n\x11precipitation_low\x18\x18 \x01(\x01\x12\x19\n\x11precipitation_med\x18\x19 \x01(\x01\x12\x32\n\x0ftemperatureData\x18\x1a \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12\x19\n\x11temperature_polar\x18\x1b \x01(\x01\x12\x1a\n\x12temperature_alpine\x18\x1c \x01(\x01\x12\x1a\n\x12temperature_boreal\x18\x1d \x01(\x01\x12\x18\n\x10temperature_cool\x18\x1e \x01(\x01\x12\x18\n\x10temperature_warm\x18\x1f \x01(\x01\x12\x1f\n\x17temperature_subtropical\x18  \x01(\x01\x12\x33\n\x0egenerationData\x18! \x01(\x0b\x32\x1b.World.World.GenerationData\x12*\n\x07lakemap\x18\" \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12+\n\x08rivermap\x18# \x01(\x0b\x32\x19.World.World.DoubleMatrix\x12)\n\x06icecap\x18$ \x01(\x0b\x32\x19.World.World.DoubleMatrix\x1a\x1a\n\tDoubleRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x01\x1a\x1b\n\nBooleanRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x08\x1a\x1b\n\nIntegerRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x05\x1a\x18\n\x07\x42yteRow\x12\r\n\x05\x63\x65lls\x18\x01 \x03(\x05\x1a\x34\n\x0c\x44oubleMatrix\x12$\n\x04rows\x18\x01 \x03(\x0b\x32\x16.World.World.DoubleRow\x1a\x36\n\rBooleanMatrix\x12%\n\x04rows\x18\x01 \x03(\x0b\x32\x17.World.World.BooleanRow\x1a\x36\n\rIntegerMatrix\x12%\n\x04rows\x18\x01 \x03(\x0b\x32\x17.World.World.IntegerRow\x1a,\n\x0e\x44oubleQuantile\x12\x0b\n\x03key\x18\x01 \x02(\x05\x12\r\n\x05value\x18\x02 \x02(\x01\x1aq\n\x19\x44oubleMatrixWithQuantiles\x12.\n\tquantiles\x18\x01 \x03(\x0b\x32\x1b.World.World.DoubleQuantile\x12$\n\x04rows\x18\x02 \x03(\x0b\x32\x16.World.World.DoubleRow\x1aS\n\x0eGenerationData\x12\x0c\n\x04seed\x18\x01 \x01(\x05\x12\x10\n\x08n_plates\x18\x02 \x01(\x05\x12\x13\n\x0bocean_level\x18\x03 \x01(\x02\x12\x0c\n\x04step\x18\x04 \x01(\t'
)
_sym_db.RegisterFileDescriptor(DESCRIPTOR)

Any ideas for what to look at next? I'm sure I'm screwing up the install somehow.

tcld commented 8 years ago

Is there no virtualenv on OSX? It took me an eternity to find out that installing WorldEngine under Linux only worked inside a virtual environment.

The error message hints at you using Protobuf 2. The syntax-keyword was introduced in Protobuf 3, I believe.

rbb commented 8 years ago

Thanks much for the tips!

psi29a commented 8 years ago

https://github.com/Mindwerks/worldengine#osx <-- using brew worked for me but there were problems with protobuf and I had to manually copy files around.

tcld commented 8 years ago

How is the release going? Is anybody creating binary packages? I have Linux and a Windows inside a virtual machine, with some instructions I might be able to help.

psi29a commented 8 years ago

No packages are created until we are tagged for release.

ftomassetti commented 8 years ago

We have the tag for release and the source packages on Pypi but no binary packages

ftomassetti commented 8 years ago

The tag is: v0.19.0

psi29a commented 8 years ago

yay, a release! @ftomassetti I thought you wanted me to review (and possibly merge) your PR. We'll save that for 0.20 ?

ftomassetti commented 8 years ago

Sorry for the misunderstanding, I did not want to sneak a release behind your back: I thought the plan was to tagging the release as soon as all the stuff marked as 0.19.0 was in.

Yes, I think it is better to merge such big PRs at the beginning of a new development cycle.

psi29a commented 8 years ago

Roger that, I'll try to get around to making binary packages when I have time. It may or may not be with GDAL. If someone else wants to follow my guide and get everything included, by all means, please do.

I don't want to hold up the release because of real-life getting in the way on my side. :)

ftomassetti commented 8 years ago

Well, the release has been tagged, binary packages can arrive when real-life permits that :)

psi29a commented 8 years ago

OpenMW is also prepping for a big release (last release was in June... major switch from Ogre3D to OSG. ;)

tcld commented 8 years ago

I assume moving the tag to the current state isn't possible? We would be including the latest manual that way.

@psi29a What guide is there to follow for a release?

ftomassetti commented 8 years ago

The manual is available online: if we start moving the tag we will keep moving it :)

ftomassetti commented 8 years ago

But we can release v0.19.1 anytime you want. Release early, release often!

psi29a commented 8 years ago

Created linux binary package for x86_64 and uploaded it here: https://github.com/Mindwerks/worldengine/releases/tag/untagged-3bd72a11fa787656f531

this is a draft, it is possible that some people will not be able to see this.

tcld commented 8 years ago

Ok, I am one of those people.

psi29a commented 8 years ago

OK, here is it on my dropbox... not for public consumption: https://dl.dropboxusercontent.com/u/396161/worldengine-0.19.0-Linux_x86_64.tar.bz2

When we're ready, we'll use github for releases that can be posted all over reddit and etc.

It is just a dirtectory that can be extracted, in that directory is the worldengine binary and all the supporting libs. I managed to cram gdal and hdf5 in.

ftomassetti commented 8 years ago

Ah ah I just put it on my dropbox: https://www.dropbox.com/s/as4p4x9i3wzxals/worldengine-0.19.0-Linux_x86_64.tar.bz2?dl=0

tcld commented 8 years ago

One is enough, but thanks.^^ A first run on my machine worked fine. I guess I'll try to create an ancient map from a saved world, too - other than that should I do any specific testing? Or is it enough that nothing crashes?

The release has grown quite a bit, 0.18 was at ~18 MB. libicudata.so.55 and libopenblas.so.0 are huge files - what do they do?

tcld commented 8 years ago

Loading the world and creating an ancient map worked fine, too.

psi29a commented 8 years ago

likely from gdal ;) gdal is HUGE

psi29a commented 8 years ago

Added OSX as well, please test: https://dl.dropboxusercontent.com/u/396161/worldengine-0.19.0-OSX-x86_64.tar.bz2

Also attached to this release, so please do not share. I'll delete the dropbox files when we publish the release.

tcld commented 8 years ago

Who can even test this? You might want to use @...

psi29a commented 8 years ago

@ftomassetti apparently as a Yosemite laptop. I have Mountain Lion in a virtualbox. The more than can test this the better of course. ;)

I just tested, apparently GDAL is included. I'm surprised at the size, it came in at 19.7 MiB as apposed to the Linux build. I suspect that the Linux build has more GDAL plugins (and the kitchen sink). ;)

ftomassetti commented 8 years ago

I could generate a world correctly but I got an error running ./worldengine -x 100 -y 100 --hdf5 --sat

Traceback (most recent call last):
  File "<string>", line 619, in <module>
  File "<string>", line 550, in main
  File "<string>", line 92, in draw_satellite_map
  File "/Users/psi29a/Workspace/WorldEngine/worldengine/worldengine/draw.py", line 816, in draw_satellite_on_file
  File "/Users/psi29a/Workspace/WorldEngine/worldengine/worldengine/draw.py", line 375, in draw_satellite
  File "/Users/psi29a/Workspace/WorldEngine/worldengine/worldengine/draw.py", line 250, in get_normalized_elevation_array
TypeError: astype() takes no keyword arguments
main returned -1

It can be replicated with: ./worldengine -x 100 -y 100 --sat -s 28753

ftomassetti commented 8 years ago

Same command run fine on linux. I am not sure it is due to the way we packaged the project on Mac.

ftomassetti commented 8 years ago

Running on mac from master it runs fine. That is strange

tcld commented 8 years ago

That's here, and astype() takes a keyword argument since numpy version 1.7. 1.6 didn't have that argument, though. Maybe the bundled numpy is outdated?

ftomassetti commented 8 years ago

chapeau @tlcd, good catch!

psi29a commented 8 years ago

Nice catch, I'll take a look to see what version is installed.

tcld commented 8 years ago

Removing the dtype-keyword from astype() would be an option, too. Although if there really is an old numpy-version in use, other problems could crop up later.

worldengine\draw.py
    Line 250:    c = numpy.rint(c).astype(dtype=numpy.int32)  # proper rounding
    Line 538:    colors = numpy.rint(colors).astype(dtype=numpy.int32)  # proper rounding
    Line 588:    colors = numpy.rint(colors).astype(dtype=numpy.int32)  # proper rounding
worldengine\image_io.py
    Line 130:    _array = numpy.rint(_array).astype(dtype=PNGWriter.get_dtype(channel_bitdepth))  # proper rounding
psi29a commented 8 years ago

OK... according to pip freeze on my OSX instance, the version is 1.9.2

however, running what @ftomassetti above does indeed trigger the error. Perhaps this bug is in the OSX version of numpy.

ftomassetti commented 8 years ago

I ran the command on the mac using master (not the binary package) and it seems to run fine

psi29a commented 8 years ago

ok, something is up... fired up ipython: from numpy import version; version.full_version 1.6.1

looks like something is being used in the place...

psi29a commented 8 years ago

Fixed (I hope): https://dl.dropboxusercontent.com/u/396161/worldengine-0.19.0-OSX-x86_64.tar.bz2 This is a 30MiB file... bigger than the 19MiB from the one above. I think more plugins are included this time around. ;)

psi29a commented 8 years ago

Any testers?

ftomassetti commented 8 years ago

Run fine for me!

psi29a commented 8 years ago

Super! The only think left is try our hand at creating windows packages.

tcld commented 8 years ago

0.19 still isn't out yet. What's necessary to create a Windows package?

psi29a commented 8 years ago

Just follow my guidelines and you should be good. ;)

I just need 1) time and 2) find a windows VM to do this in.

tcld commented 8 years ago

What guidelines?