Very Short Agenda

[tcld]

Here is a short list of things that are currently on my mind. I have at least a couple more days that I could use to work on worldengine, let's see what I can get finished. If there is anything here that you don't like, please tell me and I will focus on the other points first. If there is anything much more pressing, maybe I can look at that instead. But since my background-knowledge is at a flat zero, I cannot check or improve any of the generation algorithms. Optimizing is in the cards, though, thanks to numpy.

• [x] Replace several, maybe all, arrays with numpy arrays.
  • Top on the list: The ocean-map. Used extremely often, maybe a bit of speed can be gained.
  • elevation and watermap would be the next candidates.
• [x] Create a custom, recursive equality operator (in common.py?). Good for numpy and floats, at least.
• [ ] Think about the funny rivers and try to fix them. ( @psi29a called dibs)
• [x] Look at the memory usage of worldengine. Maybe it is just platec, but something needs a lot of RAM. (I didn't find any glaring holes.)
• [x] Add support for a different image format so heightmaps can store more than 256 heights.
• [x] Look at temperatures.py and give it an additional parameter to...do something nice :)

I will edit the list as I go.

[ftomassetti]

Consider that we are going to release soon the new version of plate-tectonics (our platec fork) which is up to 6 times faster. Perhaps it also uses less RAM

[psi29a]

Numpy is the way to go for ALL arrays. It has built in functions like find Max and Min that are super fast, we should be abusing these functions.

[psi29a]

If you don't mind, I want to work on Rivers... I just need to find time. If you beat me too it, fair is fair. ;)

[tcld]

Ok, rivers are yours then.^^ (but please specify which source files you mean, as there are two...kind of)

Abusing numpy is a good idea, but there is a lot more necessary than just replacing arrays. Many portions of the code would have to be rethought to make really good use of numpy. Iterating has to be avoided wherever possible, as can be seen here https://github.com/Mindwerks/worldengine/pull/132 . The very first iteration used a numpy-array, no further changes. That gives a nicer memory footprint (not even that much smaller) but that's it. The second iteration was basically an emulation of the former version that didn't use numpy. That sped things up by only 10%. Only once I went all the way and completely rethought things, things sped up (by a factor of 50!).

Replacing the arrays etc. is a very good first step, though. Since then, whenever new code is written, one at least has an easy option to really use numpy.

@ftomassetti That sounds great, I am looking forward to it! :)

[psi29a]

While I love optimization, getting things right first is more important. No use optimizing rivers that don't look natural. :)

[tcld]

Of course. The first iteration of an algorithm should be functionable, and readable. (And since I don't know anything about geology, I probably won't provide first iterations.^^)

[ftomassetti]

About optimization: I released plate-tectonics and sent a PR to use it in WorldEngine (#137)

[tcld]

Is this what you mentioned above? I'm going to look at it.

[tcld]

I am a little clueless about the memory-usage. Still don't have proper tools for an analysis, but:

When creating a 3000x2000 world (I know that's big) worldengine usually used about 2GB of RAM on my system. There is a short time, though, where that doubles (suspecting that's either when the results are written out or after platec finishes...I don't know).

Having a single data-map that contains 3000x2000x8 Byte that would mean about 50 MB of RAM needed. Having about 15 (?) of those maps means that 1GB of total RAM used would be very reasonable. 2 I wouldn't mind, but 4 is a lot. Maybe it is perfectly fine and I just don't understand it yet, but this justifies a bit of an investigation. Do you have any ideas?

On a side-note: I did post this at some point:

         95833445 function calls (95705636 primitive calls) in 92.104 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  4014080   19.543    0.000   19.543    0.000 worldengine/common.py:97(anti_alias_point)
  1262482   18.308    0.000   27.963    0.000 worldengine/world.py:375(tiles_around)
     1150   14.349    0.012   14.349    0.012 {built-in method step} -- this refers to platecs step()-method
 56451705   10.417    0.000   10.417    0.000 worldengine/world.py:341(is_land)
        1    2.312    2.312    4.155    4.155 worldengine/simulations/erosion.py:136(river_sources)

And I did notice that a deep copy is forced in _anti_alias_step() (which calls anti_alias_point for every pixel on the map...which in this case means it itself is called 40 times...40*50MB is roughly 2GB <- I miscalculated, it's a lot less.). I am not sure that deep copy is entirely necessary.

[tcld]

Now that @psi29a is "MIA", rivers will stay pretty broken for a little longer. Aside from that my list is mostly done, even though some of the changes haven't been merged yet.

[psi29a]

If you want to take a stab, go ahead. I suspect that rivers are growing up from the sea, but I could be wrong. It should happen as I mentioned in another thread...

[tcld]

I think I prefer leaving the rivers to you. They must have been flowing uphill for a while, and nobody seemed to notice. Let's just hope that continues.^^

[ftomassetti]

I think people noticed... I got a few comments on Reddit about it. I think it is something we should fix. Perhaps we could just get the code from WorldSynth. I think erosion and rivers were strongly connected in WorldSynth, am I wrong?

[ftomassetti]

All the points but rivers seem to have been addressed. Great work here! Should we close this issue and discuss rivers in #119 ?

[tcld]

Ok.

[psi29a]

Super! :D