Problem with Rivers

[esampson]

No one seems to have brought this up before but there seems to be a problem with river generation. For some reason they have an extremely strong tendency to run to the northwest. Here is a river map generated from the latest build and I can't see a single river that runs outside of the range of 270-360 degrees north. This also doesn't seem to be a new occurrence. It is something I noticed a while back but I just didn't get around to filing an issue on it.

[psi29a]

That is because the current algorithm uses wind and rain maps, the water collects above sea-level and is blown in just one direction. As precipitation falls, then further along in that direction is less rain to fall.

In my original code, the wind direction was random... so I'm going to assume that it is the same as before. This means that there will be areas with no rain at all which contributes to biomes.

You can verify this by looking at your biome map, if there are deserts or similar biomes in the area, then it is likely that there isn't any rainfall and rivers there. Rivers can begin in other biomes and flow into deserts however... just like the Nile.

We have ideas on how to create better wind directions and rain, it exists in another issue/ticket.

[esampson]

Are rivers really that affected by wind in the real world?

[psi29a]

Interesting that you have asked that, if there is no wind to carry precipitation, then you'll never get rain or snow in some areas. That means no snow-pack that melts, no erosion through run-off, no collection into into streams that later collect into other streams than into rivers...

Another effect is rain-shadows, that when wind hits mountains that precipitation falls heavily on that side of the mountain. The result is that on the other side gets little to no rain-fall as a result.

The only thing you'll have are brons or sources, that bubble up from out the ground... that is based on aquifers and underground rivers.

There is a bit of overlap here, but these two things can be considered separate.

[esampson]

Ok, I realize that the direction of the wind is going to affect rainfall, but it doesn't seem like it should have that strong an effect on the rivers themselves. Once a river has formed it always has to run downhill, but in the example above (seed 28427) I can find at least one spot where it seems to clearly be running up hill (coordinates 338, 172) by laying the river map over the grayscale heightmap.

[psi29a]

I didn't mean to imply that the wind pushes rivers anywhere... it only effects where they begin. From there, they flow downhill... if there is enough pressure and it can't find anywhere to flow, it will use A* to find the next lower point within a small area.

It was my intention that if the next lower point couldn't be found, it would then fill up the area to form a lake.

What you are showing me here is unfortunate, but if you can give me the seed number I can try to reproduce it and see if I can refine how rivers flow.

[tcld]

Just to verify the issue, here is a map that contains a lot of rivers: seed: 2047, dimensions: 2048x2048, version/last commit: d3593f61c50c5c0acaa64fc8f9812c7e3dcea745

So if my understanding of this is correct, the lakes that should fill up instead form "crazy" rivers? Can you maybe point me to a source-file where this most likely is triggered? I would want to take a look at it.

[psi29a]

That's just bizarre, I've never seen this in WorldSynth. It looks like all the rivers are flowing downward from bottom right to top left.

Can someone verify that the heightmap isn't sloping downward in that direction?

[esampson]

I have a comment above where I show the river definitely running up hill over a short duration. It seems hard to believe that in tcld's example that the slope is consistently running downhill in all of those cases and if it is that is probably still an indication of some kind of problem (although it could be a problem with height generation instead of the rivers themselves)

It would be easier to verify that there is an actual problem with water running up or down if the grayscale map wasn't limited to 256 shades of gray. Is there any possibility of getting 16 bit grayscale as an option for the height map?

[tcld]

I didn't know how to overlay the images, but with the rivers behaving so obviously this might be enough of a hint:

@esampson : Isn't there an option for that? I thought I saw one... EDIT: Might have misinterpreted that. There is mention of an option for 8,16 or 32 bit exports in main.py but I don't see if and where it is used.

EDIT2: Maybe the elevation map is more useful? Sorry for the super-large images.

EDIT3: There is a clear bump in the middle of the large continent, and it looks like all rivers go over it. Doesn't look like a coincidence.

[esampson]

There is mention, but I believe that is bits per pixel for the colored images. I ran a test on the 16 bit export at one point and the grayscale heightmap that was produced was only 8 bit (and I'm not sure there is a 32 bit grayscale format, so I doubt that option could be applied to the heightmap).

[tcld]

Either way, 256 different values for the differences in elevation of a whole world is pretty rough. That would correspond to something like 80 meters per value if one were to display earth that way.

[esampson]

Around 70 meters per level, if my math is right. 16 bit grayscale on the other hand is about 30 centimeters per level.

There's definitely some uphill flow occurring in that example. The easiest way I know to composite the two images is through GIMP. Put the river on a layer above the grayscale, create a mask, select the black and white areas and fill the mask in those location (so it only leaves the blue rivers).

Here's a good example showing problems:

[tcld]

Is that hydrology.py? That would be the next on my list anyway. If this is caused by some other part of the program, however, I would need a hint.

[esampson]

I think it is, but I'm not positive on that.

[tcld]

Can somebody give a short explanation of what I see on the precipitation-map? Is it about the possibility of rain?

I'm trying to understand this:

if True and world.precipitation['data'][y][x] > 0:

[ftomassetti]

Precipitation map should represent the yearly amount of precipitation in a cell

I suspect these poorly drawn rivers derived from some early experiments before me and Bret merged our projects. If this is the code from lands and worldsynth had rivers we could get that code and throw out this one I think. Federico On Oct 15, 2015 10:44 PM, "tcld" notifications@github.com wrote:

Can somebody give a short explanation of what I see on the precipitation-map? Is it about the possibility of rain?

— Reply to this email directly or view it on GitHub https://github.com/Mindwerks/worldengine/issues/119#issuecomment-148514902 .

[tcld]

@ftomassetti Thank you very much.

[tcld]

I think this could be the cause (from hydrology.py line 20 and 27):

pos_elev = world.elevation['data'][y][x] + _watermap[y][x]

This says that the watermap increases the elevation. But the watermap, as far as I understand, stores how much rain "drifts" onto a tile. Granted, that could be seen as an elevation increase - but I don't see any normalization in the code. Is that double-checked? Could it be that a single drop of rain causes a mountain of water to grow which then screws with the river-generation?

[tcld]

More: Line 365 of world.py, _def tilesaround(self, pos, radius=1, predicate=None): That function returns the tiles around, as a simple (and impossible) example, (0,0). The order of those tiles will be: (-1,-1), (-1, 0), (-1, 1), (0, -1), (0, 0), (0, 1), (1, -1), (1, 0), (1, 1) That last pair of coordinates looks like it is the south-east tile.

I guess the assumption in my last message was right:

Imagine these tiles, Dwarf Fortress-style (higher number, more elevation):

000
022
023

If one were to iterate from top-left to bottom-right and added a tremendous amount of water to the bottom-right tile, this could happen (added numbers refer to water):

0+8 0+8 0+8
0+8 2+24 2+24
0+8 2+24 3+92

Now the last tile that is iterated over has the highest chance of accumulating water and the water just went upwards. (Just imagine a tile with height 12 south-east of that tile, the water would still "flow down" in that direction.) This is a very rough example, currently not fully backed up by understanding the code (there is a multiplication by 4 for elevation differences, that's why I chose the numbers). But I don't think it is a coincidence that the order of the coordinates returned by tiles_around() happens to be such that the south-east tile is always last to be looked at.

[tcld]

No matter if I am right or not, I will look at hydrology.py tomorrow and hopefully find a fix for this. I don't expect a massive speed-up but the function looks like it could make good use of numpy.

Some questions: There is a hard-coded value of up to 20000 droplets coming down on a world. That sounds like it wants to be replaced by something like width*height*random.uniform(0.7, 1.3). Is that reasonable or stupid?

How much elevation-height should a droplet add to a pixel of the world? I have absolutely no measure for this since I don't even know typical elevation changes. That number could depend on the biome, or be a bit random, or... well, any kind of input is highly appreciated. @psi29a , you wrote the original method, right? Do I have to stick to some scientific framework when modifying it?

Is it safe to add changes to a branch that another branch already covered in a slightly different way? I need to touch the World.operator==() again to make it fully recursive for future numpy-replacements. But I think I already made changes to it in that irrigation-rework.

Sorry for all the questions. And good night, I guess. :)

[ftomassetti]

This was inspired by some erosion algorithm but you are right, it is just wrong. The point is that we do not have units of measure, perhaps we should have watermap/1000 + elevation

Federico

On Thu, Oct 15, 2015 at 11:13 PM, tcld notifications@github.com wrote:

I think this could be the cause:

pos_elev = world.elevation['data'][y][x] + _watermap[y][x]

This says that the watermap increases the elevation. But the watermap, as far as I understand, stores how much rain "drifts" onto a tile. Granted, that could be seen as an elevation increase - but I don't see any normalization in the code. Is that double-checked? Could it be that a single drop of rain causes a mount of water to grow which then screws with the river-generation?

— Reply to this email directly or view it on GitHub https://github.com/Mindwerks/worldengine/issues/119#issuecomment-148522918 .

Website at http://tomassetti.me http://www.federico-tomassetti.it GitHub https://github.com/ftomassetti

[ftomassetti]

On Fri, Oct 16, 2015 at 12:19 AM, tcld notifications@github.com wrote:

No matter if I am right or not, I will look at hydrology.py tomorrow and hopefully find a fix for this. I don't expect a massive speed-up but the function looks like it could make good use of numpy.

Some questions: There is a hard-coded value of up to 20000 droplets coming down on a world. That sounds like it wants to be replaced by something like width_height_random.uniform(0.7, 1.3). Is that reasonable or stupid?

You probably want to have it vary with the size of the world, I does not necessarily need to be random I think.

How much elevation-height should a droplet add to a pixel of the world? I have absolutely no measure for this since I don't even know typical elevation changes. That number could depend on the biome, or be a bit random, or... well, any kind of input is highly appreciated. @psi29a https://github.com/psi29a , you wrote the original method, right? Do I have to stick to some scientific framework when modifying it?

Is it safe to add changes to a branch that another branch already covered in a slightly different way? I need to touch the World.operator==() again to make it fully recursive for future numpy-replacements. But I think I already made changes to it in that irrigation-rework.

Sorry for all the questions. And good night, I guess. :)

They are very welcome :)

— Reply to this email directly or view it on GitHub https://github.com/Mindwerks/worldengine/issues/119#issuecomment-148539413 .

Website at http://tomassetti.me http://www.federico-tomassetti.it GitHub https://github.com/ftomassetti

[tcld]

Units of measure... I have been thinking about this. A "drop" falls onto a tile and spreads over all surrounding tiles, depending on how much volume there is available on tiles that are lower. If that is filled up, higher tiles within a certain height difference will be taken into account.

As I understand it, hydrology.py creates rivers from rain. (Those kinds of rivers are usually seasonal, but ok.) But there is more talk of rivers in erosion.py - is there another mechanism at work to create rivers? Am I maybe misunderstand this and the watermap is used for something else entirely (it doesn't seem to be used by erosion.py).

Are the strange rivers in the image above maybe failed lakes?

[psi29a]

I had two processes going on. One was natural water-based erosion, based on the precipitation model that I had in WorldSynth[1]. Where the rain fell it would 'roll' down to the next lower elevation. If none was found, it stopped and we moved on to the next location. If one was found, it would erode a small % of elevation... flowing down to the next lowest elevation point beside it.

At a certain point when this is all happening, the amount of water collected into these valleys would result in rivers on a map. My "river" map was is basically a water map. The highest concentration of above ground sweet-water would be the clue lines resembling rivers.

I haven't looked at how this translated into WorldEngine but I would suspect that it works the same way.

The way I implemented was based on research I did. I had documentation in my code[2].

https://github.com/psi29a/worldsynth https://github.com/psi29a/worldsynth/blob/master/library/rivers.py

[tcld]

This is very interesting information! (And now I have so many more reasons to not touch worldengines rivers but let you do it instead. ^^)

For the current algorithm: Maybe the water-height should not be added to the elevation. That seems to be only useful for lakes, and as far as I can see those don't exist yet.

[ftomassetti]

yes but without the increase in height you would risk to have many dead-ends. Perhaps you can play with a reduction factor, for example dividing the contribution of the watermap by a large factor. For me the most important factor is that maps look good: so I would try a few values to see if I get good rivers or not

[psi29a]

Unless I'm not understanding what dead-ends mean... they form naturally and become lakes that may or may not swell up enough to lead to lower elevation points.

[ftomassetti]

I think they do so if the level of water is considered: if we just consider the elevation of the terrain beneath, without adding the contribution given by water it is easy to end up in a pit.

Suppose you have a cell ("the pit") surrounded by a 8 slightly higher cells. Water which arrives in the pit does not leave the pit because the pit has lower elevation of the sorrounding cells. If we consider also a contribution from water to the elevation than as more water flows into the pit, the level of water in the pit grows until it can compensate the difference in elevation with the sorrounding cells and eventually a small lake is formed in the pit, with a river leaving the pit.

On Fri, Oct 16, 2015 at 10:11 AM, Bret Curtis notifications@github.com wrote:

Unless I'm not understanding what dead-ends mean... they form naturally and become lakes that may or may not swell up enough to lead to lower elevation points.

— Reply to this email directly or view it on GitHub https://github.com/Mindwerks/worldengine/issues/119#issuecomment-148647173 .

Website at http://tomassetti.me http://www.federico-tomassetti.it GitHub https://github.com/ftomassetti

[tcld]

That is true, of course. Right now either the volume of the water is too large or it doesn't always spread around evenly (behaving more like snow). Some kind of refinement would be good.

EDIT: Yes, it is most likely a combination of both. The rivers seem to flow upwards (too much water) and they keep a very dominant direction while doing it (unevenly spread water).

[ftomassetti]

I think it is way too large, so large that it overcomes big differences in height. It could also not spread around evenly.

On Fri, Oct 16, 2015 at 10:19 AM, tcld notifications@github.com wrote:

That is true, of course. Right now either the volume of the water is too large or it doesn't always spread around evenly (behaving more like snow). Some kind of refinement would be good.

— Reply to this email directly or view it on GitHub https://github.com/Mindwerks/worldengine/issues/119#issuecomment-148649340 .

Website at http://tomassetti.me http://www.federico-tomassetti.it GitHub https://github.com/ftomassetti

[psi29a]

Don't forget A*... this is my last ditch effort for rivers ending in a pit. It looks around within a certain radius for lowest points then finds the path of least resistance to that point.

This is likely a contributing factor. This likely worked well when I was generating high-maps based on diamond-square and perlin-noise... I think the maps we get are significantly different than the ones I was working with.

[ftomassetti]

I THINK the rivers were better in WorldSynth...

[ftomassetti]

Do you want to know something... peculiar? We have the code from WorldSynth and we use it to generate the riversmap. It is just then when drawing rivers we do not use that. I guess we do not use that because WorldSynth rivers do not necessarily end in the sea so we were trying to generate something "cool" on the fly. Big fail.

Drawing rivers "on the fly":

Drawing rivers using the rivermap:

Note that to use that you can just replace draw_rivers_on_image with:

    for y in range(world.height):
        for x in range(world.width):
            if world.is_land((x, y)) and world.river_map[y, x] > 0.0:
                for dx in range(factor):
                    for dy in range(factor):
                        target.set_pixel(x * factor + dx, y * factor + dy, (0, 0, 128, 255))

Perhaps we could restart from the rivermap created by the WorldSynth code but somehow make erosion stronger so that rivers reach the sea

One thing abour rivers is that I would love to have main rivers, rivers and creeks: on the map you can see just the larger streams but the world should have also the smaller ones.

[tcld]

This looks quite impressive. WorldSynth's rivers sure look a lot more realistic (and it seems they follow the heightmap). Is it necessarily erosion that is causing problems here and not maybe lack of water or even lack of lake-formation? If a river just ended in a mountainous lake, I would consider that alright. But it seems that worldengine cannot (?) create lakes currently.

[ftomassetti]

We actually have also a lake map :)

[tcld]

You are right, and it is created as part of the erosion-process, too. erosion.py is quite daunting and hides a lot of potential we should totally leverage.

What exactly does hydrology.py do then? Is it just a rough draft of a replacement module?

[ftomassetti]

Currently the lake map is not so exciting. For this world there is a single "lake" point (the red one, hard to spot)

[ftomassetti]

Watermap is used by irrigation. I think there is a dualism between watermap and rivermap. Perhaps the idea was: 1) Erosion to consider the large flows of water which evolved over time and caused erosion 2) Watermap then given a final elevation map consider how precipitation would flow in such map. At this stage we could consider not only main streams but also smaller one, obtaining a more complete picture 3) Irrigation finally the presence of streams would affect the surrounding areas (think of the Nile) and it would be affected by the drainage

[ftomassetti]

Then humidity would depend on irrigation, precipitation and temperature. And biome from humidity and temperature. Not trivial :)

[ftomassetti]

ACTUALLY the rivermap (generated by the WorldSynth code) is much better than I thought: I rendered it incorrectly (inverting x and y). Look at this:

[tcld]

And there are several lakes!

[ftomassetti]

Yep. And the same routine is used as part of the ancient map generation

Did I already say that I like how WorldEngine is coming along? :sunglasses:

[tcld]

Beautiful map! There is a huge mountain on it, should be a volcano.^^

[ftomassetti]

And guess what? Now rivers and valleys overlaps... So basically we had already much better code available but we were saving it for... Halloween I guess.

Kudos to @psi29a :)

[ftomassetti]

Yeah mountains are bit too triangular, the code commented out was to make them look more irregular...

[tcld]

Does this change to the rivers have any negative side-effects? What is the watermap used for now?

[ftomassetti]

well if you look at the patch the change is very simple and just throw away some code. The role of watermap could be discussed but right now it is used to calculate the irrigation I think. We could want to revise that but I think that fixing how rivers look like is very important at this stage and we can do that very easily.

[psi29a]

hehehehe..... you letting my work do the work for you? ;)

I thought I noticed 'my rivers' in the erosion map but didn't make the link that rivers.py didn't use that data... glad someone found it! :)

[ftomassetti]

Ok, I think we can close this one. Perhaps we could improve rivers further later but we are in a better position now.

[psi29a]

I still want to improve it! :) This can be closed.